Analysis of Neural Network Inference Response Times on Embedded Platforms

The response time of Artificial Neural Network (ANN)-inference is of utmost importance in embedded applications, particularly continual stream-processing. Predictive maintenance applications require timely predictions of state changes. This study serves to enable the reader to estimate the response time of a given model based on the underlying platform, and emphasises the relevance of benchmarking generic ANN applications on edge devices. We analyse the influence of net parameters, activation functions as well as single- and multi-threading on execution times. Potential side effects such as tact rate variances or other hardware-related influences are being outlined and accounted for. The results underline the complexity of task-partitioning and scheduling strategies while emphasising the necessity of precise concertation of the parameters to achieve optimal performance on any platform. This study shows that cutting-edge frameworks don't necessarily perform the required concertations automatically for all configurations, which may negatively impact performance.