What are the aspects that most impact on the inference time for neural networks in embedded systems?
You can expect that the inference time will strongly depend on particular hardware and software present on your platform. First, GPU equipped devices (eg NVidia TX) will outperform non-GPU equipped devices (eg. Intel Movidius). Second, software support (eg. cudnn, TensorRT) will make dramatic further impact. 

For instance, we have measured the inference time of two convolutional models. The model A requires 250% more floating point operations than the model B. Yet, the two models take roughly the same time to evaluate on our device, since the layers of model A are better optimized in software. Conclusion: algorithmic complexity and practical execution time on a particular computing platform are not bound to be proportional any more.