Machine Learning hardware usage in embedded applications I've been reading a lot about hardware development and implementation for AI/ML, mainly about Deep Learning, and I have a question about its usage.
From what I understand, there are 2 stages for DL: first is training and second is inference. The first is often done on GPUs because of their massive parallelism capabilities among other things, and inference, while can be done on GPUs, it's not used that much, because of power usage, and because the data presented while inferring are much less so the full capabilities of GPUs won't be much needed. Instead FPGAs and CPUs are often used for that.

My understanding also is that a complete DL system will have both, a training system and an inferring system.

My question is that: are both systems required on the same application? Let's assume an autonomous car or an application where visual and image recognition is done, will it have both training system to be trained and an inference system to execute? Or it has only the inference system and will communicate with a distant system which is already trained and has built a database?

Also, if the application has both systems, will it have a big enough memory to store the training data? Given that it can be a small system and memory is ultimately limited.
Deep learning seems mostly to be a buzzword for what is essentially a neural network. You train with a data set to recognize a pattern, then input new data which is then classified by the trained network.

So you train a neural network with 10 different kinds of animals using thousands of pictures. Then you show the network say 100 new images and have the network "guess"  what each animal is. 

The point here is that training a neural network would required code for feedback that an application using the trained network would not need. So an application just using the trained network would be a bit more streamlined than an application which could allow additional training data. 

What is missing is the ability for machine learning to form hierarchical or otherwise interrelated rules from the trained network so that the trained network is closer to being able to rationally explain why the classification works. Going back to the 10 animal trained network, without formulation of rules during training, there is no practical way for the network to reveal why any of the animals was classified the way it was.