Graphs demonstrating the structure of neural networks are very unclear
You are making one mistake which cascades on towards other mistakes. The multiple inputs are different features of the same sample. Let's say we want to regress house prices based on some features of a house. These three nodes could represent the rating of the neighborhood, the surface area and the number of bedrooms for example. Then you take a weighted sum to every of the hidden nodes. Every hidden node takes different weights over these features. The order you mention is per layer, since we need the output of the previous layer as input for the next one, these dependencies force this order.

With regards to output, you can have a combined loss. In case of binary classification you would probably just have a sigmoid activation at the end, however if you would have multiclass classification you would have a softmax layer at the end, one output for every potential class. There are a lot of other things you could want to output, for example a bounding box so you would need coordinates of where the network thinks the box is, or even per pixel if it's part of the object or not, then you would have a m x n grid as output.

EDIT: All the things you mentioned about the GPU etcetera, that is way too far for what this image represents. It's one network for one sample. I do agree it would have been better to just have one output node because most more basic examples in supervised learning have just 1 target.