Neural Network training time? I built a fairly standard backpropagation algorithm and just the process of forward propagating through a 5 layer x 5 nodes network using a data set of 10,000 observations of 39 variables takes almost 5 minutes for one iteration. 

Do neural networks typically take many hours to train using data sets this size? My initial data set was 10x as long, but I couldn't wait an hour just for one forward pass to be completed.
This is quite standard for the training time. It depends on how much optimization you did on your code. The speed of your processing unit, it's often better to use a GPU as opposed to a CPU. GPUs do mathematical operations much faster. Also, you should use parallel computing when you can, in the case of NN you definitely can.

Training a machine learning algorithm only needs to be done once. Let it run all night and then you will be ready to do some pretty good predictions.