Foreign exchange market forecasting with neural networks
The results you're seeing aren't a byproduct of your training product, but rather that neural nets are not a great choice for this task. Neural nets are effectively a means to create a high order non-linear function by composing a number of simpler functions. This is often a really good thing, because it allows neural nets to fit very complex patterns.

However, in a stock exchange any complex pattern, when traded upon will quickly decay. Detecting a complicated pattern will generally not generate useful results, because it is typically complex patterns in the short term. Additionally, depending on the metric you choose, there are a number of ways of performing well that actually won't pay off in investing (such as just predicting the last value in your example).

In addition the stock market is startlingly chaotic which can result in a neural net overfitting. This means that the patterns it learns will generalize poorly. Something along the lines of just seeing a stock decrease over a day and uniformly deciding that the stock will always decrease just because it was seen on a relatively short term. Instead techniques like ridge and robust regression, which will identify more general, less complex patterns, do better.

The winner of a similar Kaggle competition used robust regression for this very reason. You are likely to see better results if you switch to a shallow learning model that will find functions of a lower polynomial order, over the deep complex functions of a neural net.