Accuracy of KFold Cross Validation for Neural Network
For some context, the entire point of running K-Fold cross-validation to compute an estimate of the performance. If the obtained values vary a lot, it just means that your estimate is less precise (i.e. has a larger standard deviation)

In a neural network setting, your network evolves, so obviously, averaging all folds might give you a less accurate estimate if the network learned a lot in the last few iterations for instance.

Typically, you would always take the fold at the end of training, since this is the performance you currently have.

However, it is good to note that analyzing the variance of your performance within a short window can help you understand whether or not your model has converged or whether it is still "exploring".