Convolution Neural Network Loss and performance
Precision

Your step change in precision looks to be almost entirely explained by the change in positive class frequency. It is reasonable to expect the proportion of false positives to increase when increasing the proportion of negative examples. Even if you assume your cv results were perfect, then you would see some increase.

As an example, assume you have cv results representative of test results 
 - which means same distribution before random under-sampling, and no over-fit to the cv set. 

Say you measured precision at 0.97 with a t:f ratio of 1:2, and for the sake of simplicity that this was due to the following confusion table:

      Predicted:  T    F
Real T           97    3
Real F            3  197


What precision should you expect when going to the real distribution? That is the same as multiplying the bottom row of the confusion table by 50. Precision is $\frac{TP}{TP+FP}$, so your expected precision would be $\frac{97}{97+150} \approx 0.39$

Recall

The same effect does not impact recall, because it is about the ratio between true positive and false negative. So when you change the ratio of positive to negative classes, in theory recall should be unaffected.

In your case, recall has been affected, but a lot less than precision. that is promising. A drop from 0.95 to 0.85 between cv and test is not great perhaps, but it doesn't point to a really major problem, just room for improvement.

There are a few possible causes. The ones that I can think of are:


Your test set might be too small, so estimates of precision and recall have large error. So in fact there is no problem . . .
Test distribution might be different to train and cv set. 
Train/CV set split might allow some data leakage (e.g. they share some common features such as data about the same person, and should be split by that common feature). In which case CV estimates could be too high.
Your mechanism for under-sampling the negative class may be biased.


What to do?

First of all, these results are unlikely to be anything directly do with faults in the model, and are not informed much by the training curves. They are also not that bad out of context (i.e. they are much better than simply guessing which items are in the positive class) - the question is more whether you could improve on them, and what the costs are to you for the different types of error. It might be worth you actually assigning real-world comparable costs to each type of error, to help decide whether your model is successful/useful and to pick the best model later on.

One thing from the training curves is that your cv and training loss look pretty close. It implies you are not over-fitting to the training data (or you should check to a train/cv data leak). You may have room to add more parameters and improve the model in general.

It is possible you could make the model even better with different hyper-parameter choices, feature engineering etc, and that would improve the scores. There is no general advice for that though, it depends on what you have available.

It might be worth experimenting with training on the unbalanced training set (take the raw data without undersampling) and instead weighting the loss function, so that costs are larger for inaccurate classification of positive class. This is not guaranteed to fix your problem, but will increase the amount of data you use for training.

Otherwise, you should investigate whether any of the possible causes listed above is likely and try to apply fixes.

Finally, in this situation, it is not unheard of to have a four-way data split:


A ratio-adjusted set split two ways:


Training data
CV or "Dev" set A

A same as production set split two ways:


CV or "Dev" set B
Test set



CV set A is used to perform early stopping and low-level model selection.

CV set B is used to perform high-level model selection against production metric.

Test set is used to assess the chosen "best" model without bias.