Performance of model in production varying greatly from train-test data
This happened to me in the past.

The first question I would ask myself will be, how am I splitting train, validation and test sets? The second one would be, are my production data in the same domain as of those in train/validation/test?

Sometimes a simple split of data might still result in testing data leaking into the validation and even test data, this is the case of domains where we want to study the behaviour of users, normally you will not want the same users in train/test/validation sets (that is, ideally a user should remain within the same data set).

Apart from that, looks like you might just be overfitting the data.