Derive features from test-set? I have a dataset of choices (between A,B and C) done by certain users, and I want to train a neural network to predict the choices. I divide in train and test sets.
An instance is formed by: [UserId, some features, choice]. Choice is the target variable.
Same user can have multiple instances (with different features and choice).
My assumption is that the same user makes often the same choice, so I want a feature describing the &quot;frequency&quot; of a choice (e.g. User 1 choose A 60% of times).
From which dataset can I compute this frequency?

I cannot use the training set, otherwise is overfitting (with a frequency=0, the model is sure it was never chosen in the training-set).
I do not have an additional dataset of choices in a different period (this would be ideal).
Can I compute the frequency from the test-set? In that way, I'm actually training a model (on training set), based on the frequencies in a different period (the test set), which is what I want. Does it overfit the performances in the test-set? I assume not, since the model is trained on the training set. My results confirm similar performances in the two sets.
I think that the issue depends on what you'd expect the model to learn:

If the model is supposed to &quot;know&quot; the users it has seen during training, i.e. exploit the user id in order to infer particular choices for a specific user, then I don't see the point in adding this kind of frequency feature: the model already &quot;knows&quot; what choices this user tends to make, and it's supposed to know it only from the training set. Disadvantage: for any new/unique user not seen during training, the model probably has to fall back to a generic prediction.
If the user id is not used (removed from the features), then I think it makes sense to add the frequency features. I would see this mostly as a matter of feature engineering: let's assume that there are 3 features which represent how many times so far this user has chosen option A, B and C. The counting is made from any previous data available for this user. Of course, this is assuming that the same features can be calculated at testing time (equivalently, in a production environment).

So in the second option the features exist before any question about what is the training set and what is the test set. However they introduce a potential risk of data leakage, so what I would do is to separate the training set and test set based on the users, i.e. make sure that users in the training set and test set don't overlap.