Time series binary classificaiton with labelling issues My situation is quite complicated so I will give a similar example from a simpler domain. Suppose we want to try to predict WHEN a mobile game users will make a purchase if given a sale. Almost every user is always instantaneously a non-purchaser because everybody is constantly not buying anything. Some people buy something to instantaneously become a purchaser but go back to the standard state of not purchasing. We have a time-stamp for that purchase. If we could understand the user behavior well enough, we could predict when a user will purchase before they do so. If this is the case then users with actions and game states very similar to the purchasers but who do not actually purchase would do so if given a sale. So the question is how do we turn this into a machine learning question. My current plan is to use binary classification by labeling the YES cases as the purchasers since those who purchase at regular price would have purchased if given a sale. I build my features based on actions in recent look back windows, like how many actions of a type in the last day. For all users who never purchase I choose random time-stamps, build my features and use them as the NO case. Since I can choose as many NO time-stamps as I want I have been doing 100 times my YES case. Then I can use a classifier, I like tree ensemble methods but I do not think the classifier will really matter here.

The problem is that this is not working. There are two things worth noting which I think are at the core of the issue I am having. First, we are trying to predict the "when" not just the "who". Users who play a lot or who have purchased before are much more likely to purchase in general so it is easy to make a time independent classifier and predict who is likely to purchase. This means that the features which are useful for the easy time-independant problem can sort of "contaminate" the feature set. I have taken out a number of these features and seen some improvement. The second issue has to do with labelling. We have many YES cases for the purchasers but how does one define a NO case. I described what I am doing above but I am not sure that it is really a best method. Many of the users I defined as a NO might have effectively been a YES since if they were given the sale they would have bought.

Also, I should point out two situations which this is not. This is not one class binary classification which generally uses the large class in the imbalanced case reducing the problem to unsupervised anomaly/outlier detection. This is also not the semi-supervised situation where I have both classes labelled for a subset of the data. I really only have some of the YES cases for sure and none of the NO case. Any thoughts welcome.
From your description of the data, this is not a time-series problem. Time is not a factor here, for each user you have a set of variables after you choose a time threshold. Although you seek the "when", that's probably out of your reach, based on your explanation.

If I understand correctly you're doing classification but you're unsure about your "data collection process", at least for the "NO" labeled obs.

Why not fix a time stamp, and register all the variables you think are correlated with the outcome, and keep the outcome as you defined (never buys = "NO", buys at least once = "YES).

I'd treat this as a classification problem but instead of looking at the classic metrics (from confusion matrix), I'd be more interested at the probabilities you get from each obs, compared to the truth (that you know up to that point in time).

Imagine a user, with an estimated probability of 0.90 (90%), but the labels says "NO". He's someone who the model thinks that should have bought but didn't, he's highly likely to be interested in a sale.

Maybe you could cluster on this kind of information, creating groups of users where your model is wrong, but it's really confident in it's estimation. Basically you're more interested in false positive obs.

But also false negatives can give you insight, again looking at the probabilities you get.

I know this won't probabily be possible but if you could collect more data, I'd suggest a variable (or more), that measure the relationship of the user with sales.

Also, I agree about the decision of removing the fact that the user plays for long, that's probably "too much correlated" with your outcome.

Without knowing more about the data, that's my two cents.