Stratified selection based on the y response creates a bias in information (Berkson's bias)?
It would be better to frame your problem as survival analysis, rather than classification. The goal of survival analysis is predict the duration until an event happens.