Proper evaluation method for recommendation system with implicit feedback?
First off, in terms of how to make a train/validation/test split, I would only include users with 2 or more interactions, and for each user, remove the latest interaction. Use this latest interaction as a target and recommend, for each user, their top streamers. You then compare your recommendations with the target, and compute a performance metric. Depending on your performance metric of choice (next paragraph), the number of recommendations you make might differ. This gives you a training/validation split. If you want more training data, and you believe a user's interactions are not sequential in nature (eg. liking StreamerA before StreamerB displays a different preference than vice-versa), you can permute each user into more users by leaving out a different target each time. So for each streamer u, who has n_u interactions, leave out one interaction (target), and keep the rest, and do this n_u times.
Secondly, the appropriate performance metric depends on your use case. If you are going to recommend, say k, streamers for each user, then the performance metric of choice should consider only these k. In other words, for any model performance, rank k+10 is no better than rank k+20. Moreover, maybe it is only of importance that the target is part of the top-k recommendations, then some XYZ@K-metric is most appropriate. Personally, I would probably go with top-k accuracy and average this across all users. This metric then specifies, for all my users, how often, on average, is the target in their top k recommendations.