How should one deal with implicit data in recommendation
Your system isn't just trained on items that are recommended right? if so you have a big feedback loop here. You want to learn from all clicks/views, I hope.

You suggest that not-looking at an item is a negative signal. I strongly suggest you do not treat it that way. Not interacting with something is almost always best treated as no information. If you have an explicit signal that indicates a dislike, like a down vote (or, maybe watched 10 seconds of a video and stopped), maybe that's valid.

I would not construe this input as rating-like data. (Although in your case, you may get away with it.) Instead think of them as weights, which is exactly the treatment in the Hu Koren Volinsky paper on ALS that @Trey mentions in a comment. This lets you record relative strength of positive/negative interactions.

Finally I would note that this paper, while is very likely to be what you're looking for, does not provide for negative weights. It is simple to extend in this way. If you get that far I can point you to the easy extension, which exists already in two implementations that I know of, in Spark and Oryx.