How to review a paper that relies heavily on references to unpublished work?
Every matter in a paper should be verifiable, including the references. All results should be supported by relevant explanation with sufficient detail. 

Instead of rejecting the paper straight away, you may initially tell the author of correspondence (through the editor) to provide the unpublished cited materials. The author is to be obliged to provide you the tangible material from which you may assess the credibility of the facts cited. 

Bear in mind that all communication between the author and reviewer must go via the editor of the journal.