As a referee, is it okay to ask for references to be removed?
Yes, the spurious references should be removed, as stated by @Solar Mike. There is a diplomatic way to accomplish this without making any assumptions or accusing anyone of anything. Write something along the lines of 


  The current bibliography contains a number of references which are not explicitly cited in the main text. They are: (insert complete list of numbers). The main text should either be extended to include a discussion of these references or the references should be deleted. 


You compile the list by ticking of all explicitly cited references as you read through the main text. Any entries left are spurious. 

A tool like LaTeX typically has a command which is used to print the entire bibliography. Spurious references are often generated by forgetting to disable this command when the final draft is compiled. If you can identify the tool, then you can insert a helpful statement to that effect, i.e.


  As you are using LaTeX, please check if manuscript contains a \nocite{*} command.



EDIT: I just realized that my original text does not cover the case where the manuscript contains references which are cited in the main text, but appear to be irrelevant. A diplomatic approach can proceed along the lines of:


  The author(s) should extend the discussion of the following references: (insert complete list of dubious references). Currently, it is not clear to the reader how these references relate to the subject under discussion.


This allows the authors to either explain why the references are relevant or remove them if they are irrelevant.