how to delete the nodes having a specific entity reference but not the parent
If this a one-time need, you could build a custom query DB query if you wanted to go the developer route. However, if this is an ongoing need you're better off with the site builder route you've hinted at in your question. 

In your case, you want to a create a node view filtered by content type Students. Add a Entity Reference: Referenced Entity for the field that contains your references to Courses. The last step is to add an exposed filter for the same Courses field making sure enable relationship that you added; that will present a select list of referenced Courses from which you can filter down the Students and present your VBO delete widget.