No studs for glass shower door - Can I hollow wall anchor?
Yes you can install it.   I would use a very rigid highly rated (100 lbs plus) toggle anchor and epoxy anchor to tile.  

Honestly these screws aren't doing much.   Your side bar is sitting on shower pan, the top bar should not be loose, therefore pushing side bars, the screws literally just need to make sure the side bar stays on the wall/tile.   There may be some slight force against during shower door openings but nothing significant.   No issues at all with this install.