How to choose docking score cutoff? I made a virtual screening of a large database (~5M) and want to filter ligand structures for further more accurate screening (from Schrodinger HTVS to SP resolution). My next step is to define a cutoff docking score, but it seems arbitrary to choose one. Are there any established methods for filtering the ligands out?
It is utterly arbitrary and depends on your resources.
We can all agree your cutoff would be a lot lower during the ordering step than at followup in silico analyses, but even then if you were ordering the top N at \$100-250 each, should you spend \$1,000 or \$10,000?
In the in silico step you have real time and CPU-time to factor in making this call even blurrier.
But there are some things to put those scores into context:

The units are kcal/mol, i.e. these are predicted ∆∆G of binding. Here is an explanation of what does a kcal/mol mean for docking — 1 kcal/mol is a hydrogen bond or the average collision energy of water at 37°C.
If you have a crystal structure with a known ligand, re-docking and cross-docking that would give you a benchmark of what you should expect ——keeping my mind that natural ligand are horrid and don't obey Lipinski's rule. The natural ligand will wander and to assess how good the docking went RMSD is used, but for this I mean looking at its GlideScore as if it were any other compound.
Rescore with a different tool —Gold etc.

Ligand efficiency
An important thing to remember is that ∆∆G is size dependent —a huge peptidomimetic will score more than a smaller molecule that hits the hotspots perfectly. Therefore, ligand efficiency (LE), ∆∆G over heavy atoms (reviewed in detail here) is a good metric to prevent huge molecules dominating spuriously. If your library has a variety of different sizes, it is best to sort by LE than ∆∆G alone. In fact, if you get a few small hits you can merge/link them or expand on them, whereas if you get nothing because you gambled on larger hits you have to start from square one. I would argue that this is true for repurposing screens and the low success of repurposing VLS despite the enthusiasm is due to the neglect of this.