How do we know which compounds have higher solubility?
It depends on the solution and the solute. Usually, the rule is "like" dissolves "like". A polar solution will dissolve polar compounds and a non-polar solution will dissolve non-polar compounds.

https://www.youtube.com/watch?v=PVL24HAesnc

As far as I know, to find the solubility you have to determine it experimentally. This can be done by "feel" first. If you have a compound that doesn't have many polar groups on it, chances are it can be dissolved by a non-polar solution. If you know what the structure looks like try to match it with a solvent that looks similar to it.