Why isn't there a single best airfoil for subsonic flight?
For the same reason why we do not have one single type of aircraft flying all commercial and military missions worldwide: flight has many variables, and there is no single optimum solution. 

First off, you mention transonic flight in your question, which usually involves sonic flow at over at least part of the airfoil, which significantly alters the desired properties of the airfoil. To follow with this example, have a look at supercritical airfoils. They exhibit good qualities for their typical flight regime, albeit at substantial development and manufacture cost. A simpler NACA airfoil would be more cost-effective for applications that do not require high-subsonic cruise efficiency as one of their main points.

Additionally, there are more specific situations, like flying wings, that usually use reflex camber to deal with the lack of horizontal stabilizer. Again, this modification is rendered into pointless weight and cost if used outside this particular scenario.

Now, to specifically address your Lift-to-Drag requirement: that is simply not a good way to decide "the best airfoil for subsonic flight". High efficiency is important for aircraft with a mission that emphasizes endurance. This category is populated by gliders, and is indeed full of highly optimized designs ($C_L / C_d \approx 50$ is not unheard of) with little differences, mostly due to structural considerations. 

Now take another subsonic aircraft with an emphasis on endurance: the maritime patrol aircraft, the best known example here is probably the P-3 Orion. A glider airfoil would be terrible because of the need for thickness to accommodate a spar that can carry the engines without breaking under a bad gust. Instead it uses a NACA 0014-1.10 at the root, transitioning to a NACA 0012-1.10 at the tip, meaning the designers had to compromise between two airfoils even within the same wing.