Can plastic leak into water from a HDPE "2" plastic container at room temperature?
HDPE 
has a molecular weight more than several hundreds of thousands, 
"leaking" is plainly impossible. 
This taste is the tase of stale water, nobody knows 
what that taste is actually. 
Maybe its just the taste of not cold enough water,
lacking some dissolved air, or some bacteriae 
develop after contact with air and light. 
(google for pseudomonas) 
The only substance which ever got into the jug prior 
to the water is the air (or some other gas?) used to
inflate the jug in production. 
But that cause would impair the taste of the water in 
the fridge as well.

BTW why do You drink such botteled water? 
What about the tap water in your town/city/village?