5 Gallon Drinking Water Container
Use a bucket.

Lid sold separately, about ~20 for the two of them. Alternatively, you can pick up an oak barrel. They tend to be much more expensive in the 150-200 range. You could make your own container out of clay, though price effectiveness is questionable

More to the point, I think you underestimate the utility and glory that is plastic. Any container will flavor the water, plastic least so. Plastic is the lightest especially for its durability. It's incredibly cheap. It's safe (Don't let BPA alarmists mislead you, containers exhibit safe levels or they wouldn't be approved by the FDA. It's far more worrying how much of it winds up in other places.) It doesn't rust, and is non-porous. It is transparent, which allows natural sunlight to decontaminate drinking water from harmful microbes. It is hypoallergenic.

Overall plastic is pretty neat, and I strongly recommend you reconsider your aversion to it.