How to produce dry pure oxygen
Commercial drying agents, such as Drierite (anhydrous calcium sulfate), or other desiccant, should do the job for small quantities of oxygen.

Most inorganic desiccants should not react with diatomic oxygen; as long as the oxygen is produced separately from the desiccant, there should be no issue. (Dropping 90% or stronger $\ce{H2O2}$ on many substances can cause dangerously rapid decomposition, but with 30% or lower concentration, that danger is minimal.)

As far as preventing atmospheric gases from entering, first bleed the apparatus of residual air by discarding the initial output. Seal the later containers underwater, e.g. by sliding a glass plate across the mouth. See Janice Crowleys' demo on YouTube.