If aliens lived in a hydrogen (or any flammable gas) based atmosphere, would they perceive oxygen to be flammable?
Absolutely yes.  

Lighting a torch in such an environment would simply be the reverse physical process (and same chemical process) of what is done in our oxygen-containing atmosphere. In the chamber or alien world of hydrogen gas, providing an ignition source to a stream of oxygen would give a flame. The chemical reaction would actually be the same as if you had lit a hydrogen stream in an oxygen-containing atmosphere:  

$$\ce{2H2 + O2 -&gt; 2H2O + energy}$$  

The only thing we've done is reversed the sources of the $\ce{H2}$ and $\ce{O2}$, so the hydrogen-breathing aliens would perceive oxygen to be explosively flammable, just as we perceive hydrogen gas to be so.  The bottom line is that it is the two of the gases together that are flammable.