Should we always think of problems higher in the polynomial hierarchy as harder than problems lower in the hierarchy?
The quote you give is an informal claim/intuition.  Don't try to read too much into informal claims; they are intended to give you a sense of what's going on, but for specific details, you need to look at the precise technical claim.

I suspect the source of your confusion is that you are using the same phrase  in two or three different ways in different places, without realizing it.  This is the logical fallacy of equivocation.  In particular, I don't think you're using "harder than" in exactly the same way each time you use it.  This then leads to you reaching conclusions that seem absurd or confusing.  One way to remedy the confusion is to understand more clearly exactly what you mean by "harder than", and then walk through your logic.

One possible meaning of "P is harder than Q" is that the "running time of the fastest algorithm to solve P is larger than the running time of the fastest algorithm to solve Q, in some specific theoretical model of computation".  This is a statement about theoretical complexity.  Note that this meaning depends on the model of computation.  P might be harder than Q in one model of computation (e.g., classical computation), but not in another model (e.g., quantum computation).

A second possible meaning of "P is harder than Q" is "Q reduces to P".  This is a theoretical statement, which has implications for the previous statement (if Q reduces to P, then the running time of the fastest algorithm for P can't be too much less than the running time of the fastest algorithm for Q).  Note that this notion of "harder than" is as a partial order.  There might be two problems P,Q such that neither "P is harder than Q" nor "Q is harder than P" are true; the problems might be incomparable.

A third possible meaning of "P is harder than Q" is "in practice it's probably harder (takes more time or costs more) to solve instances of P than instances of Q".  This is a statement about engineering and what is feasible in practice, rather than theory.  For instance, this will depend on whether quantum computers are feasible to build, in practice -- something that no one knows the answer to right now.  If quantum computers are feasible, then we'll discover that factoring is much easier than all NP-complete problems, because a quantum computer can solve factoring quickly but can't solve NP-complete problems quickly (we think).  If quantum computers aren't feasible, then factoring might turn out to be not much easier than some NP-complete problems.

Finally, just because a problem isn't in PH doesn't necessarily mean it is harder than all problems in PH -- so just because Recursive Fourier Sampling is not in PH doesn't necessarily mean it is harder than all problems in PH (in the sense of the existence of a reduction).

As a final complication, Recursive Fourier Sampling is a funny kind of problem.  It's not an ordinary decision problem.  Rather, it's an oracle problem.  Oracle problems can't be directly compared to ordinary problems.