Can I use a graph or figures of a paywalled paper under fair use?
Whether the articles were paywalled or not, you should resolve your problem in three steps:

Content license: check if you can find a version of the figure with liberal licensing terms. The paywalled version certainly does not fit the bill… but you may find the same figure (or a similar one) in a pre-print version of the paper, under a different license.

Obtaining permission for use: even if the content is not available under a free license, the publisher can grant you permission to reproduce the image. Nowadays, many publishers use a nice interface called Rightslink. If you look for example at this random article from the American Chemical Society (which is far from being the most liberal of publishers), you can see a “Rights and permissions” link. In that particular example, the publisher will grant you reproducing of up to 4 figures of the paper on the internet, even on a for-profit website.

Fair use: the fair use doctrine is a tricky, murky area of US copyright law. It's not as wide as people generally assume. It has to be judged on a case-by-case basis: see here and there. See here a notorious case from back in 2007 which is very close to your particular case.
So, my own conclusion on fair use: it's nice in principle, but unless you're ready to hire a lawyer to double-check (or get formal training on that), just play it safe.