How much do teaching responsibilities hinder a researcher at a small university?
Yes, you will have a heavier teaching load at most small universities. Such places usually have teaching as their primary mission. Teaching is usually required at large US universities as well, unless you are completely grant funded. But that isn't the only issue. 

At a small institution it is much harder to develop a synergistic core of researchers. Even if there are, say five math faculty, they are not especially likely to have the same research interests or be focused on the same field. 

Things are a bit easier in the last couple of decades with advances in communication, but you still aren't likely to see deep research seminars in a topic of interest to you at a small place. 

Also, note that in the US, most faculty are actually working at smaller, primarily teaching, institutions than large research universities. If you look at, for example, the list of colleges in a given state, say New York, you will find only a few large institutions and many, many small ones. It is a bit complicated since some private research institutions are actually fairly small. In those, the mission is primarily research or it is balanced.