Views with attachment - How do I exclude pagination offset on every page but the first?
There is a workaround described here: https://drupal.org/comment/4109006#comment-4109006. It is for D6 but as far as I can tell it should work in D7 too. The approach seems a little odd and I didn't try it out on my own, but it might solve your problem.