30+ small modules versus 5 large modules versus 1 HUGE module: Which is better?
It sounds like you don't really know what your bottleneck is. This restructuring will likely have very little impact on performance (unless you happen to find and trim some major bloat in the process), but it may make things easier for you to maintain. If you do decide to restructure I would recommend splitting up along "site features" (blog, image galleries, etc.).

But if performance is your concern you first have to diagnose the problem before you attempt surgery.