Disable modules when browsing mobile site
The load caused by modules is handled by the server, so merely having the modules enabled will not affect the performance of your mobile site.  In any case, it is impossible to disable the modules because you could be serving web pages to both mobile and desktop machines at the same time.

Instead, what you want to do is add a mobile theme.  There are lots of ways to do this but the Mobile Tools module is a good way to get started.

For a more specific solution, you'll have to identify which modules are causing the performance problems on your site so that workarounds can be found.