iPhone 4 volume is really quiet
For Music anyways, check your EQ in the Music tab of Settings. Some sound quieter then others and vice versa.

For Speaker Phone I'm not so sure, but as a universal problem it may be Volume Limit, also found in the Music tab of Settings.