Folding bike vs mountain bike
My stable contains two folding bikes, and I love them both. I use them for different purposes. If this is going to be your only bike and you want to use that bike for cargo (with racks, etc), I recommend against using a folding bike unless it's one with larger wheels. 

Disadvantages of folders:


For offroad riding, nothing beats a standard rigid-frame mountain bike. If you forsee a decent amount of offroad riding, I strongly recommend against getting any folding bike.  
Coasting on a folding bike isn't quite as easy, due to the diminished momentum of the smaller wheels. This is a very minor point, though. 
Getting parts for a folding bike can sometimes be a little more difficult. A little planning can alleviate this, as can keeping spare tubes, spokes, etc around the house. 
Mounting panniers on a folder's rear rack tends to result in heelstrike, unless you have a folder with a very long wheelbase that's meant for touring. If you want to use this bike for shopping, that's a point against folders. 
Folding bikes are more expensive, and can be heavier, then their non-folding counterparts. 
Folders usually have a weight limit. The wheels are usually smaller and therefore sturdier than larger wheels, but the frame is a limiting factor. For example, the Dahon Curve has a rider limit of 100kg/220lbs. (When I bought mine, that was an across-the-board limitation, but they may have improved that.) So if you're a big guy, look into this.  


Advantages of folders: 


Storage is easier, obviously. A plus in small apartments. It's also simpler to put the bike in a car trunk or back seat, or take it on public transportation. 
You can accelerate crazy fast on a folding bike, for the same reason you have to put a little more effort into coasting - the wheels are smaller and therefore lighter. 
Small wheels are fun! Folding bikes can be more nimble than bikes with full-sized wheels. 


I love folding bikes, but unless you have no other options, a small-wheeled folding bike wouldn't be a good fit for your purposes. A folding bike with 26" or even 20" wheels, however, might work very well if you can fit it with a rear rack. Try one out in a store and see how it feels and if you can carry cargo easily.