Do I really need to use primer first?
I have extensive experience painting metal. Here are my rules of thumb. 


Rough up the metal, lots of nooks and crannies ensure the primer will stick. Sanding or sandblasting is the best way to do this. As pointed out in the comments, aluminium should be roughed up "nicely" :)
Clean the surface with some type of alcohol after you're done sanding. After you've cleaned it, wear gloves when handling the frame from now on. The oils on your hands can taint the paint job.
I always double coat the metal with primer because metal is prone to oxidization. Effect is diminished in aluminium, but can't be too careful. If your bicycle frame is aluminium, you'll need a special type of primer. Ask the guys at the hardware store or a car repair place what they recommend. 
Single component paint works better on aluminium. 


In conclusion:


Primer is necessary for all types of metal 
Special attention is needed for aluminium
The amount of effort you put into the paint job will show in the final product. Do it right once and your bike will look fine for years.