Accessing certain diacritical vowel characters in the Spanish layout in Mac OS X
Option/alt + ñ on the Spanish ISO layout, followed by a or o gives you ã and õ.

To make a custom layout, use Ukelele

http://scripts.sil.org/ukelele

PS If you are running 10.7, you can also get those characters by holding down the a and o keys until the Character Picker popup menu appears.