How do I change the output power on the OS X internet sharing wifi network?
I'm not aware of a good way to modify the radio broadcast power level on the built-in card on a Mac.  However, you may find that setting up a shared internet connection using bluetooth achieves your goal of limiting the range.

Regarding your second question, there have been previous inquiries about changing the WiFi encryption protocol on Mac Ad-Hoc networks from WEP with no results.