Programmable MP3 player with audio recording capability
Just buy any Android phone and you can easily have all of the features of an iTouch and more. I'm sure you could find one on Ebay now for $100 or less. Maybe an old G1 or something like that. You can use cell phones just fine without activating them, including Android phones. But you will want an Android that has WiFi in it if you won't be activating it. That way you can still use the marketplace and such.

Update:
I forgot that you could also look at devices supported by Rockbox. You won't likely get anywhere near the functionality of Android or IOS, but might be enough for you depending on the device.