iOS 5 or iOS 4 performance
No rush. If you're having a smooth experience with OS 4 for now, keep using that OS and wait a few updates... the first version (in this case 5.0) always has a few bumps. I know I have!

But if you need any one of the new gadgets (like iCloud, or iMessage), then you can switch anytime. Just make sure you do a full sync before with your iTunes (the one the iPod has always been synched to) and do a restore, not an update. It'll clean up the whole iPod and start fresh, then sync your personal data. Installing a fresh OS from time to time really does help... :-)