How do I update my jailbroken Apple TV to 5.3?
If you are doing over the air(thru apple tv) update, you may have installed a hack that disables automatic updates so you won't lose the jailbreak. 
Connect to your computer and do a DFU update or uninstall the hack that disables the update from within apple tv/nito or where ever it's installed and try updating again. Although I recommend DFU upgrade.