Low FPS after JellyBean Upgrade
Well I think I panicked too soon. It was slow because it needs to sync and index all the media files and may be some other sync. I found it out from one of my friends and also experienced it after seeing my media files (video, music etc.) getting added to the list one by one. When I upgraded to ICS there was no media in my device, so it was ready with full performance. And since my phone is overflowing with media now, it was taking some time and running the media sync in the background.  

So if you're facing the same problem, just give it a day or two to sync completely, before you try factory reset or downgrade.

PS: I've contacted Sony's customer care and they said that a downgrade can be done at any Sony service center without any charge. Just FYI.