Deployment Issue with Heroku
I eventually got the site working by disabling caching. But better approach to this problem is, creating the site on your local instance first.
You need to create and install Drupal on your local system. If you install it directly on Heroku, it creates some temporary file on Heroku, which Heroku discards after every dyno restart. Even settings.php loses all the changes made to it(during installation) after dyno restart. (I installed my site directly on Heroku, so I got through this issue by adding changes to settings.php manually)
Once you create the Drupal site on your local instance (using the GUI), you will have some files in sites/default/files, as well as some changes in settings.php (hash salt, database setting etc). Now, you can track all these files and push it to Heroku.
The site should work fine, except you won't be able to store any public files, you will need S3 bucket for that.
Hope this helps who encountered similar issue!