Device in Bootloop using encryption
After 2 hours of waiting, I long pressed the power button for 15sec which gives battery pull effect. Then I rebooted to recovery. Tried to format /data sparing /data/media which gave the error unable to mount /data. Then tried formatting /data including /data/media and it was formatted.

Then rebooted and first set a lock screen pattern and then tried encryption. This time, it got successfully encrypted in around 10minutes. The device asked for pattern on every boot, and it takes couple of minutes for every reboot. 

So, I think the issue was doing encryption without lock screen password and then adding password later.