Activity iCal : Start time = End time
Check:

templates/CRM/Activity/Calendar/ICal.tpl

I just took a quick look there and seems like the duration is not taken into account at all, which is a bug :(

Can you take a look and see if you can fix it. If so, please do submit a PR against the CiviCRM github repo. I'm pretty sure the activity object will be sending in the duration field. You should also file an issue for this on the CiviCRM issue tracker