How to prepopulate custom fields
I ended up going for a very hacky solution.

The buildForm hook runs twice when you are adding/editing an activity with custom fields. Once for the activity and once for the custom fields. So, during the activity run I am taking the contact ID from the form array and putting it into $_SESSION. Then on the custom fields run I retrieve it and use it to get the values from the API with which to populate them.

I had tried getting the contact ID from the URL as Sleewok suggested, but it is not in there!