Creating activities in code adds extraneous contact to 'with' field
So I'm fairly sure this issue was actually a dev environment specific issue. The dev database wasn't using the full 'activity' table (as it was so large) and thus creating some oddities. 

If I see it again on a live env I'll have to investigate in future though.