How strict is pronoun gender applied in writing
If I refer to a group of people of both genders. would "their"be 他们的， does the male form take precedence here?

Yup.

If I refer to a group of females, is 他们的 acceptable or does it have to be 她们的 ？

Both are acceptable but if you really wanted to be specific then 她们的 would be better.

If I refer to a group of animals, should it always be 它们的 ？

Maybe  牠 would be more appropriate.

there are a few different 'ta's depending on what you want to say -- even one for 'gods' or deities