Which mathematics are required for fully understanding the theories of the firm?
I haven't gone through all the papers so I just sampled 'The Firm as an Incentive System'. Browsing through it, it relies on Linear Algebra and Real Analysis. Again, I warn that I have not gone through the entire paper but seeing some of the terminology used there, I could guess that these two a clearly involved (maybe they use some bit of topology as well). They use a concept of sublattice somewhere and that pertains to group theory.
A good starting place for real analysis could be Steven R. Lay's 'Analysis'. There you can learn some of the basic terminologies that they have used like supremum and infimum. For Linear Algebra, it has got to be Gilbert Strang's text 'Introduction to Linear Algebra'. Linear Algebra is used extensively in Econometrics as well. You can find his lectures at: https://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-spring-2010/ .
Hope this helps!