Two solenoid coils and ferrite rod, how to calculate mutual inductance or coupling co-eff
The main complexity is the fact that the magnetic medium is mixed, and without obvious symmetry (apart from cylindrical).

If the coils were entirely in air, then you could do it exactly with Biot Savart, and there are two-coil approximations that are quite easy to find via your favourite search engine.

If the ferrite core was closed, then as its permeability is so much higher than 1, it could be treated as a simple inductor with magnetic length, area, and ignore the air.

As the ferrite rod is open, as there is a large air path to be negotiated by lines of flux that return round the RX coil, its influence on the inductance of that coil will be small, a factor of just a few, notwithstanding that its ur is 1000. Its effect on the TX coil will be even smaller. 

There are no simple approximations. Your options are
a) a full mixed media finite element simulation
b) empirical measurement
c) neglect the ferrite, use air as a lower bound, and say the inductances will be 'a bit more'