Calculating capacitance of a cable
Is my formula wrong? What am I not considering here?

You are not recognizing that the solid copper volume that surrounds the two insulated conductors shortens the distance dramatically. Then if that solid copper is earthed: -

Yes the big copper is earthened. Which tool do I have to use? Or which
formula?

And if the big copper is earthed than you are using the wrong calculator tool. So, with it being earthed the capacitance between the cores is largely immaterial and of no consequence.
You can certainly measure capacitance but what does that tell you that is useful to know. Now the two cores should be treated as two separate coax cables and if you did that and asked someone what the capacitance is between the two coax cable inner conductors/cores then you would just get blank stares because it is meaningless without further context.

But when I use a time domain reflectometer I can still measure the
cable length.

TDR relies on an anomaly at the end of the cable i.e it has an open circuit. TDR does not use a capacitance measurement; it emits a very thin pulse and measures the time for it to hit the end of the cable and cause a reflection that is then received after it travels back up the cable. Totally different thing entirely.