My simulated coax cut off frequency is way far off the calculated value
This model unfortunately does not work. You have taken a inductance per meter, and assumed it means that 1 meter gives you that inductance as a lumped L. What you should do instead is break up a meter (or any other length) of coax into say 100 to 1000 pieces, calculate the Rl/Rc lump elements per segment (like the one you drew), and cascade them all. The smaller the segment the better your approximation.
You'll need to create a Matlab function to calculate the voltage divider, and then call it over and over again until you have all segments covered. You will see that the resulting impedance and transfer function are different from the one-lump model you have.
Here is an example of a cascade. It's taken from wikipedia on the topic of filters, but this is the idea. You need to add your R losses, and work from one end to the other, one segment at a time. Add a matched load Zload = Z0 to start with on the right hand side.