Caching and TTFB with Craft
This isn't an issue with Craft, from the POV that the TTFB should be pretty consistent.  How are you testing it?

My guess would be that the issue is with your hosting. These two articles may help you:

The Craft {% cache %} Tag In-Depth

A Pretty Website Isn't Enough

Run some of the diagnostic tests listed in the second article.