Should I be worried about the fact that I will be too old for applying for tenure track positions by the time I complete postdoc?
Hiring committees do care about how many years it’s been since you got your PhD, but they more-or-less don’t care at all about what happened before you received your PhD.  I think you’ve misunderstood what people meant by “younger.”  I took 7 years in grad school and it caused no issues whatsoever.