Media playback is not supported on this device
Plenty about those days, growing up as teenage boy in north London who weighed more than 20 stone, must have been difficult.
But without that experience, Davis says he would never have learnt the skills that last month saw him become Great Britain's first European champion in fencing.
"I was 132 kilos. I was always a big lad. But that didn't stop me being good," Davis, who turned 23 at the start of this month, told BBC London.
"The speed of my hand was good. It's a sport where you can learn the basics whatever size you are.
"I always knew I had it in me, I always knew I could do it, but perhaps some people around me didn't. Perhaps when they lost to me they would think 'how did I lose to this big kid'."
When the World Championships begin in Kazan on Tuesday, hopes are high that Davis can go on to become the first Briton to make it onto the podium in the event since 1965.
While being overweight on the junior circuit might not have been a barrier to success, to really make it in the sport Davis realised he had to prioritise fencing over food.
"I was technically very good but the fitness wasn't there. I would get to a certain point and then I would start to die," he said.
"I had to make a choice. Do I want to be successful in this sport? The answer was yes. so I had to lose this weight.
"I lost it through working with my strength and conditioning coach Jon Cree at Middlesex University, who were supporting me on their scholarship scheme when I was not part of the World-Class programme.
"I started eating well, cut out all the rubbish and just started drinking water. Since I've turned professional I've trained every day and it pays off.
"There is no way someone my size could have made it onto the senior circuit, but it's a sport that does offer something for everyone who is starting out.
"I'd encourage anyone to give it a go. I learnt so much in those fat days that is still relevant today."
Last month at the European Championships in Strasbourg, Davis, who nowadays stands at 6ft 4in (1.94m) and weighs 98kg (15st 6lbs), beat Russia's Alexey Cheremisinov 15-11 in the foil final.
It was his first championship gold medal and his country's first European title.
While Britain's Gwendoline Neligan originally won European gold back in 1933, that event in Budapest was later re-classified as a World Championships and Neligan was awarded the world title instead.
Davis, who grew up and lived in Barnet until the end of last year, puts his recent success down to his relocation to San Francisco.
Last November he moved out to California to train with some of the top-ranked US fencers.
Davis continued: "Immediately I thought 'wow, something's working here'.
"Training is amazing out there, the guys are incredible. They are world class and they have such a big set-up with so many talented youngsters coming through.
"I am doing two training sessions a day, that's 10 to 12 matches a day, which is a lot. I am smashing the sessions.
"I can get four to five left-handers to fight against which is something we really struggle with in most countries. They have got an abundance of them.
"There is so much talent to train with."
Former GB foil coach Ziemek Wojciechowski continues to work with Davis as his personal coach and has travelled to San Francisco too.
"Ziemek has taught me so much and we've both learnt so much in the US," Davis added.
"My footwork has changed a bit, where my hand is has changed a bit and I'm just moving in different ways.
"It's made a big difference to my fencing. I wouldn't have won the Euros without being there."
The World Championships begin in Russia on Tuesday, with Davis' first bout scheduled for this Saturday.
Davis hopes to re-live the thrill of his Strasbourg success, but says the challenge will be tougher.
"Being on the podium for me was the most nerve-wracking bit," he said.
"I was just in the zone when I was competing in the semi-final and the final. I can't even remember the hits.
"I've spoken to friends and family afterwards and they've said 'no, that wasn't what happened. It happened like this'.
"Once it was over, I couldn't believe it. It was crazy. Being up there was the best feeling in my life.
"The quality at the worlds is a step up, but not much. All the top Europeans are there again, but then there are a lot from China, Japan and the USA in the top 16 too."
Having competed in the London Olympics, Davis' ultimate goal is to end Great Britain's half-century wait for an Olympic medal at Rio 2016.
Media playback is not supported on this device
The last time a British fencer made it onto the Olympic podium was when Bill Hoskyns won silver in the epee at the Tokyo Games in 1964.
Davis will have to qualify first but believes his London 2012 experience will serve him well.
"Competing in London was incredible," said Davis.
"It was the turning point of my career. There were thousands of people there, it was sold out for the foil, and TV cameras too. I was so nervous.
"In the individuals in the opening round I lost to (Germany's) Peter Joppich, my favourite fencer of all time. And when I look back at the match I realise that's why I lost.
"I had fenced him two months before in a Grand Prix and won 15-12. He won at the Olympics because he had already been there before.
"He knew how to deal with it. You only learn that from being there.
"The world number ones are there for a reason. They can deal with that pressure.
"I would love to go to Rio with all that experience under my belt because I will know how to deal with it. But we have to qualify first."
So as he prepares for this week's World Championships and hopefully another successful step on the route to Rio, what is he enjoying most about life Stateside?
"The help and support of everyone I am training with," Davis says.
"They have no reason to help but everyone does. That, and the food.
"The food is amazing. I have to be careful though - I don't want to go back to the fat old days."
James Davis calls them his "fat old days".