It was back in the early 1980s that Mr Gardner, then aged 27, and his toddler son were homeless for a year in San Francisco.
Enrolled on a low-paid trainee scheme at a stock brokerage, he didn't have enough money to raise the deposit to rent an apartment.
So Mr Gardner, who was estranged from his partner, and Chris Jr would instead sleep wherever they could.
In addition to the toilet at a railway station, they'd bed down in parks, at a church shelter, or under his desk at work after everyone else had gone home.
They ate in soup kitchens, and what little money he had was spent on putting his son in day nursery so he could go to work.
Despite this adversity, Mr Gardner thrived in his job. A natural at selling stocks and shares, at the end of his training period his firm, Dean Witter Reynolds (DWR), made him a full employee.
Finally able to rent a home for himself and his son, his career then rapidly ascended into the stratosphere, and in 1987 he opened his own investment firm, Gardner Rich.
Today Mr Gardner, 62, is worth an estimated $60m (Â£48m), travels the world as a motivational speaker, and sponsors a number of homeless charities and organisations that combat violence against women.
Add the fact that Mr Gardner had a very troubled childhood, and served time in prison immediately before the internship at DWR, and you can understand why Hollywood came calling when he was writing his best-selling autobiography The Pursuit of Happyness (the misspelling is deliberate).
The movie of the same name was released in 2006, and Will Smith was nominated for a Best Actor Oscar for his portrayal of Mr Gardner.
Looking back on his life, Mr Gardner tells the BBC that he "wouldn't change anything".
"I went through pain as a child so my children wouldn't have to," he says. "I made a decision as a five-year-old boy that my kids will know who their father is.
"The rest of my destiny came forward because I made the right choices."
Born in Milwaukee, Wisconsin, Mr Gardner never knew his real father.
Instead he was brought up in poverty by his mother Bettye Jean and a physically abusive, alcoholic stepfather.
There was also a period spent in a foster home after his mother, in a fit of despair, tried to kill her partner.
Despite the woes of his childhood, Mr Gardner says his mum was an inspiration.
"I have one of those old-fashioned mothers who told me every single day, 'Son, you can do or be anything that you want to do or be.'
"And I believed it, I bought into it 100%."
He adds that one day as a child he was watching  a college basketball game on TV when he commented that one of the players would make a million dollars.
"My mum said, 'Son, one day it will be you who'll make a million dollars.' Until she said those words the thought had never entered my mind."
The million dollars didn't arrive immediately however, and after graduating from high school he spent four years in the US Navy.
After an honourable discharge in 1974 Mr Gardner moved to San Francisco where he started selling medical equipment.
His life then changed completely when he saw a man parking his red Ferrari one day, and Mr Gardner asked him what he did for a living.
The man was a stockbroker called Bob Bridges, and after the two got talking, Mr Gardner expressed an interest in joining the industry.
They met again, and Mr Bridges helped Mr Gardner secure an interview for an internship.
However, in the days before the interview at DWR Mr Gardner was arrested and jailed for non-payment of parking tickets.
He was ultimately able to make the interview, but only in the clothes he was wearing when he was arrested, which included trainers and a vest.
Despite the scruffy attire, his sense of drive and enthusiasm was enough to get him the job.
Six years after the release of the movie, Mr Gardner's life changed again in 2012 when his wife died from cancer aged just 55.
It made him re-evaluate what he wanted to do for a living, and after three highly successful decades in finance he decided on a complete career change.
"Some of the last conversations [my wife and I] had were her saying to me, 'Now that we can see how truly short life can be, what will you do with the rest of your life?'
"When you have that conversation, that changes everything. I've said that if you're not doing something that you're passionate about, you're compromising yourself every single day."
So realising that he didn't want to work in investment banking any longer, he reinvented himself as a motivational speaker and author.
He now spends 200 days a year travelling the world speaking to packed audiences in more than 50 countries.
Scott Burns, a director at US investment firm Morningstar, says that Mr Gardner "is just an amazing testament of fortitude".
He adds: "You can be down and out, but only as down as you let yourself be."
Mr Gardner believes he disproves the theory that we are all products of our childhood environment.
"According to that school of thought I should have become another alcoholic, wife-beating, child-abusing, illiterate loser."
Instead he says he made his own positive choices thanks to the love of his mother and support from other people.
"I chose light, from my mother, and from others with whom I don't share a single drop of blood, and I embraced it."
Follow The Boss series editor Will Smale on Twitter @WillSmale1
When Chris Gardner and his young son were sleeping rough on the floor of a public toilet, he could never have dreamt that his life story would be turned into a hit Hollywood movie.