Talking to Radio 1 Breakfast show host, Nick Grimshaw, Cowell revealed his 88-year-old mum still calls the shots.
"She still loves to tell me off," the X Factor boss confessed.
"The worst thing she can say to me if I'm got my friends round is, 'Simon stop showing off' and there's no answer back to that," he added.
"It happened to me once when I was 15 and I brought this girl round for Sunday lunch.
"And I was showing off. Then my mum said, 'Simon stop showing off' and I literally couldn't think of a reply."
Simon Cowell jokes that he still relives that day regularly but he now uses the phrase to his own advantage.
"I hated her so much," he said. "And it stuck with me. It's a really good thing to say to the other judges."
Cowell is back on the X Factor judging panel for the first time in four years.
The 54-year-old hopes his return alongside Cheryl, Louis Walsh and Mel B will help make the singing contest the force it once was.
In recent years, X Factor has fallen behind in the ratings fight with its main rival Strictly Come Dancing with more people now watching the BBC One ballroom dance show.
You can hear Nick Grimshaw's interview with Simon Cowell on the Radio 1 Breakfast show on Friday from 6:30 BST.
Follow @BBCNewsbeat on Twitter and Radio1Newsbeat on YouTube
Cheryl, Louis Walsh or Mel B can't manage it but there is one person who can put Simon Cowell in his place - Julie, his mum.