McCall said he and some of his players were goaded by supporters following their 1-0 loss to the Lions at Wembley.
He questioned why stewards had not prevented the invasion.
"We're absolutely distraught, we wanted to clap our own fans, the last thing we need is people running around us and giving us verbals," said McCall.
"I think it's nonsense," he told BBC Radio Leeds. "Ten minutes later an army of stewards comes on when they should have been on straight after the final whistle and stood on the halfway line.
"I don't understand why people feel they need to come over to players, intimidate us and wind us up. I was close to clocking a couple.
"I'm not necessarily blaming the people who did it but where was the stewarding? Why march them out 10 minutes afterwards?
"They should have got to the halfway line straight away and stopped anyone from coming over."
Millwall boss Neil Harris encouraged his side's fans to leave the pitch to allow the team to enjoy their celebrations, while match-winner Steve Morison claimed they had "ruined" the moment.
"It's not ideal, it's not quite the place for a pitch invasion, I've got to be honest," Harris told BBC Radio London.
"I don't condone it whatsoever. I'm just hoping there was no disrespect towards Stuart and his team.
"For me, I wanted to get them off because I wanted to see my players enjoy it and I wanted to see my players take the admiration of the whole fan base, which happened in the end.
"I think it was overzealous, a lot of people wanting selfies and jumping about. I certainly don't condone it, but I'm hoping it was lighthearted."
Bradford City boss Stuart McCall said he was "close to clocking" Millwall fans who invaded the pitch at the end of the League One play-off final.