Wanderers parted company with Neil Lennon after former player Dean Holdsworth's company completed a takeover of the Championship side.
Brown made more than 250 appearances for Bolton before serving as assistant manager for over six years.
"I have had no approach from Dean Holdsworth," he told BBC Essex.
"I have sat the players down this morning. We have spoken about the speculation and newspaper headlines, which inevitably you have to print.
"Speculation can take our players and my eye of the ball and that's the last thing that you need at this stage of the season."
The Shrimpers, who were promoted via the play-offs last season, are four points off the League One play-off places with 10 matches remaining.
"We are involved in winning football games and on Saturday we need 100% focus from myself, the management team, the players and the supporters," Brown added.
"If the supporters think there is a manager standing on the sidelines that will not be there in a week or two, you won't get the same level of support.
"We need it at this moment in time. It is so important that we get back-to-back promotions and get the status of a Championship side."
Southend boss Phil Brown has said he will not be taking his "eye off the ball" despite speculation linking him with his former club Bolton.