Pressure has grown on Shaun Wright, who was in charge of children's services in Rotherham from 2005 to 2010, to step down in the wake of a damning report.
The report found at least 1,400 children in the town were sexually exploited by mainly Pakistani heritage criminal gangs between 1997 and 2013.
Mr Wright said he was the most appropriate person to hold the office.
Jack Dromey told the World Tonight on Radio 4: "Shaun should go and we're absolutely clear if he does not agree to resign he will be suspended from the Labour party tomorrow morning."
Mr Wright needs to "exercise his conscience and face up his responsibility", said Mr Dromey.
Senior politicians from other parties have also called on Mr Wright to step down from his role as police and crime commissioner (PCC).
Home Secretary Theresa May said he should "heed calls" to resign.
She said it was not her job as home secretary to hire and fire PCCs.
She added: "I think he has real questions to answer. I think in the circumstances he should heed those calls."
BBC Newsnight's Laura Kuenssberg earlier tweeted that Mr Wright was said to be at home, "consulting with his family about his future".
Sarah Champion, the Labour MP for Rotherham, said Mr Wright "needs to stand up and be counted for what happened under his watch".
Other senior party figures, including shadow home secretary Yvette Cooper, shadow chancellor Ed Balls and shadow welfare minister Chris Bryant, also joined calls for him to resign.
Mr Wright apologised to victims but insisted he had no knowledge of the "industrial scale" of child abuse when he was a Labour councillor in the South Yorkshire town.
A report commissioned by the council was published on Tuesday and revealed the massive level of abuse in Rotherham.
The inquiry team noted fears among council staff of being labelled "racist" if they focused on victims' descriptions of the majority of abusers as "Asian" men.
1,400
children were abused, 1997-2013
1/3 of victims were already known to social services
157 reports concerning child sexual exploitation made to police in 2013
10 prosecutions were made between 2013 and April 2014
However, Mr Wright told the BBC the scale of the problem in Rotherham had "come as a surprise".
"Had I known then what I know now I could have done more," Mr Wright said.
"As an elected member I came into this role to make a difference. At every stage I've done my utmost to protect those people.
"I have taken lessons learned in that office and brought them to bear in my new role with South Yorkshire Police.
"I believe I am the most appropriate person to hold this office at this current time," he said.
Rotherham Borough Council leader Roger Stone quit following the publication of the report, which was commissioned by the council and detailed gang rapes, grooming, trafficking and other sexual exploitation on a wide scale in the South Yorkshire town.
Race, fear and abuse in Rotherham
Town consumed by shock and disgust
But calls for further resignations have gathered momentum as the spotlight now turns on to others at the helm of local services during the time of the abuse.
It has emerged that no council employees would face disciplinary action, as the authority's chief executive Martin Kimber said there was not enough evidence to prosecute.
This is despite the report's author Professor Alexis Jay concluding there had been "blatant" collective failures by the council's leadership.
Former children's minister Tim Loughton told the BBC's Newsnight that any social worker involved in the failings in Rotherham "has absolutely no place" in the care of children.
The Conservative MP claimed it was a "common theme" in sex abuse scandals that "nobody pays the consequences".
A Conservative Party spokesman said the government was "establishing an independent inquiry panel of experts in the law and child protection to consider whether public bodies - and other non-state institutions - have taken seriously their duty of care to protect children from sexual abuse".
Iain Watson, BBC political correspondent
Shadow chancellor Ed Balls says the Labour Party has "communicated" to Shaun Wright he should stand down - but under the legislation that created the PCCs he couldn't compel him to go.
While PCCs can fire chief constables, there are very narrow grounds for their own dismissal.
The Home Office says ultimately it's the local electorate who will hold them to account.
Fewer than 15% of eligible voters took part in the 2012 election in South Yorkshire - though the current controversy might spark more interest.
Local police and crime panels were set up to "provide checks and balances" and can scrutinise their actions.
But these panels can only suspend a PCC if they are charged with an offence which carries a prison term of more than two years.
As he has committed no crime, Shaun Wright can simply refuse to go.
Labour could start disciplinary proceedings for bringing the party into disrepute but neither Ed Miliband nor the government could stop Mr Wright continuing as an independent PCC.
One of the Rotherham victims, who is now 24, told the BBC's Today programme she had been groomed by men from the age of 12.
Solicitors Switalkskis confirmed on Tuesday they were acting for 15 abuse victims suing South Yorkshire Police and Rotherham Council for failing to protect them.
Karen Froggatt, a director of the charity Victim Support, told the BBC: "The details in that report were truly horrific and it's obvious that some serious mistakes were made in Rotherham.
"For the victims not to have been believed and to actually have been treated with disdain and contempt is just appalling and just makes the trauma that they've gone through a million times worse."
Several political opponents also joined calls saying Mr Wright's position as the PCC was now untenable.
The leader of the Lib Dem group on Sheffield City Council, Colin Ross, said: "Lessons must be learnt and those responsible must be held to account.
"Shaun Wright was the councillor in charge of children's services at Rotherham Council and also sat on the Authority of South Yorkshire Police when both organisations knew about the level of child sexual exploitation, but chose not do anything about it.
"It's difficult to see how local people can have confidence in him to continue as our PCC."
A member of the South Yorkshire police and crime panel, which scrutinises the PCC, has also called for Mr Wright's resignation.
UKIP councillor Caven Vines, said: "To me he should never have even put up for the job in the first place.
"He should go. He should have gone straight away."
Mr Wright has previously apologised for the failure of Rotherham Council while he was in its cabinet from 2005 to 2010.
The deputy leader of Rotherham Council, Labour's Paul Lakin, said he would not resign from the authority.
"I've been on the council since 1999, and I am one of the councillors that's seen this through," he said.
"I can categorically say that until I came into children's services in 2010, I was not aware of the depth and breadth of child sexual exploitation in Rotherham."
A Labour frontbencher has said the party will suspend South Yorkshire's PCC if he fails to quit by the morning.