The department, which employs 247 staff, said it plans to shut the office by 2018 but has not said how many, if any, redundancies there may be.
It said the decision was aimed at cutting costs by creating a combined central HQ and policy centre in London.
PCS, the union, say a further 100 jobs in the UK are threatened by the move.
Permanent secretary Martin Donnelly said the move was intended to "modernise the way we work, reduce operating costs and deliver a simpler, smaller department that is more flexible and responsive to stakeholders and businesses.
"The decision to close Sheffield by 2018 has not been taken lightly.
"We will provide comprehensive support to all those facing a potential change or loss of job."
However, he said the department was planning to create "around six business centres" across the country.
The department currently employs about 2,500 people in total across 10 locations.
Mark Serwotka, general secretary of the Public and Commercial Services union, said: "The business department is supposed to lead the way in helping local economies grow after the recession, yet it is retreating from towns and cities across the country.
"We do not accept the need for these offices to close and for jobs to be at risk and will be opposing these moves."
The union has not said where the other 100 jobs could be lost.
Julie Dore, leader of Sheffield City Council, described the plan as "appalling" and said she would be writing to the Secretary of State to "make it clear that this is unacceptable".
Nick Clegg, Liberal Democrat MP for Sheffield Hallam, said the decision "flies directly in the face of the Northern Powerhouse agenda".
Nearly 250 jobs have been put at risk after the Department for Business, Innovation and Skills announced it will close its office in Sheffield.