England manager Sam Allardyce has backed the idea of a British team at future Olympics, but the Football Association of Wales fears it could damage Wales' position as an independent footballing nation.
Team GB entered men's and women's sides at London 2012, but not at Rio 2016.
"For us, no. I don't agree with that," said Coleman.
"Anything that could put what we've got here, what we've built here, in jeopardy, we would not be for that.
"We've always had a stance that we don't agree with it, and that hasn't changed.
"Why should we say go on, take [our players] - and we know they're only going to take two or three of ours.
"It should be called the England Olympic team really because that's what it will be. But they'll cherry-pick [Gareth] Bale or Rambo [Aaron Ramsey] maybe - why should we give them up, say water them down, take the best out of them and then throw them back to us when you finish with them?
"I don't think that's fair at all. I think they have enough football."
After taking part at London 2012, the four national football associations could not agree on sending a British women's team to the Rio Games.
A place was earned by the England women's side finishing third in the 2015 World Cup.
England's Football Association had put forward the idea of sending Great Britain teams to the Olympics, but Fifa said it would need the agreement of the ruling bodies in Scotland, Wales and Northern Ireland, who were against it.
Those three home nations fear such a move could affect their independent status within the sport's world governing body, Fifa, and at international tournaments such as the World Cup.
"One thing we've got surrounding us at the minute - and you can smell it on us - is the players we have got, there's an identity there," said Coleman.
"They put the jersey on and they wear it with pride. That's all our nation has ever wanted really.
"Of course they want success, they want to see a team on the pitch that when they walk on the pitch they're emptying it for the jersey, for the badge, for the honour. That's what we've got, so no need for us to mess about with that."
Coleman added that competing at the Olympics would add to players' already busy schedule.
"Our boys have a lot of football, what they don't need is another [tournament]," he said.
"It's not just the football... it's being in camp, being away every day, there's more pressure, there's more preparation.
"That's the last thing any of our top players need and I'm talking about the England players as well."
Wales boss Chris Coleman has dismissed the idea of Wales being part of a Team GB football side at the 2020 Olympics.