As a result, people should be cautious about using them to judge what to eat, Stanford University scientists said.
The study recommended that companies release data showing how their devices work out measurements.
The accuracy of seven wrist devices were tested while 60 volunteers were asked to walk, run and cycle.
Researchers found that six out of seven of the fitness devices were good at estimating the heart rate of the person wearing it, with an error rate under 5%.
They were the Apple Watch, Fitbit Surge, Basis Peak, Microsoft Band, PulseOn and MIP Alpha 2 - but the Samsung Gear S2 had the highest error rate of 6.8%.
However, when it came to keeping track of energy used during exercise, the five devices that performed this function were all a long way out.
Not one of the devices had an error rate below 20% - and some, such as the PulseOn, were much more inaccurate, the US research team found.
Dr Euan Ashley, co-author of the study from the department of cardiovascular medicine at Stanford University, said the public should be aware of the strengths and limitations of fitness devices worn on the wrist.
"People need to know that on energy expenditure they give rough estimates.
"If you go to the gym, and you think you've lost 400 calories, then you might feel you've got 400 calories to play with," he said.
That could be an issue if people were basing their diet on what they thought they had burnt off, he said.
Technology for measuring heart rate had moved on quickly over the past five or six years, but on energy expenditure "it's not quite there yet", he added.
It may be that companies are not using heart rates in their calculations. There is also a very wide difference in calories burnt between one person and another.
For example, 10,000 steps could equate to anything from 400 kilocalories to 800 kilocalories lost, depending on a person's height and weight alone, the study said.
Dr David Ellis, lecturer in computation social science at Lancaster University, said working out the number of calories burnt was a tricky business and relied on many different factors such as height, weight, percentage of body fat, heart rate and more.
"However, because manufactures do not share the algorithms [which are constantly updated] used to determine calories burned, it is almost impossible to know the exact source of error at this stage," he said.
Fitness trackers can have a very positive effect on people's activity levels, so complete 100% accuracy may not be vital if the devices are encouraging more exercise.
But recent trials have suggested that wearing them doesn't always result in a positive outcome, or improve the chances of losing weight.
The researchers urged companies to be more transparent and publish the results of their own accuracy tests.
They said this would ensure the public and clinicians were aware of the devices' limitations.
The Stanford team has set up their own website to share this kind of data with others.
Most fitness trackers are good at measuring heart rate but poor at measuring calories burned, a study suggests.