New study reveals where the Apple Watch gets fitness data right -- and wrong

AppleInsider

A new University of Mississippi meta-analysis of 56 studies finds the Apple Watch delivers fairly accurate data for heart rate and step counts -- but shows a much wider margin of error when it comes to calories burned.

The Apple Watch is pretty accurate -- at least for certain metrics.

In the year 2025, wearable tech is almost startlingly common. If you walk into any gym in the United States, chances are you're going to see wrists adorned with some kind of fitness tracker or smartwatch.

And it makes sense, too. Whether you're concerned about your health or just looking to dial in your workouts, wearing an Apple Watch -- or any other fitness tracker -- seems like an ideal way to do it.

But that begs the question -- just how accurate are health trackers like the Apple Watch? That's the exact question that University of Mississippi researchers set out to answer.

The Apple Watch knows how much you move, not how much you burn

The Ole Miss study analyzed data from 56 studies comparing the Apple Watch data to clinical reference tools. Researchers took factors like age, existing health conditions, device age, and activity into account.

And, as it turns out, the Apple Watch is, well, fairly accurate.

If you're using yours to track your heart rate or step count, they're quite accurate. The study found mean absolute percent errors, a standard measure of accuracy, of 4.43% for heart rate and 8.17% for step counts.

Energy, expenditure, on the other hand, was notably less accurate. The study showed that when it came to tracking calories burned, the Apple Watch was inaccurate nearly 28% of the time.

The energy expenditure inaccuracy was observed across all types of users and activities tested, including walking, running, cycling, and mixed-intensity workouts.

The Apple Watch remains a useful tool, not a diagnostic device

So you may be wondering if this means you should throw your Apple Watch into the trash. Obviously, it doesn't.

University of Mississippi professor Minsoo Kang, who headed the study, still thinks these trackers have a place.

"These devices are great for keeping track of habits and staying motivated, but do not take every number as 100% truth, especially the calories," he told Eureka Alert. "Think of it as a helpful guide, not a diagnostic tool. It is useful but not perfect."

While they're not perfect now, that doesn't mean they're not improving over time, either. Newer models of Apple Watch appear to be significantly more accurate than previous iterations.

"While we cannot say every update is a big leap forward, there is a noticeable trend of gradual improvements over time," Doctoral student Ju-Pil Choe said. "It shows that Apple is refining the technology over time."

Kang remains optimistic about the future of wearable tech. He hopes that the study can help users make informed decisions about purchasing wearable tech and that it could help inspire fitness tech creators to reexamine where devices fall short.

"By showing where the weaknesses are, we can help developers get real feedback," he says "They know what needs to be fixed, they can design better sensors or algorithms."

"Our findings can guide improvements and help make these devices more useful for both everyday users and health care providers."



Read on AppleInsider

rabbittomato

So which device is the most accurate for calories burned? 

Appreciate the advice of the AI followers. 

hmurchison

We all have different metabolic rates so the variance will always be there.  I'd rather be close enough because in the end the scale is going to tell you whether you've ingested more calories or less over a given period. 

massiveattack

I would not be surprised if Apple becomes a healthcare company one day. 

The big picture of Apple is that Apple wants to sell AppleCare for you and yourself (not devices). 

I am wondering if Apple will make it. It is hard. But obviously, it is the only way that Apple can grow from now on as their business model is challenged from every corner (App Store f*cked ip, Siri voice assistant f*cked up etc.).

crossplatformfrogger

I noticed this when I stopped using my Apple Watch. I used to do the cycle classes on Apple fitness and it would tell me I burned 600 calories after 30 mins, then I'd do weight lifting for a total of 1000 calories burned a day. When I stopped using my Apple Watch the bike told me I was only burning about 300 calories. I've noticed this in the past as well while hiking. Before switching to all Apple I had Samsung and the mileage on both devices would be different with the Samsung one mirroring what the mileage should be per the hiking info and the apple one way off 

king editor the grate

“So you may be wondering if this means you should throw your Apple Watch into the trash. Obviously, it doesn't.”

O, crap. Gotta remember to read to the end of stories before impetuously hucking things into the bin …

lowededwookie

Tracking calories burned is nomenclature at best.

You don’t burn calories at all. What happens is you convert fat or sugar into Carbon Dioxide and water via a chemical reaction. This chemical reaction generates energy which is used by our muscles.

The rate of this chemical reaction is dependant on how much exercise we do, how much oxygen intake, and how much fat we actually have.

Fat is not burned off and become nothing. We literally breathe out 84% of it as carbon dioxide and the rest is water we either sweat out, absorb into our cells, pee out, or poo out. That’s it. Nothing more nothing else.

As it varies from person to person the idea that a tracker can be accurate to the nth degree is in itself wildly inaccurate. But 28% means the Apple Watch is still accurate 73% of the time. That’s freaking good really.

tht

lowededwookie said:

Tracking calories burned is nomenclature at best.

You don’t burn calories at all. What happens is you convert fat or sugar into Carbon Dioxide and water via a chemical reaction. This chemical reaction generates energy which is used by our muscles.

The rate of this chemical reaction is dependant on how much exercise we do, how much oxygen intake, and how much fat we actually have.

Fat is not burned off and become nothing. We literally breathe out 84% of it as carbon dioxide and the rest is water we either sweat out, absorb into our cells, pee out, or poo out. That’s it. Nothing more nothing else.

As it varies from person to person the idea that a tracker can be accurate to the nth degree is in itself wildly inaccurate. But 28% means the Apple Watch is still accurate 73% of the time. That’s freaking good really.

Nice post!

Everyone should be skeptical of advertised accurate measurements of calories burned without knowing how they measure, this includes UMiss. You need to know a lot of things, including what you are pooping out, room temperature, body mass to surface area, microbiome, and whatnot. 

All the Watch is doing is correlating heart rate and weight to a set of data where they have good estimates of calories used. With the workouts, they can narrow the correlation to the specific set of data corresponding to the workout, and include things like distance and elevation traveled.

If you are comparing to the Watch’s calorie count to a workout machine’s calorie count, and they are off, I’d bet the machine or the Watch does not know your weight, or doesn’t have the same weight. 

The workout machine’s calorie count likely doesn’t have a good time history of your heart rate. How they take sporadic heart rate measurements and apply it to the whole workout can also change things. 

anonymouse

lowededwookie said:

But 28% means the Apple Watch is still accurate 73% of the time.

Despite the wording in the article, I don't think this is what it means. I think it means something closer to it's off by an average of 28%, meaning it may never be entirely accurate, but could, for a crude example, vary from being off by 20%-36%. In any case, as a rough estimate, it's probably fine and as someone else pointed out above, in the end, the scale will tell you what you need to know.

maltz

I've wondered about its accuracy for sleep statistics as well.  I've no specific reason to doubt them, just curious, since it's obviously not going to be as robust as even the home sleep study devices that have a finger O2 meter and a chest sensor.

neutrino23

anonymouse said:

lowededwookie said:

But 28% means the Apple Watch is still accurate 73% of the time.

Despite the wording in the article, I don't think this is what it means. I think it means something closer to it's off by an average of 28%, meaning it may never be entirely accurate, but could, for a crude example, vary from being off by 20%-36%. In any case, as a rough estimate, it's probably fine and as someone else pointed out above, in the end, the scale will tell you what you need to know.

I suspect that if they tracked one person it would be consistently off the same amount (either too high or too low). As a physicist I deal with measurements all the time. Noisy measurements are better than no measurements. 

I have my calorie target set to a certain value. I don’t much care what the number is. I just like to know that I’m reaching roughly the same level when I work out. I set it high enough that I have to strain a bit to achieve it.

mpantone

MassiveAttack said:

I would not be surprised if Apple becomes a healthcare company one day. 

There is no chance Apple will become a full fledged healthcare company. The regulatory oversight is tremendous and not just the FDA in the USA. There are Apple Watch features that aren't available in other countries because getting local regulatory approval is too difficult. So those features are left off in those countries. And, Apple debuted then dropped pulse oximetry because of their legal action with Masimo. Apple will continue to add health-focused features as informational goodies, not as clinical diagnostic tools.

There are tons of stories of "my Apple Watch alerted me to _____ medical condition that I did not know about" which always leads to a visit to a physician who does the actual investigation. Apple simply does not want to be in that healthcare space. And we haven't even touched on liability exposure.

The best Apple can do is provide wearables that tell the user "readings are out of normal range, go see a doctor". Apple simply cannot put 10+ years of medical schooling, years of training and experience on a $300 device on your wrist and expect medical regulators to rubber stamp it. NOT GOING TO HAPPEN.

Same with banking. Apple does not want to be a bank. Again, regulatory compliance is very, very stringent and Apple does not want to subject themselves to that. Why do you think they use third party financial institutions for Apple Pay Cash and the Apple Card? And previously teamed up with BarclayCard? Hell, Apple hasn't brought the Apple Card to any other market. It's still a USA-only service after all these years.

Apple doesn't want to be in the business of consumer debt. Hell, Goldman Sucks -- their Apple Card partner -- has belated decided that they don't either. And banking is their business.

And Apple doesn't want to be an ISP either, a dumb pipe. Again, the telecommunications industry is heavily regulated and there's not much in the way of innovation. Something like an in-house modem with better battery performance potentially benefits everyone using an iPhone. Something like being an American cellular carrier only affects a small percentage of your userbase. You think Apple wants to be responsible for hundreds of thousands of cellular towers, sending technicians to replace balky transmitters in the middle of a hurricane? NO.

edited June 5

entropys

I would like Apple to get into hearing aids as an obvious extension/derivation of their investment in airpods. There is a market ripe for disruption every bit as communication aids for those unable to speak by the iPad.

edited June 5

loopless

Well, "calories burned" was always going to be a "best guess". More to the point the researchers should be doing the same analysis on watches from Garmin, Samsung etc as well.

I can see how this will morph into an Apple-bashing series of articles about how inaccurate the Apple Watch is without the background that all these devices can only make a rough estimate.  I wonder if a lawsuit will follow...

stabitha_christie

Two points: 

1. Several commenters obviously didn't take so much as a passing glance at the study.

2. The article does the study a bit of a disservice. It is a meta-study, so they didn't test anything; they relied on other testing. Nothing wrong with meta-studies, but it is important to know when looking at the results. Of the studies they looked at, almost half used the Series 1 Apple Watch, and when it comes to calorie expenditure, the newest watch was the Series 6. The percentage of error for the Series 1 was 31.5%, and the Series 6 was 26.2%. So the average is 28%, but Apple has improved accuracy over time. Heart rate and steps also have become more accurate. Given that we are currently on the Series 10, we can reasonably assume that it has probably improved from the Series 6. There were some other tidbits that were interesting, like the calorie count is more accurate the more intense the exercise is and running is more accurate than cycling.

mpantone

entropys said:

I would like Apple to get into hearing aids as an obvious extension/derivation of their investment in airpods. There is a market ripe for disruption every bit as communication aids for those unable to speak by the iPad.

Again, this is veering toward the "medical device" category. Already Apple cannot tout AirPods Pro as being a hearing aid alternative in neighboring Canada based on local medical device regulatory control.

The most reasonably realistic approach is for Apple to create "helpful gadgets" that suggests users to seek professional medical consultation when some sort of usage is recorded out of the normal window.

True hearing aids are heavily regulated by the FDA. Apple does not want to get into this otherwise they would already be in this space. The regulatory hurdles are substantial, and not just here in the USA.

They are already facing enough regulatory headwinds just getting iPhones certified in global markets. Selling a limited demand niche device in a heavily regulated environment simply isn't in Apple's business model. Hell, they could sell Xserve-type servers and they don't bother anymore even if those devices are essentially free of regulatory approval hurdles.

edited June 5

kiwimachead

Been an Apple customer for 30 years and have considered an Apple Watch, but Garmin smartwatches IMO are superior for fitness, sport, health tracking and you also get battery life for weeks (or more) in almost all their watches. The depth of information in Garmin Connect on iOS and other free iOS apps like Garmin Golf is amazingly informative and you pretty much get everything without a paid subscription. The Apple One premier subscription (incl Apple Fitness+) is a reasonably slick looking package but it currently costs me $670NZD per year which ain’t cheap. If you are on a budget and want more features than the Garmin Connect free service provides, Garmin Connect+ is more affordable than Apple Fitness+, so it pays to shop around and consider your usage requirements. 

edited June 6

tht

Stabitha_Christie said:

Two points: 

1. Several commenters obviously didn't take so much as a passing glance at the study.

2. The article does the study a bit of a disservice. It is a meta-study, so they didn't test anything; they relied on other testing. Nothing wrong with meta-studies, but it is important to know when looking at the results. Of the studies they looked at, almost half used the Series 1 Apple Watch, and when it comes to calorie expenditure, the newest watch was the Series 6. The percentage of error for the Series 1 was 31.5%, and the Series 6 was 26.2%. So the average is 28%, but Apple has improved accuracy over time. Heart rate and steps also have become more accurate. Given that we are currently on the Series 10, we can reasonably assume that it has probably improved from the Series 6. There were some other tidbits that were interesting, like the calorie count is more accurate the more intense the exercise is and running is more accurate than cycling.

Nice post! The conclusion of the paper is:

"Our findings reaffirm the high accuracy of the Apple Watch for HR and step measurements, consistent with previous systematic reviews while highlighting persistent challenges in accurately estimating EE [energy expenditure]. The implementation of the Bland–Altman meta-analysis framework allowed us to not only estimate mean bias but also explore the LoA, revealing significant variability across studies. This variability of LoA underscores the influence of factors such as activity type, intensity, participant characteristics, and device series on measurement accuracy.

Despite these variations, our study indicates that the Apple Watch can provide valid data for HR and steps in both everyday and clinical settings, although further improvements are required for EE measurements. The recent IEC standards advocating a MAPE of less than 10% will likely guide future enhancements in wearable device technologies, ensuring that they meet established accuracy criteria. Our analysis thus contributes to the ongoing discourse on wearable technology’s role in health monitoring, providing critical insights that will aid manufacturers, healthcare professionals, and consumers in leveraging these devices more effectively.

Future research should focus on refining EE measurement techniques in wearable devices by integrating advanced machine learning algorithms and improving software calibration, such as optimizing exercise modes. A critical step involves developing standardized protocols for evaluating EE accuracy across diverse activity types and populations. These protocols should include consistent calibration methods aligned with indirect calorimetry, the gold standard, and controlled testing environments to assess accuracy under diverse conditions (e.g. treadmill-based exercise and free-living activities).

Additionally, to improve the generalizability of results, the study population should be broadened to include a wider range of participant characteristics. For example, considering physiological traits (e.g. wrist size, hair density, skin tone, BMI categories), different age groups (e.g. adolescents, older adults), and individuals with clinical conditions (e.g. cardiovascular disease, diabetes) will provide insights into how these

factors influence EE accuracy. By adopting standardized EE validation protocols and ensuring inclusive participant recruitment, future research can establish more reliable and generalizable accuracy estimates for wearable technologies."

This is a bit different from what the AI is saying. Their conclusion is what you should expect, with at least an admonition that measuring energy expenditure needs better ways to measure it and to broaden the population of the data sets. A lot of these studies essentially boiled down to measuring heart rate, with a presumably more accurate heart rate measurement device, like a chest strap or ECG, plus a good treadmill and accurate weight measurement. Didn't see any that measured oxygen levels or CO2 levels from your breath.

I think it behooves people to keep in their mind that numbers for calories burned are estimates. They are not accurate to 10%, 20%, even 50%, in of themselves, even from the professionals. What they do is provide a target for you, one that you can adjust, and meet everyday. The Watch will recommends higher targets if it sees that you are over your targets a lot. So you must balance how you feel with achieving the goals with how your body feels.

I recently moved from a Series 4 to a Watch Ultra 2 myself. Suffice it to say, it is a huge jump in all categories so far. Performance has been phenomenal. GUI speed high enough and latency low enough to below my thresholds for being annoyed. The https request and data transfer is still the biggest latency in like 99% of apps. The pgysical design of the Ultra is phenomenal. The sapphire glass display cover is so good, so flat, that I think they should use it on iPhone Pros and iPad Pros. The black finish has so far has been durable, even on the sharp corner on the display. The ocean band has been comfortable.

edited June 6

mustseeuhdtv

One thing that is totally not accurate on the Apple Watch is the stand sensor. I have had like 6 different models of the watch and still had issues realizing that I’m standing on all of them.

Here are examples. I have walked back and forth from my house to the mail box at midnight when everything resets. It says walked 280 steps, but didn’t think that I have stood up????

Another time is lunch time. I work remotely still. At noon, I go downstars to prepare my food. Go back up while it is cooking. Come down to eat and then go back upstairs to work…..my watch still didn’t think stood up….