Is My Heart Rate Tracker Accurate?

Image of man jogging with many tracking devices and multiple parameters being tracked

Is My Heart Rate Tracker Accurate?

On January 5, 2016, wearable maker Fitbit Inc. was hit with a class-action lawsuit regarding the accuracy of two of their heart rate trackers - the Fitbit Charge Heart Rate (HR) and the Fitbit Surge. If you are a proud owner of one of the over 20 million devices (not all of which are heart rate trackers) that Fitbit has sold to date, this news will probably make you wonder if your heart rate tracker is telling you the truth about your heart rate. As the lawsuit against Fitbit seeks injunctive relief as well as actual damages payments to compensate users for their "economic injuries" from Fitbit's heart rate claims, this could very well be a billion-dollar question for the company.  

George Bernard Shaw once said that science never solves a problem without creating ten more. Measuring and reporting on the accuracy of a bio-sensor like a continuous heart rate tracker is a pretty good example of that, in my opinion. Here’s why: Let’s say you plan to take the first step towards challenging your tracker’s veracity which is to collect some concurrent heart rate readings from your device and another “gold standard” device. The first problem you will encounter is choosing a gold standard device fit for the job. In the case of continuous heart rate tracking there is a wide range of options you can choose from. For example, you could spend a few hundred dollars on buying another consumer grade tracker like the Apple Watch or find a physician friend who will give you access to a Food and Drug Administration (FDA) approved electrocardiogram (EKG) machine that typically goes for more than two thousand dollars. You could go hi-tech and get heart rate sensing shirt or go the old school route and ask a friend to measure your radial pulse using a simple wrist watch. Amongst other technical considerations like the device’s sampling rate, this choice will also depend on who you are trying to convince of the tracker’s accuracy. Your fitness coach and primary care doc will likely not require as much proof as the FDA or a jury of your peers, like in the case of the lawsuit against Fitbit.  

The next problem is selecting a statistical test for accuracy that you can use to analyze the data you have collected from the two devices. Once again, you will find a variety of options, each with their own pros and cons. I will list a few here. Spoiler alert - the last one is my favorite.

  1. Paired T Tests

Paired t tests are easy to conduct and they, in this case, will compare the means of the readings from the two devices. This method could be problematic because the means of the heart rate readings that you have collected can be the same even if the readings themselves are vastly different. You can see this in the table below where the readings don’t overlap but the means are identical.  

Text Box: Means are same even though there is a wide variation in some heart rate pairs

 

Heart rate from your heart rate tracker

Heart rate from gold standard device

Reading 1

54

59

Reading 2

63

64

Reading 3

76

79

Reading 4

79

81

Reading 5

58

47

MEAN

66

66

Means are same even though there is a wide variation in some heart rate pairs

  1. Correlation Coefficient

The correlation coefficient measures linear agreement--whether the readings go up-and-down together. This has a similar limitation to the paired t test where the coefficient can be close to 1 (which is good) even when the readings from the two devices have large differences. You can see this in the table below where the readings are different but the correlation coefficient is 1.  

 

Heart rate from your heart rate tracker

Heart rate from gold standard device

Reading 1

60

64

Reading 2

62

66

Reading 3

76

80

Reading 4

79

83

Reading 5

56

60

There is a negative bias in the tracker being tested but correlation coefficient is 1

  1. Bland Altman Analysis 

My favorite test for this problem was published in the Lancet by JM Bland and DG Altman about 25 years before the first wearable heart rate tracker hit the market. There are a couple of reasons why I like this method. First of all, unlike the tests mentioned above that test for correlation, Bland Altman Analysis tests for agreement. As we have seen from the tables above the two are not always one and the same. Another reason I am a fan is because even before conducting this test, if you are doing it right, you have to answer the question “how much difference is clinically acceptable”? For example, if your doctor is using this test to compare heart rate outputs from two monitors, he or she will have to think about the heart rate value for which your medication may have to be adjusted before deciding how much variation is acceptable. Similarly, your fitness trainer will have to wonder if your training regimen will change if the readings were off by 10 or 50 beats per minute.  Finally, the Bland Altman technique accounts for the fact that a gold standard devices also have some measurement error which tends to even the playing field while comparing devices. Here is how you can conduct this test in excel.

  • Enter the values from your tracker in column A
  • Enter the corresponding values from a gold standard device in column B
  • Calculate the means of the two values in column C. You can do so by typing the formula =AVERAGE(A1,B1) in cell C1 and dragging it down the column to your last measurement
  • Calculate the differences between the two heart rate values in column D. You can do so by typing the formula =A1-B1 in cell D1 and dragging it down the column to your last measurement
  • Your table at this point should look something like this:

 

A

B

C

D

1

54

59

56.5

-5

2

63

64

63.5

-1

3

76

79

77.5

-3

4

79

81

80

-2

5

58

47

52.5

11

  • Now select all the values in column C and D and click in Insert > Chart > XY Scatter
  • Your chart, if you followed the instructions correctly, should look something like this:

Graph for Bland Altman

Correctly interpreting this chart to answer the question about your heart rate tracker’s accuracy is the subject of a follow up blog post, one that I will create if I get more than ten comments on this one, but broadly speaking the closer your data points are to the X axis on this plot the more likely that your heart rate tracker is telling you the truth. Our vision for this blog is for it to be a two-way communication channel between our team of multidisciplinary researchers and those interested in any aspect of technology oriented research so please be generous with your feedback on this post and feel free to reach out to us with new project ideas using our Contact Us form.  

Author: Jiten Chhabra, MD MS 
Graphic: Amy Lambeth

Add new comment

CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.

Contact IMTC

CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.