Sleep wearables: how do they stack up against a chest-strap reference?

1. Study design

This is an observational, repeated-measures device validation study comparing overnight measurements of resting heart rate (RHR, in beats per minute) and heart-rate variability (RMSSD, in milliseconds) between consumer wearables and a research-grade reference. Each subject wore the Polar H10 chest-strap (the criterion device) simultaneously with one or more wearables across multiple sleep periods. Each night for which both the criterion and a given wearable produced a valid overnight value contributes one paired observation.

Reference device selection

The Polar H10 single-lead ECG chest strap is the de-facto reference for HRV research because (a) it samples the electrocardiogram directly rather than inferring beats from photoplethysmography (PPG), (b) its R-R interval detection has been independently validated against clinical Holter recordings, and (c) it is the dominant criterion device cited in published wearable-validation literature, providing comparability across studies.

Reference window for Polar wearables

Polar's own consumer wearables (Polar 360, Polar Grit, Polar Loop) compute overnight HRV using a fixed 4-minute orthostatic-window algorithm rather than averaging across the entire sleep period. Comparing those devices to the full overnight H10 mean would conflate algorithmic design with measurement error. To isolate measurement agreement, those three wearables are compared instead to POLAR H10 RMSSD FIRST 4 and POLAR H10 RHR FIRST 4 — the H10's value computed over the same first-4 window the Polar wearables use. All other wearables are compared to the standard overnight H10 reference. This pairing rule is fixed before analysis and reported transparently next to each device.

2. Data handling and inclusion

Each device-night pair is included if and only if both the wearable and the appropriate reference produced a numeric value for that night and the reference value is strictly positive (a non-positive RHR or RMSSD is biologically implausible and would create an undefined percent-error). No outlier removal is applied; we deliberately keep extreme observations because Bland-Altman analysis and robust error metrics (MAPE-median) are designed to handle them and removing them would inflate apparent agreement.

Devices with fewer than five paired observations are excluded entirely (insufficient data to compute meaningful aggregate metrics). Devices with five to fourteen paired observations are reported but flagged in the user interface, because percent-error and limits-of-agreement estimates with such small samples have wide confidence intervals.

3. Primary metric — Mean Absolute Percent Error (MAPE)

MAPE is the headline ranking metric in this report because it is unit-free (allowing direct comparison across HRV and RHR), aligns with how the practitioner survey thresholds were specified, and is interpretable to a non-statistical audience.

where x_i is the reference value on night i, y_i is the wearable value, and n is the number of paired nights. We additionally report the median absolute percent error (MdAPE), which is robust to a small number of extreme nights. Where MdAPE is materially lower than MAPE the device is well-calibrated on most nights but produces occasional gross outliers; the inverse (MdAPE ≈ MAPE) indicates a more consistently shifted device.

4. Absolute and squared error metrics

Mean Absolute Error (MAE)

MAE retains the original measurement units (ms for HRV, BPM for RHR), making it directly interpretable when MAPE feels abstract.

Root Mean Squared Error (RMSE)

RMSE penalises large errors more heavily than MAE; the gap RMSE − MAE is therefore a useful diagnostic of error distribution. A device with RMSE substantially larger than MAE has a long-tail error problem.

5. Systematic bias (mean signed error)

Unlike MAE/RMSE/MAPE, the bias is signed: positive values indicate the wearable systematically over-reads the reference, negative values indicate systematic under-reading. A device may have small bias yet large MAE if its errors cancel out — bias and MAE together characterise both directional drift and overall noise.

6. Linear association — Pearson r

Pearson's product-moment correlation coefficient quantifies the linear association between the wearable and reference series. It is reported but explicitly not used as a primary agreement measure, because two series can be perfectly correlated (r = 1.0) yet disagree wildly in magnitude — the wearable could read systematically double the reference and still produce r = 1.0. Pearson r is included here for completeness and to permit direct comparison with prior validation papers that (questionably) treat it as their headline measure.

7. Agreement metrics

Lin's Concordance Correlation Coefficient (CCC)

CCC modifies Pearson r by penalising any departure of the regression line from the 45° identity line — that is, it requires both linear association (correlation) and identical means and variances (no bias, no scale shift) to reach 1.0. CCC is interpreted on the same 0–1 scale as r but is a strictly more demanding test of agreement.

Intraclass Correlation Coefficient — ICC(A,1)

We report ICC(A,1) using the McGraw & Wong notation, equivalent to ICC(2,1) in the Shrout & Fleiss (1979) scheme: a two-way random-effects model with absolute agreement, single-rater (single-measure) form. This is the appropriate ICC variant when (a) both raters — here, the wearable and the H10 — are treated as random effects, and (b) the question of interest is whether they produce identical absolute values, not merely consistent rankings.

where MS_R is the between-subject (between-night) mean square, MS_C is the between-rater mean square, MS_E is the residual mean square, n is the number of nights, and k = 2 is the number of raters per night. Following McBride (2005) strength-of-agreement criteria for CCC: > 0.99 almost perfect, 0.95–0.99 substantial, 0.90–0.95 moderate, ≤ 0.90 poor. Note these thresholds are considerably more demanding than the Koo & Li (2016) ICC benchmarks — by the McBride standard, no device in this study achieves better than substantial agreement.

8. Bland-Altman analysis

The Bland-Altman framework is the canonical tool for quantifying agreement between two measurement methods. We report the bias (mean signed difference) and the 95% Limits of Agreement:

The 95% LoA define the interval within which 95% of the observed device-reference differences fall, assuming the differences are approximately normally distributed (a reasonable assumption that we verify visually on the per-device diagnostic plots). The width of the LoA — not the bias — is the practical answer to the question "how much can I trust a single reading from this device?". A narrow LoA indicates a device whose nightly reading is reliably close to truth; a wide LoA indicates that even with no systematic bias, any individual night's value could be far from the reference.

9. Practical-tolerance metrics

For interpretability we additionally report the percentage of nights on which the absolute device-reference error is within fixed tolerances (3, 5, and 10 units, where the unit is ms for HRV or BPM for RHR). These are the empirical equivalents of asking "how often does this device get within X of the reference?", and translate the headline MAPE into a more concrete frequency.

10. Tiering relative to practitioner thresholds

Devices are classified into four tiers based on MAPE relative to the threshold endorsed by practitioners in Vol 4:

• tier 1 — MAPE ≤ threshold (meets the practitioner standard).
• tier 2 — threshold < MAPE ≤ 1.5 × threshold (close; may be acceptable for lower-stakes monitoring).
• tier 3 — 1.5 × threshold < MAPE ≤ 2 × threshold.
• tier 4 — MAPE > 2 × threshold (substantially above what practitioners said they could tolerate).

The thresholds themselves derive from chi-square goodness-of-fit testing and modal selection across a 210-respondent practitioner survey, with effect sizes (Cramér's V) ranging from moderate to strong; full details in the Vol 4 report.

11. Important limitations

• Repeated measures within subjects. A small number of subjects contribute many nights each, so per-device samples are not statistically independent. The aggregate metrics here describe device-level performance averaged across the available nights; they should not be interpreted as population estimates with the precision a fully independent sample of equal size would afford. A formal extension would compute per-subject MAPE and then aggregate across subjects (or fit a mixed-effects model).
• Subject heterogeneity. Per-subject distributions of HRV and RHR vary; devices with disproportionate observations from a single subject inherit that subject's physiology in their estimates.
• Reference is not ground truth. The Polar H10 is the best available consumer-accessible reference but is not a 12-lead clinical ECG; it has its own measurement noise floor. Reported errors are therefore device-vs-reference disagreement, not absolute device error.
• Algorithm versioning. Wearable manufacturers ship algorithm updates between firmware revisions. Results here reflect the firmware in use during the Nov 2024 – Nov 2025 measurement window.
• Sleep-stage confounding. Some devices report HRV computed over specific sleep stages (e.g. deep sleep only) rather than the full overnight period. Where this is suspected, comparison to the overnight H10 will systematically inflate apparent error. The FIRST-4 reference partially addresses this for the three Polar wearables; for other devices the user should consult the manufacturer's algorithm documentation when interpreting device-specific results.

12. Software and reproducibility

All computations performed in Python 3.12 using NumPy 2.x, pandas 2.x, and SciPy 1.x (scipy.stats.pearsonr). Lin's CCC and ICC(A,1) implemented from first principles per the formulas above; verified against worked examples from the original Lin (1989) and McGraw & Wong (1996) papers. Source code and the underlying paired data file are bundled with this report.

13. References

• Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;327(8476):307-310.
• Lin LI. A concordance correlation coefficient to evaluate reproducibility. Biometrics. 1989;45(1):255-268.
• Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin. 1979;86(2):420-428.
• McGraw KO, Wong SP. Forming inferences about some intraclass correlation coefficients. Psychological Methods. 1996;1(1):30-46.
• Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine. 2016;15(2):155-163.