Paper Explained: WearableQA — Testing Whether LLMs Can Reason Over 500 Days of a Real User's Wearable Data

4,084 ten-option questions built from 200 real users' daily wearable records (up to 500 days each) plus blood panels. Separating 'answer from knowledge' from 'read it off the numbers' splits 14 models between 19.6% and 72.9% accuracy, and simply giving the model Python adds 20 points. A ground-up walkthrough of how the questions are built and what the results actually say.

WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data

What the paper is asking

A smartwatch emits numbers every day: resting heart rate, sleep, steps, HRV (heart-rate variability). Lining up hundreds of days of those numbers and deciding "how has this person been doing lately" or "which of these signals move together" is a different skill from knowing medicine. An AI that has the knowledge but cannot pull facts out of a noisy sequence is not one you would trust as a health assistant.

The paper is "WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data" (arXiv:2609.05405), from researchers at Meta and KAIST.

Here is the abstract in plain terms. Wearables now record physiological and behavioral signals continuously, yet existing benchmarks rarely test whether AI can reason over a real user's longitudinal record. The authors introduce WearableQA: 4,084 ten-option multiple-choice questions built from the wearable time series (up to 500 days of daily measurements per person), blood biomarkers, and demographics of 200 real users, preserving authentic distributions including device noise and inter-individual variability. They define 16 question types along two axes — data versus health reasoning, and single- versus cross-signal reasoning — and build questions at scale with a "dual-grounding" framework combining literature-grounded physiological findings with statistically validated population-grounded patterns. Across 14 proprietary and open-source LLMs, accuracy ranges from 19.6% to 72.9% against a 10% chance baseline, and most models stay below 60%.

Analogy: reading the textbook vs. reading the diary

Picture someone who has memorized a health textbook. Ask "what does an elevated resting heart rate suggest?" and they answer instantly. Now hand them 500 days of a handwritten diary and ask "on which days was this person's heart rate clearly above their normal, and how many days did it take to come back?" That requires flipping through pages and counting. Textbook knowledge and extracting facts from a diary are separate skills.

The paper splits "data reasoning" from "health reasoning" because if you measure them together, you cannot tell which one is failing (§1). Existing medical benchmarks such as MedQA are textbook-style, and existing time-series benchmarks mostly use synthetic signals (§2). Combining real users' long records with blood work, and measuring both skills, is where this paper positions itself.

Intuition: a 2×2 grid that shows where a model trips

Questions are organized along two primary axes (§3.2).

• Reasoning type: data reasoning (trends, anomalies, excursion counts, recovery times, relationships between signals — the answer is determined by the numbers themselves) versus health reasoning (prognostic framing, recommendations, differential reasoning, phenotyping — physiological interpretation layered on top)
• Signal complexity: does one metric suffice, or must two or more be integrated?

There are roughly 2.7k data-reasoning and 1.4k health-reasoning questions; roughly 1.7k single-signal and 2.4k cross-signal. An orthogonal third label records whether each answer is literature-grounded or population-grounded. Every question is 10-option, with the correct position balanced across the benchmark.

Each user contributes 16 daily metrics (resting heart rate, HRV, VO₂max; steps, active and BMR calories, exercise count, duration and average exercise heart rate, METs; sleep duration, efficiency, deep and REM percentages, bedtime regularity; average stress), a 17-biomarker blood panel (e.g. insulin, HbA1c), and age, sex, BMI and ethnicity (§3.1). For "how do I compare to my peers" questions, cohort percentiles for five core metrics are provided, computed on a larger separate cohort.

Mechanism: the answer is computed, not asserted

The biggest danger when mass-producing health questions is a plausible-looking answer with no real basis. The paper guards against this with dual grounding (§3.3).

Literature-grounded questions start from peer-reviewed consumer-wearable studies, keep only observational ones where both exposure and outcome can be computed on the cohort, verify each against its primary record (PubMed, cross-checked by DOI/PMID/PMCID), and end up with 11 anchor papers. Each relationship needs at least two independent, same-direction studies and is recorded as a card with verbatim thresholds, cohort details, and what the source does not establish. Notably, study-specific cutoffs are deliberately not imported, because they are cohort-dependent and rarely transfer.

Population-grounded questions mine relationships directly from the cohort over a 28-day window; a candidate becomes a question only if it passes three statistical gates.

• Effect size: correlation magnitude $|\rho| \geq 0.5$ with consistent sign in both halves of the window; for comparative questions the winner must beat the runner-up by at least 0.15
• Robustness: the outcome must persist under bootstrap resampling of days (in at least 80% of 30 resamples), under leave-one-day-out, and across baseline-window choices; near-boundary cases inside a dead-band are rejected
• Authenticity: a cross-user null test recomputes each statistic on deliberately mismatched user pairings, and the ratio of null to observed prevalence serves as an empirical false-discovery rate; only patterns with $\mathrm{FDR} \leq 0.20$ survive

In words: if a pattern shows up about as often when you scramble people's data as when you don't, it is coincidence rather than something about that person, so it is thrown out. A ratio of 0.2 or less means at least four out of five retained patterns are genuine.

The correlation in the effect-size gate is nothing more than the cosine between two mean-centered time-series vectors. Rotate the two vectors in the figure below and watch how the angle maps to the dot product and cosine.