Google has unveiled SensorLM, a set of foundation models that analyzes multimodal wearable sensor signals from devices like smartwatches and fitness trackers to generate insights into one's health and activities.
"While we can easily see what our body is doing (e.g., a heart rate of 150 bpm), the crucial context of why (say, "a brisk uphill run" vs. "a stressful public speaking event") is often missing," Google said in a statement.
The tech giant said SensorLM was trained on 59.7 million hours of multimodal sensory data from 103,643 people across 127 countries, and was collected between March 1 and May 1, 2024, from Fitbit or Pixel Watch devices with users' consent.
The model's creators developed a pipeline that "automatically generates descriptive text captions by calculating statistics, identifying trends, and describing events from the sensor data itself."
SensorLM uses contrastive learning, which teaches the model to match sensor data with the correct text descriptions, and generative pre-training, which helps it learn to write those descriptions on its own.
Google said SensorLM can identify human activity, such as going on a long walk or performing cardio versus strength training.
"Given only the high-dimensional sensor signals from a wearable device, SensorLM can produce hierarchical and contextually relevant captions. Experimental results indicate that these generated captions were more coherent and factually correct than those produced by powerful non-specialist LLMs," Google said.
The tech giant said the models were shown to consistently improve with more data.
"Looking forward, we plan to scale pre-training data into new domains, including metabolic health and detailed sleep analysis, to address the messy reality of consumer health devices," Google added.
"We envision SensorLM leading to a future generation of digital health coaches, clinical monitoring tools and personal wellness applications that can offer advice through natural language query, interaction and generation."
THE LARGER TREND
With Fitbit and Pixel Watch under its umbrella, Google has increasingly been focused on using AI within its wearables for health-related purposes.
Earlier this year, the company received FDA clearance for its loss of pulse detection feature available on Pixel Watch 3, which it initially announced during its Made by Google event in 2024.
The feature uses infrared and red lights, plus motion sensors, to detect a loss of pulse. An AI algorithm confirms the event and triggers a countdown. If there's no response, the watch calls emergency services and shares the user’s status and location.
Google has also had a long-term partnership with Samsung, which earlier this month, during its Galaxy Unpacked event, announced the release of its latest Galaxy Watch8 Series with new health features.
Samsung's new lineup includes sleep apnea detection, Bedtime Guidance, vascular stress monitoring and the first smartwatch Antioxidant Index.
