Updated with a statement from Glow
Wired Magazine splashed some cold water on fertility app Glow's assertion that a new study shows their users have a 40 percent higher likelihood of getting pregnant than those who don't use the app.
The problem, the magazine asserted after speaking with multiple fertility scientists not associated with the company, is based on a classic scientific aphorism: correlation does not imply causation.
The study used Glow data from 108,469 users and compared frequent users of the app to infrequent users. They found that the frequent users were more likely to get pregnant and conceived faster than the infrequent ones. But the study didn't control for any other variables that could effect conception. And there's reason to believe that users who were sufficiently motivated to engage more with the app might also be more motivated to have sex more often, to track their cycles more meticulously, or to engage in other behaviors to improve chances of conception.
"It may just be measuring the motivation to conceive, rather than helping to conceive,” Kutluk Oktay, a doctor and reproductive endocrinologist at New York Medical College, observed.
That's not to say that Glow doesn't work, just that the data the company is proffering doesn't necessarily prove that it does. Wired hopes, and we do too, that Glow will respond with new, more rigorous studies.
Glow's not the lone culprit in this sort of fuzzy science; it's something that consumers always have to be on guard for at the intersection of consumer-facing apps and healthcare. And Wired has proved itself pretty committed to the watchdog role. Earlier this year, when a widely-circulated study derided the accuracy of wearable fitness trackers, Wired not only critiqued the study design but launched their own informal study finding that the trackers in question were more accurate than phones.
When we wrote about that, I noted that sometimes formal studies, especially rigorous, peer-reviewed ones can seem too slow to do justice to consumer products. The lesson here is that even when consumer companies commit to studying their own products, they need to make sure that data is rigorous, and really shows what it purports to show.
Update: Glow wrote to MobiHealthNews with some clarifications and a statement suggesting that Wired's story was "unbalanced", although they don't dispute all of the particulars.
"We are shocked by this unbalanced story; we stand by the paper and the effectiveness of the Glow app in helping women better conceive," the company wrote. "Wired spoke to Glow’s medical advisor and author of the published report by the prominent American Society of Reproductive Medicine yet never quoted him, instead choosing to quote doctors who have not even reviewed the report. We’ve offered more detail about the size and scope of the study and clarity on how the data sets were analyzed, neither of which Wired included.
Glow is one of the few women’s health companies to share its anonymized data with the medical community in hopes of learning more about reproductive health. There hasn’t been a large scale study conducted in women’s reproductive health in more than 50 years and we’re proud to share back vital information with both the medical community and our users."
Glow admitted that the study isn't a randomized control trial, but doesn't see that as a problem since they were able to work with so much more data than a traditional RCT, and possibly reach insights they couldn't get with an RCT. They did control for some factors, they said, in that they only included women trying to conceive and excluded those who were getting fertility treatments. In addition, the conclusion that women who track their fertility more closely conceive more quickly has been observed in other studies previously.
"We will continue to address these important women's health questions as our data set grows," the company wrote. "The more data we have, the more nuanced studies we can produce. There's still so much to learn."
