I have been recently working on design of a medical device that uses a pulse oximeter. It is a small device that attaches to your fingertip to measure the oxygen saturation level in the blood and the pulse rate. This type of simple device is extensively used to help with medical decision making. There are many manufacturers and models available. A quick search of oximeters on the web brings up the latest FDA communications about oximeter accuracy and limitations.1 A 2020 correspondence article in the New England Journal of Medicine suggests that oximeters “may be less accurate with people with dark skin pigmentation.” 2 The original correspondence states that “in two large cohorts, Black patients had nearly three times the frequency of occult hypoxemia (low level of oxygen) that was not detected by pulse oximetry as White patients.” This, in turn, can increase the risk of hypoxemia among patients with darker skins.
This observation highlights the need for real-world data and evidence that includes a true reflection of the diversity of the population to be treated.
Creating Real-World Data and Evidence
Real-world data can be generated and collected by experiencing the medical device in real settings and with diversity within its targeted population. The real-world data collection effort may continue throughout the device lifecycle to ensure safety and efficacy. This is in contrast with a typical clinical trial that is time bound and has a test population limited in number and type. Because of these limitations, standard clinical trials could be biased, while a real-world approach toward data collection could help us avoid those biases.
Making It Fair
Racial biases exist in many prediction algorithms. Take, for instance, a decision-making software used to allocate healthcare to patients. 3 The software identifies which patients need extra medical care by predicting the future healthcare costs for them. It turns out that cost is not a race-agnostic metric. As a result, the software referred fewer number of Black patients to the extensive healthcare program, although they were equally sick as White patients. One study suggests that, “Remedying this disparity would increase the percentage of Black patients receiving additional help from 17.7 to 46.5%.” 4
With an increased supply and demand of artificial intelligence (AI)-powered medical technologies, there is a need to make our prediction algorithms fair. To achieve this goal we need to feed our algorithms with diverse data sets. It seems like an ambitious goal to collect data from tens and thousands of individuals, hospitals, physicians, and so on. But it also seems like a promising path toward building technologies that leave no one behind.
Where Does FDA Stand on This?
Regulatory bodies are becoming more interested in real-world data and evidence. For instance, the EU Medical Device Regulation (MDR) requires a lifecycle approach and continuous data collection for post-marketing surveillance. In 2017, FDA relied on real-world data to approve a new indication for SAPIEN 3 heart valve. 5 That was just a beginning. Looking forward, in 2021 FDA has proposed an action plan for AI/machine learning (ML)–based software as a medical device. The action plan embraces ways to eliminate algorithm bias and to collect real-world data in support of risk benefit assessment of device submissions. Most recently, an FDA report provides several examples where real-world data and evidence from registries, administrative claims, and medical records have been used to support regulatory decision making. 6
Final Note Highlighting the Need for Real-World Medical Device Data
If there is one lesson we can learn from the oximeter cautions or disparities in prediction algorithms, it is that real-world data and evidence can help us avoid racial biases in medical technologies. It does so by increasing the size and diversity in testing populations. However, there are challenges in creating real-world evidence. For instance, how can we collect good quality and reliable data? Nevertheless, medical device manufacturers need to adapt to these regulatory changes and market needs. This also highlights the need for collaborative business models and development strategies that could help us collect real-world data and then use it as a feedback throughout the product lifecycle.
What is your strategy toward creating real-world data and evidence?
1. “Pulse Oximeter Accuracy and Limitations: FDA Safety Communication,” FDA, February 19, 2021, https://www.fda.gov/medical-devices/safety-communications/pulse-oximeter-accuracy-and-limitations-fda-safety-communication
2. “Racial Bias in Pulse Oximetry Measurement,” New England Journal of Medicine, December 17, 2020, Letter to the Editor, https://www.nejm.org/doi/full/10.1056/NEJMc2029240
3. “Millions of black people affected by racial bias in health-care algorithms,” Ledford, Heidi, Nature, October 24, 2019, https://www.nature.com/articles/d41586-019-03228-6
4. “Dissecting racial bias in an algorithm used to manage the health of populations,” Obermeyer, Ziad et al., Science, Oct 2019, https://pubmed.ncbi.nlm.nih.gov/31649194/
5. “FDA Used Real-World Evidence in Heart Valve Approval,” Regulatory Affairs Professional Society, https://www.raps.org/regulatory-focus%E2%84%A2/news-articles/2017/6/fda-used-real-world-evidence-in-heart-valve-approval
6. “Examples of Real-World Evidence (RWE) Used in Medical Device Regulatory Decisions,” FDA, https://www.fda.gov/media/146258/download