TRIAGE SYSTEM APP: Developed by the U.S. Army MRDC’s BHSAI, APPRAISE-HRI consists of an Android app that collects heart rate and blood pressure data received via Bluetooth from a patient’s external vital-sign monitor and analyzes the pattern in these data to estimate the patient’s potential risk of uncontrolled bleeding. (Photo courtesy of MRDC)
The first triage system ever cleared by the FDA for assessing hemorrhage risk in trauma patients is powered by AI. How did its inventors ensure it was up to the job?
by Paul Lagasse
Hemorrhage, or uncontrolled blood loss, is the leading cause of preventable death among combat casualties, representing approximately 90% of battlefield fatalities. Analysis of data from the conflicts in Iraq and Afghanistan suggests that between 6% and 24% of those fatalities could potentially have survived if their injuries had been more quickly diagnosed and treated. Given that the median survival time from hemorrhage is just two-to-three hours, the window for identification and treatment is narrow.
Combat medics have long known that hemorrhage is associated with increased heart rate and decreased blood pressure. What they lacked, however, was a way to use that data to assess a trauma patient’s hemorrhage risk. A new application developed by the U.S. Army Medical Research and Development Command’s (MRDC) Biotechnology High Performance Computing Software Applications Institute (BHSAI) promises to fill that gap with the aid of artificial intelligence (AI).
The Automated Processing of the Physiological Registry for Assessment of Injury Severity Hemorrhage Risk Index, or APPRAISE-HRI, is a smartphone health app that uses AI to quickly assess patients’ vital-sign data and assign them to one of three categories based on their risk of experiencing life-threatening blood loss. In April, APPRAISE-HRI received the U.S. Food and Drug Administration’s (FDA) first-ever regulatory clearance of an application of its kind, also becoming the first AI-enabled software from the DOD ever cleared by the FDA. The story of how the BHSAI team collected and refined vital-sign data to train the AI algorithm at the core of the APPRAISE-HRI is a fascinating tale of human-computer interaction that has valuable lessons for the future development of AI tools in military medicine.
COLLECTING AND ANALYZING VITAL-SIGN DATA
In 2001, Jaques Reifman, Ph.D., director of BHSAI, first became interested in the problem of whether data collected from the continuous monitoring of vital signs might reveal trends that could be used diagnostically.
“It took time to determine which combination of vital signs we should be using,” said Reifman. “We attempted to develop models with all sorts of potential combinations, which was very time consuming. Ultimately, we used high performance computing to identify systolic and diastolic blood pressure and heart rate as the most useful.”
The challenge Reifman faced was determining how those vital signs correlated with a patient’s hemorrhage state. He proposed training a machine-learning algorithm on data collected from trauma patients to see what trends appeared. With support from the Defense Health Program, the Advanced Medical Technology Initiative, the Henry M. Jackson Foundation for the Advancement of Military Medicine and MRDC’s own Combat Casualty Care Research Program, Reifman and his team undertook three independent studies to collect vital-sign data from adult trauma patients, with associated demographic and clinical data. The BHSAI obtained data from a study conducted by Memorial Hermann Life Flight in Houston, and established collaborative relationships with Boston MedFlight, a hospital-based air medical transportation service and the Massachusetts General Hospital Emergency Department in Boston. Over the course of several years, the team collected diastolic and systolic blood pressure and heart rate data from 2,688 trauma patients, a sufficiently large and varied body of data for them to work with.
Ensuring the data were reliable, consistent and of sufficiently high quality required additional time to develop automated algorithms, including AI algorithms, to flag data that should not be included in the development of the model. For example, the BHSAI team had to filter out-of-range vital-sign data and data artifacts induced by the stress of air-ambulance transport, such as non-physiological changes in heart rates, and confirm that data collected from different models of vital sign monitors were demonstrably consistent. The team also ended up excluding data from 1,029 patients for various reasons. After rigorous analysis and elimination of invalid data, the team ended up with 540 hours of continuous, reliable vital-sign data on which to train the algorithm.
MAJOR ASSESSMENT: Jaques Reifman, Ph.D., far right, director of U.S. Army MRDC’s BHSAI, is with Henry M. Jackson Foundation staff in support of BHSAI who worked with Reifman to develop the APPRAISE-HRI. From left, Francisco Vital-Lopex, Ph.D, research scientist; Andrew Frock, software developer; and Valmik Desai, software development manager. (Photo courtesy of BHSAI)
TRAINING THE ALGORITHM TO ASSIGN HEMORRHAGE RISK
“The training of the algorithm didn’t take a lot of time in comparison to how long it took to collect the data and develop algorithms to automatically flag invalid data,” said Reifman. “We’re not talking about billions of data points and mathematical models that have a trillion parameters, like some of the more recent AI technologies.”
After the BHSAI team had developed and applied the automated algorithms to flag and discard invalid data, they started experimenting with different AI algorithms to map vital signs into risk levels. The researchers conducted a supervised training on the vital-sign data using a multivariate regression algorithm, which computed the hemorrhage risk-level threshold on a scale of 0-to-1, with 0 representing 100% certainty of no hemorrhage risk and 1 representing 100% certainty that hemorrhage is likely to be present. The algorithm sorted the patients into one of three classes of hemorrhage risk: low (hemorrhage risk (HR) level I, representing a 2-fold lower risk than the average in the test population), average (HR level II, representing the prevalence of hemorrhage in the test population) and high (HR level III, representing a 2-fold higher risk). The results of the analysis showed that the algorithm was capable of stratifying risk to a high degree of reliability with no overlap. When they compared the algorithm’s risk assessments longitudinally over time, in more than 70% of the comparisons the risk level remained unchanged.
In its final form, the APPRAISE-HRI algorithm consists of a smartphone app that receives heart rate and blood pressure data via Bluetooth from a patient’s vital-sign monitor and runs it through three software modules. The first module conducts pre-processing of the patient’s heart rate and systolic and diastolic blood pressures. This involves trimming out unreliable physiological data, making sure that the data used to analyze the state of the patient is valid, timely and of sufficient quantity. The resulting processed heart rate and pulse pressure data are then passed along to the second module, which is the AI portion of the algorithm. It performs multivariate linear regression on the data to assess the likelihood of hemorrhage on a 0-to-1 scale. This assessment is then passed along to the third module, which analyzes the outputs of the AI model and places the results into one of the three risk levels, which is then provided to the first responder for action.
ANSWERING THE FDA’S QUESTIONS
When MRDC develops and tests a military medical product to the point of readiness, the command’s Medical Technology Transfer Office seeks partnerships with commercial manufacturers to build and market the device. If the device is regulated by the FDA, it must first be granted a clearance, or an approval to proceed to market. To make the APPRAISE-HRI more appealing to potential commercialization partners, Reifman recommended that BHSAI obtain the FDA clearance rather than passing that requirement along to the manufacturer.
Prior to applying for clearance review, while the device was in the final stages of testing, BHSAI worked with MRDC’s Office of Regulated Activities to prepare a pre-submission to the FDA to help anticipate and address issues of concern. The FDA responded with 26 questions related to the rationale for selecting the three vital signs, model development and the need for new, never-before-used data sets to independently validate the APPRAISE-HRI. The BHSAI team answered all of them to the FDA’s satisfaction, which Reifman believes accelerated the subsequent final clearance review process.
The FDA’s clearance review process takes the form of a risk and benefit analysis, in this case comparing the APPRAISE-HRI’s potential for improved quality of care, ease of use and ability to work with FDA-cleared vital-sign monitors against the potential for misinterpretation of the results, algorithm error, data corruption and other concerns. BHSAI used funding from MRDC’s Medical Materiel Development Activity to conduct another round of validation on the APPRAISE-HRI algorithm, which was performed independently and blindly on vital-sign data from nearly 6,000 patients collected from nine sites, including in-hospital and pre-hospital data. The results of this independent analysis corroborated the algorithm’s performance.
In issuing a clearance for the APPRAISE-HRI, the FDA stressed that the device is intended to provide military healthcare providers with “situational awareness” about potentially hemorrhagic patients, not to be used to “diagnose or direct treatment.” The FDA concluded that APPRAISE-HRI is “a useful tool to aid in discriminating hemorrhage risk in the trauma population.”
CONCLUSION
Reifman says the experience of developing the APPRAISE-HRI algorithm and working with the FDA to approve the device provided valuable lessons for him and his team.
“We have high standards of scientific rigor at BHSAI, but working with the FDA showed me that there are things that I could be even more rigorous about,” Reifman said. “Our focus is on developing technologies. But the FDA focuses on risk and how to mitigate it. That’s not how scientists usually think about technology. The more I worked with them, the more I understood that their approach is just as scientifically rigorous as ours. I think that alerted me to the understanding that I still have room to grow in terms of my own definition of scientific rigor, as well as how we apply it to the things we do in our organization.”
That broader perspective is already being distilled into BHSAI’s current practices, Reifman added, and because many of the scientists there are still in the early stages of their careers, he is hopeful that it will permeate BHSAI’s institutional culture.
“We have a lot of software development projects, and I encouraged my team to think about how we can incorporate these concepts and processes when we’re planning those projects,” Reifman said. “For example, now we think about cybersecurity from the start, which is something we wouldn’t normally do as part of pure research. All those things that we didn’t pay attention to because we didn’t think were important, we are now implementing throughout the organization whenever possible to increase the scientific rigor that we apply to every single project.”
For more information about the Biotechnology High Performance Computing Software Applications Institute, go to https://bhsai.org.
PAUL LAGASSE is a public affairs writer with USAMRDC. He has an M.A. in history and an M.L.S. in archival studies from the University of Maryland, and a B.A. in history from Regis University. Before working in public affairs for the Army and Navy, he was a newspaper reporter and a freelance writer and editor.
