#NephJC Chat
Tuesday January 26 9 pm Eastern
Wednesday January 27 9 pm IST
Wednesday January 27 9 pm GMT
BMJ. 2021 Jan 18;372:m4786. doi: 10.1136/bmj.m4786.
Electronic health record alerts for acute kidney injury: multicenter, randomized clinical trial
F Perry Wilson, Melissa Martin, Yu Yamamoto, Caitlin Partridge, Erica Moreira, Tanima Arora, Aditya Biswas, Harold Feldman, Amit X Garg, Jason H Greenberg, Monique Hinchcliff, Stephen Latham, Fan Li, Haiqun Lin, Sherry G Mansour, Dennis G Moledina, Paul M Palevsky, Chirag R Parikh, Michael Simonov, Jeffrey Testani, Ugochukwu Ugwuowo
PMID: 33461986
Introduction
From a certain point of view, technology and innovation stem from the human desire to live long, healthy and comfortable lives. It is only natural that technology should find its way into the science of medicine, the goal being to improve patient care. The advent of the computer age and the internet revolution have brought about remarkable transformations in practice. Technology has been used not just to aid in diagnosis and treatment but in creating tools to prevent disease.
The purpose of artificial intelligence is to fill the lacunae in human intelligence and indeed, to overcome human limitations. In a previous #NephJC discussion, we debated the use of artificial intelligence in predicting the likelihood of acute kidney injury (AKI). This week's #NephJC discussion focuses on another prospective use of artificial intelligence - alerting health care providers of the onset of AKI.
AKI is a routinely encountered entity (Lamiere et al, Lancet 2013), associated with significant morbidity and mortality (Hoste et al, Nature Rev Nephrol, 2018). Despite universal recommendations, there is notable variation in the quality of care of patients with AKI. Since AKI is often asymptomatic at onset, early identification can be entirely missed (eg Lewington et al, Kidney Int 2013). All of us would like to nip AKI in the bud, before it can cause any long-term damage.
In 1991, the Institute of Medicine laid down attributes for electronic medical records (EMRs) that focused on patient care. Working on these, more and more hospitals incorporated EMRs (or Electronic health records - EHRs) into their health care systems to obtain reliable, easy to access, and easy to interpret data. The focus at present is on utilising this technology in early identification of complications.
Imagine a scenario where a patient's serum creatinine has gone up from 0.5 mg/dL on admission to 1 mg/dL on day 3 of admission. While it is clear that the patient has developed AKI as per KDIGO criteria, it is quite possible that this increasing trend may be missed by the treating physician. We are after all humans and likely to err. Would it not be useful if you could get an alert from the EHR, bringing your attention to this event? It would prompt us to focus on the volume status, the nephrotoxicity of ongoing medication and other reversible causes of AKI. But on the other hand, would it lead to wrong decisions? For example, it might prompt overhydration. Moreover, too many alerts and the health care provider may simply consider it a modern day boy who cried wolf! How far can we let artificial intelligence guide us? The trial under study aimed to answer this question.
The principal investigator, F. Perry Wilson, had previously carried out a single blind, parallel group randomised control trial (RCT) (Wilson et al, Lancet, 2015; discussed on NephJC here). to study the role of automated electronic alerts in improving outcomes of patients with AKI. This study noted no difference in terms of dialysis initiation or patient mortality. However, another trial (Selby et al, JASN 2019) did report some benefit in mortality, dialysis initiation and hospital stay. In this multicenter, pragmatic, stepped wedge cluster RCT, an AKI detection and alerting system was used as part of a multifaceted intervention. They reported reduction in hospital stay and improved quality of care, although there was no reduction in the 30 day mortality.
In summary, AKI is bad, and being a simple number, is easy to diagnose and alert. So now, to alert or not to alert - that is the question. In an attempt to answer this question, the ELAIA-1 trial was carried out as discussed below.
The Study
The Trial
Electronic health record alerts for acute kidney injury (ELAIA-1)
Funding
Grants from the National Institute of Health (NIH)
Objective
To determine whether electronic health record alerts for acute kidney injury would improve patient outcomes of mortality, dialysis, and progression of acute kidney injury
Methods
Design
Double blinded, multicenter, parallel, randomized controlled trial
Setting
Six hospitals (four teaching and two non-teaching) in the Yale New Haven Health System in Connecticut and Rhode Island, USA
Patients
Inclusion Criteria
Aged >=18 years
AKI, as defined by KDIGO criteria as an increase in serum creatinine of 0.3 mg/dL (26.5 μmol/L) within 48 hours or 1.5 times the lowest measured creatinine within the previous seven days of admission
Urine output was not used to define AKI
Exclusion Criteria
History of End Stage Kidney Disease
History of dialysis in the past year
History of hospital creatinine more than 4 mg/dL (354 μmol/L)
Admission prior to inception of alerts
First alert after hospital discharge
Enrollment in a previous study
Enrollment during 2 week period where EHR system was being upgraded
Randomisation
Eligible patients were identified using an AKI detection algorithm which automatically screened patients as per inclusion and exclusion criteria and randomised them.
As providers could change during hospital stay, randomisation was at patient level. Moreover, all investigators were blinded to randomisation status
Intervention
Automated electronic pop up alerts (see figure 1) with the following features:
Figure 1 from Wilson et al BMJ 2020
Fired when the patients' charts were opened
(An algorithm was built into the EHR system to ensure that the alert fired only in the intervention arm. In the control arm 'silent' alarms were generated which were not displayed but could be tracked. One has to appreciate the lengths to which the investigators went to design the alerts!)
Displayed only to providers who had the authority to change or enter new patient orders (interns, residents, fellows, attendings, nurse practitioners and physician assistants)
Displayed each time the chart was opened
Were suppressed for 48 hours if provider agreed/disagreed with diagnosis of AKI
Separate alerts for each provider of the same patient
Inbuilt option to add AKI to the patient's problem list and to order an AKI order set (blood and urine tests, imaging, etc as in S2)
Fig S2 from Wilson et al BMJ 2020
Primary Outcome
A composite of
inpatient AKI progression (increase in AKI stage),
receipt of dialysis, or
death within 14 days of randomization
Secondary Outcome
Components of the primary outcomes and frequency of various practices of care for AKI. These practices included
administration of contrast, fluids, or a nephrotoxic agent
ordering a urinalysis,
documentation of AKI,
monitoring of creatinine and urine output, and
ordering a kidney consultation.
An assessment of each hospital’s alert effects was also prespecified.
Statistical Analysis
Preliminary data analysis suggested that 24.5% patients would experience the outcome. On this basis, a relative 20% reduction in composite outcome was considered clinically significant and a sample size of 2512 in each arm was determined as necessary to achieve 90% power to detect change. In order to avoid bias resulting from change in AKI management, the sample size was increased by 20%.
Primary analysis and all comparisons of categorical variables → the Mantel-Haenszel test (accounting for each hospital site as a stratum).
The Mantel-Haenszel approach was used to obtain the pooled relative risks across hospital strata without adjusting for other baseline factors.
Sensitivity analysis → modified Poisson generalized estimating equations with a robust variance estimator to present the relative risk estimates, adjusting for the several baseline characteristics (viz. age, sex, race, creatinine, blood urea nitrogen, white blood cell count, heart rate, respiratory rate, systolic and diastolic blood pressure, chronic heart failure, hypertension, diabetes, malignancy, intensive care unit status, modified sequential organ failure assessment (mSOFA) score, Elixhauser comorbidity score, and hospital).
To compare continuous variables → Van Elteren test
To compare time to event between study groups → Cox proportional hazard regression, with intervention as the independent variable, stratified by hospital, with censoring at 14 days after randomization.
For individual hospital analyses → Kaplan-Meier curves were generated and log rank tests were used.
Schoenfeld residuals were used to examine the proportional hazards assumption in Cox models and there were no violations.
The protocol was specifically amended to include a test for site heterogeneity two months after the initial patient enrollment, before alerting any of the hospitals in which adverse outcomes were ultimately detected. This was based on ongoing executive committee calls and the hypothesis that the alert effect might differ by hospital.
Statistical analyses were conducted in Stata version 15.1 (College Station TX), SAS version 9.4 (SAS Institute, Cary, NC), and R (RStudio version 1.2.5033 (R version 3.5.3), Boston, MA), and P≤0.04 was considered statistically significant for the primary outcome (to account for the interim analysis).
Results
6030 patients were enrolled and randomized (see figure S1) who generated 226,316 alerts (actual + silent).
Fig S1 from Wilson et al, BMJ 2020
The baseline characteristics of the study population are shown in Table 1.
Table 1 from Wilson et al, BMJ 2020
Patients randomized to Alerts were more likely to:
Receive an order for intravenous fluids (38.4% Vs 34.8%, absolute difference 3.8% (95% CI 1.4% to 6.2%)),
Receive an order for urinalysis (17.0% Vs 14.9%, 1.9% (0.1% to 3.7%)),
Have serum creatinine measured (87.2% Vs 85.2%, 1.8% (0.1% to 3.6%)) within 24 hours of randomization ( see table 2)
Have documentation of AKI in the problem list (70.0% Vs 63.0%, 7.0% (4.6% to 9.3%))
Table 2 from Wilson et al BMJ 2020
Primary Outcome
Progression of acute kidney injury, dialysis, or death occurred in 653 (21.3%) of 3059 patients with an alert and in 622 (20.9%) of 2971 patients receiving usual care (relative risk 1.02, 95% confidence interval 0.93 to 1.13, P=0.67) (See table 3).
Table 3 from Wilson et al, BMJ 2020
No evidence of a change was seen in the relative effect across intervention groups over time (P=0.90 for interaction).
This is where things became really complicated.
While there did not seem to be any significant advantage of alerts, it did seem like the patients in the Alert arm were actually doing worse!
But this was not the only surprising observation. When the investigators set out to find the cause behind the apparently harmful effect of alerts, they found heterogeneity of effects across the hospitals involved in the study (see figure 2). Hospital 3, for example, showed a protective effect of the alerts, while others showed harm.
Figure 2 from Wilson et al, BMJ 2020
More significantly, there appeared to be harm from alerts at the two non-teaching hospitals compared to the four teaching hospitals.These differences in the primary outcomes seemed to be driven by death as opposed to dialysis or progression of AKI. There was evidence of heterogeneity of the association between alerts and death across the enrolled hospitals (P=0.05). Again, alerts were associated with a significantly higher risk of death at 14 days in the non-teaching hospitals, where 59 (15.6%) patients in the alert group versus 33 (8.6%) patients receiving usual care met this outcome. The relative risk of death was 1.82 (1.22 to 2.72, P=0.003) in the non-teaching hospitals compared with a relative risk of 0.89 (0.74 to 1.06, P=0.18) at the teaching hospitals (P=0.001 for interaction).
Secondary outcomes and process measures
The frequency of kidney consultations was similar between the two arms with no evidence of heterogeneity across the study hospitals. Similarly, in time to event analysis, no increased rate of kidney consultations was found across the two arms (hazard ratio 1.00 (95% confidence interval 0.90 to 1.11).
No significant difference was seen in the rate of discharge to hospice overall or in the non-teaching hospitals.
Subgroup analyses
The effect of alerting was largely similar between medical and surgical patients, ICU and non-ICU patients, by age, race, and sex, and across baseline creatinine values, as shown in figure 3.
Figure 3 from Wilson et al, BMJ 2020
Sensitivity and post hoc analyses
The adjusted relative risks of the primary outcome were broadly similar in teaching and non-teaching hospitals, across various baseline factors (see S19). However, as mentioned before, the adjusted risk of death in the non-teaching hospitals was higher than in teaching hospitals.
Table S19 from Wilson et al, BMJ 2020
A formal mediation (see S20) showed that the adverse event was not mediated by use of IV fluids, medication, or nephrology consultations. The percent of alerts that went to attending versus other providers also did not affect adverse outcomes.
Table S20 from Wilson et al BMJ 2020
The risks of death and progression of AKI, in the two years before the start of this trial in each hospital (fig S7) were similar to those of the control group in the the study.
Fig S7 from Wilson et al BMJ 2020
Discussion
So alerts for recognizing AKI, despite resulting in a small increase in process measures, did not have any result on outcomes that matter: worsening of AKI, need for dialysis, or death.
Does this mean that:
Improved/higher recognition of this level of AKI does not matter?
Do we not have the right steps needed to implement in this setting to protect the kidneys in the face of incipient AKI?
In their previous trial, this group had reported little change in process measures - so the increase in process measures in the present trial, where the intervention had been tweaked, was a positive sign - though it didn’t affect outcomes. These results are in contrast to the Tackling AKI study (Selby et al, JASN 2019), which was a stepped-wedge, cluster RCT (ie sequential randomization into the intervention by month, and not patient-level), and did report changes in process measures, as well as length of stay - though not in 30 day mortality. There was another observational study which also reported improved outcomes (Al-Jaghbeer et al, JASN 2018) but those results could be from a secular improvement over time, or the Hawthorne effect since it was a before/after study.
The notable thing is that there was absolutely no signal of benefit in any of the clinical outcomes (indeed, in their previous pilot trial as well, there had been, if at all, a signal of harm).
What is the significance of the increased death rate in non-teaching hospitals?
It should be kept in mind that the investigators have stated that these results must be considered hypothesis generating and there is no evidence to suggest that they are due to the 'teaching/non teaching' nature of the institute.
Could it have been a mere coincidence? The alert:death association of p = 0.003 makes that seem unlikely.
How about a difference in baseline factors between the two groups? Again unlikely, as adjustment for a variety of baseline factors did not show any difference.
Were the study sites jinxed perhaps? A big NO again as historical death rates were found to be quite similar to the rates seen in the usual care arm.
Let's go further downstream. In his twitter thread, Dr Wilson mentions that he believed aggressive fluid administration might have been responsible for the increased death rate in the alert arm. Surprisingly, even accounting for fluid administration did not attenuate the observed effect. Nor did it change with the type of provider (medical doctors vs physician assistants).
It has to be alert fatigue right? Too many alerts and the providers just ignored the AKI alert? Wrong again! Accounting for the frequency of other electronic health record alerts did not change the observed association either.
In fact, the investigators stated that they were unable to give a satisfactory explanation. Possible reasons could be
Distraction from patient care by alarms
Absence of protective systems (like drug dose monitoring)
Fear of medico-legal implications of not doing anything, once alerted
Are alerts truly a 'minimal risk' intervention?
In this trial, patient consent was waived off with the presumption that informational alerts pose minimal risk to the patient. In the light of harmful effects noted in the alert arm, this presumption may be not true going forwards. All clinical trials based on such seemingly innocuous interventions should be based on evidence of safety and efficacy.
Limitations
Randomization at patient level, allowing for potential provider bias, as they could learn how to better identify acute kidney injury throughout the course of the trial. However, an analysis accounting for the duration that the alerts had been in effect at each hospital found no evidence of this
The alert was largely informational, with no patient-specific recommendations (such as identifying current nephrotoxic drug treatment). The heterogeneity of acute kidney injury might require more personalized interventions?
The alert was sent only to certain care providers, notably excluding nurses and pharmacists, who might also benefit from receipt of the alert.
The alert did not use urine output criteria for AKI and so the study population may not be fully representative.
Limited generalizability: The hospitals represent a diverse mix of patients and care models, suggesting that these results might apply more broadly. But perhaps the Yale hospitals do such a good job at recognizing AKI that there is not much else to do? Perhaps there are centres which are not good at recognizing AKI where this might help?
Strengths
Large size with an ability to detect clinically meaningful changes in hard endpoints
Multicentric
Fully automated enrollment and randomization system
Conclusion
'Alerts did not reduce the risk of our primary outcome among patients in hospital with acute kidney injury. The heterogeneity of effect across clinical centers should lead to a re-evaluation of existing alerting systems for acute kidney injury.'
So we are now in a curious situation. Alerts designed for the purpose of improving patient outcomes don't seem to be useful in that regard. We can't even say they are harmless, as Dr Wilson has discussed in his blog. The authors are already ready with ELAIA-2, in which they will study the effect of specific recommendations following alerts for AKI. Perhaps, then, we will have some answers. Till then we continue to remain uncertain and ponder over the improbable. Meanwhile, let’s stop alerting ourselves. Primum non nocere.
Summary by Namrata Parikh
Nephrologist, Divine Life Hospital, Gujarat, India
NSMC graduate, class of 2020