Hospital Risk Scores and the Alert Problem: Why Good Predictions Still Get Ignored

Hospitals have used early-warning scores for years. Newer versions use machine learning to estimate which inpatients may deteriorate, develop sepsis or be readmitted. On paper, they identify patients who need attention sooner.

In practice, a prediction is only the first step. The rest of the path runs through people, screens and routines.

From prediction to action

A risk model produces a number. For that number to matter, a chain has to work:

1. The model scores the patient at the right time, with current data.
2. The system shows the score or an alert in a place the clinician looks.
3. The clinician reads it and judges it credible.
4. A defined action follows, such as a bedside review.
5. That action changes the patient’s care.

A break anywhere in the chain removes the benefit. Teams often invest heavily in step one and leave the others to chance.

Where the chain breaks

• Timing. An alert that arrives after the clinician has already recognized the problem adds no information.
• Placement. An alert in a screen that nurses rarely open is invisible.
• Credibility. If past alerts were mostly false alarms, new ones are discounted.
• Ownership. If nobody is responsible for responding, everyone assumes someone else will.

Alert fatigue

Clinical staff receive many alarms and notifications. When a large share of them require no action, people learn to dismiss them. This is a well-known problem for monitors and for software alerts.

A model with modest precision can generate a stream of alerts that overwhelms a ward. The relevant numbers are not only sensitivity and specificity but also how many alerts per nurse per shift the threshold produces, and what fraction lead to a useful intervention.

A prediction nobody acts on has the same clinical effect as a prediction nobody made.

Tuning thresholds with the ward

Thresholds should be set with the people who will respond. A lower threshold catches more cases and creates more noise. A higher threshold is quieter but misses more. The right balance depends on staffing, on the cost of a missed event and on what the response involves.

Design principles that help

Explain the alert

A bare score invites doubt. Showing which inputs drove it, such as a rising heart rate or a change in lab values, helps clinicians judge it against what they see at the bedside.

Tie alerts to an action

Each alert level should map to a protocol. A moderate score might trigger a closer observation schedule, and a high one might prompt a physician review. Clear mapping reduces hesitation.

Keep the human in charge

The score supports clinical judgment and does not replace it. Staff should be able to dismiss an alert with a reason, and that feedback should be collected and reviewed.

Evaluating beyond model metrics

A good evaluation asks questions that model metrics cannot answer:

• Did patients receive attention earlier than before?
• Did outcomes such as unplanned transfers or length of stay change?
• How did staff workload change?
• Were there groups of patients for whom the score worked less well?

Prospective evaluation, in which the tool runs in a real ward and results are compared with a baseline, gives stronger evidence than retrospective testing alone. Models can also lose accuracy over time as practice, patient mix and data entry change, so monitoring after launch is part of the job.

A realistic starting point

Teams that begin with one ward, one outcome and a written response protocol learn quickly. They can adjust thresholds, retire alerts that do not help and expand only when the workflow proves stable. The model matters, but the surrounding process determines whether it makes a difference to patients.