Generative AI offers many benefits to healthcare systems such as reducing burnout, improving population health, increasing quality and safety, and reducing health disparities. Any technology that offers benefits also has risks, and some of those risks will be offsetting behaviors (i.e., the Peltzman Effect). One significant risk, given human nature, is when people refrain from critically evaluating AI outputs and simply do what the AI suggests. Automation bias is the term for this behavior which Goddard and colleagues define as the tendency to accept the generative AI output “as a heuristic replacement of vigilant information seeking and processing.”[i] Recall that all heuristics are mental shortcuts that facilitate faster decision making. Unfortunately, these heuristics are error prone, and those predictable errors are called cognitive biases.
Automation bias is a serious concern within healthcare, but also across all domains that are using generative AI. In the national security context, there have been concerns for over 20 years. Patriot missiles, for example, have autonomous lethality capability. In 2003 during the Gulf War, one British and one American aircraft were shot down when the Patriot missiles mistook the aircraft for hostile missiles.[ii] These errors were a combination of human and technology errors. Understanding the unique risks in the national security context, Horowitz and Kahn expand our understanding of automation bias as a tendency to “rely on AI decision aids above and beyond the extent to which they should, given the reliability of the algorithms.”[iii]
Given these serious risks, what drives automation bias in healthcare and in national security? Kücking and colleagues reviewed the literature and concluded that automation bias is driven by the following factors: 1) overconfidence in the AI system (see below); 2) overconfidence in self (see Dunning-Krueger Effect); 3) perceived task difficulty and cognitive demand (see Choice Overload, Cognitive Dissonance and Decision Fatigue); and 4) time pressure.[iv] In each of these, the path of least resistance biases the individual towards reflexively accepting the AI output without critical evaluation.
In researching automation bias for this discussion, I thought it would be useful to test out Google’s new genAI Gemini. I had read that it was beating ChatGPT in some metrics and thought I would give it a spin. I asked it to review the medical literature (e.g., National Library of Medicine, Google Scholar) and identify papers on automation bias and the Peltzman effect. The first paper it produced was Wang et al “The double-edge sword: LLM-driven automation bias in clinical decision making.”[v] Sounded great. I tried the URL provided only to learn it didn’t work. I informed Gemini, and it apologized and tried again, and again, and again. I had already fallen victim to automation bias – I assumed the paper was real. I looked up the paper myself in Pubmed and Google Scholar and could not find it. Overly trusting the AI, I assumed that I had erred in my searches and went directly to the Journal of Medical Ethics only to find, yet again, it wasn’t there. Gemini fabricated (“hallucinated”) a reference. It provided useless data, and I only knew it was useless when I tested the data by going to the original source myself.
Automation bias is an example of the Peltzman Effect (also known as the offsetting effect). Sam Peltzman proposed that humans adjust or offset their behaviors based on changes in their perceived risk. And this is the concern: if a physician assumes that the generative AI work is, in effect, a safety measure providing better information or data, the temptation is to refrain from a critical review of the information it provides. Since people are often using generative AI to save time, there is the temptation to avoid a critical review of all the information it produces, since that would take more time and defeat the shortcut.
Understanding the limitations of generative AI and the risks of Automation Bias, what is a responsible use of generative AI in these high-risk situations? In healthcare, it is critical for healthcare providers to understand that generative AI provides information which, like all health information, should be viewed with skepticism. When a physician receives a report that the patient’s sodium is 134 meq/mL, this does not mean that it is actually 134. It means that, with reasonable amount of certainty, the true value lies somewhere close to that value. This is true for all health information. It should never be taken as true with 100% confidence, but rather it has varying degrees of confidence and should be tested against other data. Generative AI outputs are the same – the output provides data that requires critical thinking. The important difference is that the standard deviation for the sodium is known (the range of ‘normal’), but the standard deviation for accuracy in a generative AI output is unknown. So, until there is more data on the accuracy of the output, it will remain important for healthcare providers to use critical thinking. Indeed, critical thinking is the key skill that we must teach all healthcare providers if we are going to help moderate the effects of automation bias as an offsetting behavior.
[i] Goddard, K., Roudsari, A., & Wyatt, J. C. (2012). Automation bias: a systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association : JAMIA, 19(1), 121–127. https://doi.org/10.1136/amiajnl-2011-000089
[ii] Hawley, J. K. (2017). Patriot wars. Center for a New American Security. Retrieved March 4, 2026 from https://www.cnas.org/publications/reports/patriot-wars
[iii] Horowitz, M. C., & Kahn, L. (2024). Bending the automation bias curve: A study of human and AI-based decision making in national security contexts. International Studies Quarterly, 68(2), sqae020.
[iv] Kücking, F., Hübner, U., Przysucha, M., Hannemann, N., Kutza, J. O., Moelleken, M., ... & Busch, D. (2024). Automation bias in AI-decision support: Results from an empirical study. In German Medical Data Sciences 2024 (pp. 298-304). IOS Press.
[v] Wang, X., Li, X., Wu, X., & Liu, Q. (2024). The double-edged sword: LLM-driven automation bias in clinical decision-making. The Journal of Medical Ethics, 50(4), 213–218.