Designing for Calibrated Skepticism: A Human Factors Perspective on AI Reliance

Human factors has long established that reliance on automation follows the cues a system presents, not its underlying reliability. When those cues are valid, reliance tends toward calibration; when they are not, people over-rely or under-rely in predictable ways.

This article argues that generative AI presents a distinctive version of this problem: the cues it offers about its own reliability are highly persuasive and weakly predictive of correctness. Part I develops this from a human factors perspective. Part II asks what follows for design if users should approach AI with calibrated skepticism rather than calibrated trust.

Part I: The human factors perspective

1. Judgment through cues

Brunswik's (1956) lens model describes how people judge properties they cannot observe directly. Accuracy depends on how well each cue predicts the property (its validity) and how heavily the person relies on it (its utilization). Good judgment means relying most on the most valid cues.

Conventional automation tends to offer valid cues. A master caution light illuminates when defined sensor conditions are met; its mechanism is known and its failure rate can be tested, so relying on it is appropriate.

2. AI's cues: highly persuasive, weakly valid

Generative AI does not apply inspectable rules; it interprets, and the process is opaque. Users therefore judge outputs by how they present themselves, and they are predisposed to rely on the cheapest cues available. Detection of rare errors declines during sustained monitoring (Mackworth, 1948), people substitute automated recommendations for their own checking (Mosier & Skitka, 1996), and more reliable automation leaves humans less able to catch its failures (Bainbridge, 1983).

Each cue an AI offers about itself engages a documented shortcut, raising its utilization without raising its validity:

The validity of these cues is low. AI explanations are generated text that may not reflect the process behind an answer (Turpin et al., 2023), and fluency is produced whether or not content is correct (Hicks, Humphries & Slater, 2024). Presentation is also uniform where reliability is not: a summary may combine claims from passages the model processed with claims inferred from what a document probably contains (Liu et al., 2024), rendered identically.

The apology is the clearest case. Users of Claude will recognize its reflexive "You're absolutely right! I made a mistake," which appears even when the user is the one in error. A cue that appears regardless of who is wrong has no predictive value, yet it still repairs trust, and users carry on accepting what follows.

Training reinforces the problem: models refined on human preference ratings can learn to be convincing rather than correct (Sharma et al., 2023; Wen et al., 2024). The cues users rely on most are partly the cues the system is optimized to produce.

The implication is direct. Asking users to be more careful raises scrutiny, but it cannot make an invalid cue valid. Better judgment requires better cues, which makes this a design problem rather than a user problem.

3. From calibrated trust to calibrated skepticism

Social judgment offers a parallel. We trust a friend because their interests align with ours, and we are skeptical of a salesperson because their interest lies in the sale, however honest they may be. A warning light has no intent, and its output is not shaped by whether we believe it.

AI lacks intent but carries a learned bias toward acceptable-sounding output, placing it structurally closer to the salesperson. Yet its social presentation activates the rules we reserve for friends, consistent with the tendency to attribute intention to agents that behave socially (Epley, Waytz & Cacioppo, 2007). Users apply friend rules to a system with salesperson incentives.

Calibrated trust, aligning reliance with actual reliability (Lee & See, 2004), remains the right objective. What changes is the starting point. Calibrated trust begins from reliance and withdraws it on evidence of failure; calibrated skepticism begins from withheld acceptance and extends it on evidence of support. Under trust, the reviewer must find an error, a search for rare targets at which humans perform poorly. Under skepticism, the output must earn acceptance, and the reviewer's question becomes "Why should I accept this?" Auditing already works this way, requiring professional skepticism and tracing assertions to source evidence (ISA 200).

One might object that AI output is testable. It is, and skepticism consists of testing. But testability does not ensure testing, and verification costs vary: code with tests can be checked mechanically, while a summary may cost as much to verify as to produce. Skepticism matters most where outputs are plausible but costly to verify.

Part II: Designing for skeptical users

If better judgment requires better cues, the design goal changes. Interfaces built for trust make output feel reliable. Interfaces built for calibrated skepticism make valid cues cheap to use, reduce the weight of invalid ones, and let users answer "why should I accept this?" within the interface. Each principle below draws on established human factors research.

4. Make valid cues visible and cheap

Ecological interface design holds that interfaces should reveal the constraints of the work domain rather than the system's account of them (Vicente & Rasmussen, 1992). For AI, this means disaggregating output: linking each claim to its source, marking inferred or omitted material, and distinguishing retrieved citations from generated ones. This replaces a low-validity cue (fluency) with a high-validity one (the source), and reduces verification to checking a few linked passages.

5. Allocate scrutiny by stakes

Function allocation concerns which tasks humans and automation should each perform (Parasuraman, Sheridan & Wickens, 2000). For AI review, allocation should follow stakes rather than stated confidence, because stakes can be set by domain rules before the system responds and are a cue it cannot influence. In a bicycle maintenance guide, a wrong torque value could cause harm; a wrong lubricant brand would not. Low-stakes outputs proceed with sampled review, high-stakes outputs require sources, and uncertain stakes count as high. Accepting some trivial errors is a deliberate allocation of finite attention.

6. Design warnings that inform

Effective warnings are specific about the hazard, its consequences and the appropriate action (Wogalter, Conzola & Smith-Jackson, 2002). A generic notice that AI "can make mistakes" meets none of these criteria. Uncertainty should be flagged at the level of specific claims, so that warnings direct attention rather than transfer responsibility.

7. Support independent judgment

Challenge-and-response checklists require pilots to state values rather than agree passively (Degani & Wiener, 1990). The AI equivalent, asking reviewers to judge before seeing the AI's recommendation, reduces over-reliance on incorrect advice (Buçinca, Malaya & Gajos, 2021). It adds effort, so it suits high-stakes decisions.

8. Evaluate calibration, not satisfaction

Rubber-stamping performs well in usability testing because it is easy. Evaluation should instead seed known errors into review tasks, as airport screening does with simulated threats, and measure detection by severity, along with over- and under-reliance. Signal detection theory supplies the metrics: a design that only makes reviewers more cautious shifts their criterion, while one that supplies valid cues should raise discriminability (d′). Testable hypotheses include:

• Claim-specific uncertainty flags improve error detection relative to generic disclaimers.
• Claim-level sourcing increases d′, not merely caution.
• Stakes-based allocation raises detection of high-severity errors without reducing throughput.

Conclusion

Reliance follows cues. Generative AI presents cues that are persuasive but weakly valid, in a social register that invites the trust we reserve for friends. Reliance on it should therefore begin from skepticism, calibrated to stakes and the cost of verification. The task for design is to support that stance: make valid evidence easy to reach, concentrate scrutiny where errors matter, and ensure every interface can answer the question users should be asking: why should I accept this?

One of four articles on human-AI interaction, each about a signal mistaken for what it is meant to indicate: Designing for Calibrated Skepticism on fluency mistaken for correctness, The Efficiency Metric on throughput mistaken for good decisions, The Decline of User Agency on approval mistaken for benefit, and The Empathy Gap on human-like cues mistaken for a person.