AI Confidence vs. User Uncertainty: The Trust Problem Nobody's Solving

Research shows AI trust degrades over time and doesn't recover when systems improve. This isn't an AI problem, it's a UX problem. And it's compounding.

A few years ago I was using ChatGPT for all sorts of tasks. Writing assistance, brainstorming, problem-solving. I was experimenting everything. Now I use it almost exclusively as a search engine (and a million other tools for a million other tasks). Quick queries, simple questions, nothing that requires me to trust the reasoning.

This didn’t happen because of one catastrophic failure. It happened gradually—memory issues here, nonsensical answers there, basic calculations wrong. Each small failure narrowed the scope of what I’d trust it with. I started writing prompts like Google searches: short, simple, constrained. I stopped expecting it to understand context or reason through complexity.

Here’s the problem about this: ChatGPT might have improved in the areas where I stopped trusting it. I wouldn’t know. There’s no feedback mechanism telling me it got better. No signal that would make me try those tasks again. My trust degraded, and now it’s stuck.

This isn’t just my experience. It’s a pattern researchers are documenting across AI systems, and it reveals a fundamental UX problem that’s getting worse as AI becomes more embedded in products.

The Research Problem: Trust Doesn’t Update When It Should

Recent research shows something concerning: users struggle to calibrate their trust even when AI accuracy improves or degrades. Trust is “sticky”—it anchors on early experiences and persists regardless of actual performance changes.

When AI improves, users continue undertrusting because they remember when it was unreliable. When AI degrades, users continue overtrusting because they remember when it worked well. Trust updates slowly and requires many contradictory experiences to shift.

A 2024 study from CHI (the leading human-computer interaction conference) found that this is actually a three-way calibration problem, not a two-way one. It’s not just about whether the AI is confident and accurate. It’s about:

1. AI confidence (how sure is the AI about its answer?)

2. Human self-confidence (how sure am I about my own judgment?)

3. Actual correctness (who’s actually right?)

The study found that humans often have miscalibrated self-confidence—they’re overconfident when wrong or uncertain when right. This affects whether they accept or reject AI suggestions. When you combine miscalibrated AI confidence with miscalibrated human confidence, you get compounding errors.

The researchers tested interventions to calibrate human self-confidence and found that showing people their past performance reduced under-reliance on AI. But it didn’t reduce over-reliance—when AI was wrong but confident, people still followed it.

The Overtrust Problem Is Getting Worse

A January 2025 study from MIT Media Lab documented a striking example of overtrust: people preferred AI-generated medical advice even when physicians labeled it as low accuracy.

Three hundred participants evaluated medical responses from both doctors and AI systems. They couldn’t distinguish between them. More concerningly, they indicated a high tendency to follow low-accuracy AI advice because it sounded complete and confident.

This isn’t just about medical domains. Research from 2024-2025 shows specific scenarios where overtrust is most likely:

• After positive experience strings: When AI succeeds multiple times, users overtrust even when it explicitly fails next

• When output “looks good”: Well-formatted, confident tone, complete-seeming responses trigger trust regardless of accuracy

• When verification is difficult: If checking would take more time than it saves, users accept AI output without validation

Perhaps most surprising: recent research found that higher AI literacy correlates with overestimation of AI competence. People who understand AI better think they can “work around” its limitations through better prompting.

What Users Actually Do: The Behaviour Nobody’s Tracking

Most research on AI trust is short-term—single sessions, maybe a few weeks. But trust changes over time, and those changes affect behaviour in ways that current research doesn’t capture.

Here’s what we know happens but isn’t well-documented in longitudinal studies:

Task narrowing: Users start by trying AI for everything, then after failures, they narrow to specific low-risk tasks. Eventually AI becomes a tool for constrained use cases only, not a general assistant.

Increased verification: Early adoption means accepting AI output directly. After encountering errors, users check everything. Eventually, they use AI for ideas but verify all facts independently.

Trust compartmentalisation: Users develop mental models of where AI works and where it fails. “Good for X, bad for Y.” But here’s the critical part: there’s no feedback loop to update these mental models when AI improves in areas where trust already degraded.

Research shows that only 33% of developers trust AI code output accuracy (down from 43% in 2024), yet 58% say they wouldn’t want to work without AI. Usage is up while trust is down. Developers are locked into using tools they increasingly distrust out of professional necessity.

This creates what I call “approximated trust”—users building mental models based on incomplete information (the errors they catch) while missing all the errors they don’t catch. It’s approximation built on partial visibility.

The UX Problem: Missing Feedback Loops

From a human-computer interaction perspective, this is fundamentally a design problem.

Current AI interfaces provide no mechanism for trust calibration. They don’t signal uncertainty appropriately. They don’t help users understand when to trust versus when to doubt. They don’t provide feedback that helps users learn the AI’s actual capabilities over time.

ChatGPT sounds equally confident whether it’s 95% likely to be right or 60% likely to be right. There’s no UX signal for “I’m guessing here” versus “I’m certain.” When it makes a mistake, there’s no acknowledgment that this is an area where it struggles. When it improves at something, there’s no signal that capability changed.

This violates core HCI principles around feedback loops. Good interfaces help users build accurate mental models through consistent, clear feedback. AI interfaces do the opposite—they present uniform confidence regardless of actual reliability, forcing users to construct mental models from incomplete and often misleading signals.

Research on explainable AI shows that simply showing users how the system works doesn’t fix this. Explanations sometimes increase trust even when AI is wrong (overtrust) or create confusion without improving decisions. Explanation doesn’t equal calibration.

What’s missing isn’t just transparency about how AI works. It’s transparency about when it works, for what tasks, with what level of reliability.

Trust Is Feature-Specific, But Interfaces Treat It as Monolithic

Research on trust in autonomous vehicles found that users trust different features differently: lane-keeping separate from adaptive cruise control. Trust in one feature doesn’t automatically transfer to another.

The same pattern likely applies to AI systems, but current interfaces don’t support feature-specific trust. Everything comes through the same chat interface with the same confident tone.

There’s no way for users to learn: “This AI is 85% accurate at search queries, 60% accurate at code generation, and shouldn’t be trusted for calculations at all.”

When one feature fails, does trust in all features degrade? Or do users learn to compartmentalise? We don’t have good research on this yet, particularly for newer tools like AI coding assistants where multiple capabilities are bundled together.

But here’s what concerns me: users might not stay long enough to learn those distinctions. If trust breaks too early, before users understand AI’s actual boundaries, they leave. The damage happens faster than repair, and faster than users can build nuanced mental models.

The Compounding Risk Nobody’s Planning For

For product teams adding AI features, there’s a risk that isn’t well-understood yet: AI feature failure might damage trust in the entire product, not just the AI component.

Research shows trust is feature-specific in some contexts, but we don’t know if users will compartmentalise AI feature failures or if those failures will spread to their perception of the whole product. Early trust breaks might cause users to leave before they have time to separate AI errors from product quality (let’s think about services that use AI chats for support and make it impossible to get in touch with a human).

This means the decision to add AI features carries more risk than teams realise. One bad AI experience could potentially damage trust in features that worked perfectly before. The trust cost of an AI failure might be higher than the value gained from AI success.

What This Means for Product Teams

If you’re building products with AI features, here’s what current research suggests you should consider:

Study limitations thoroughly before shipping. Understand not just what your AI can do, but what happens when it fails. Map the trust cost of failures in different contexts.

Constrain AI agents intentionally. Don’t use AI for tasks where failure would cause catastrophic trust damage. Some use cases are too risky for current AI reliability.

Plan for trust compounding over time. Early experiences anchor trust strongly. First impressions of AI capability will persist even if performance changes. Design for the long-term trust trajectory, not just the launch experience.

Automate only tasks that don’t require critical thinking or can be constrained enough that errors don’t damage user experience. If a task requires users to trust AI reasoning, make sure the feedback loops exist to calibrate that trust appropriately.

The most important shift: stop thinking about AI features as “adding intelligence” and start thinking about them as “introducing a new trust relationship that will evolve over time in ways you can’t fully control.”

What AI Companies Need to Do

For companies building these AI systems (OpenAI, Google, Anthropic and others) the research is clear: trust calibration is a UX problem that needs UX solutions.

Current approaches focus on making AI more accurate or more explainable. Both matter. But neither solves the fundamental interface problem: users can’t tell when to trust what they’re seeing.

What’s needed:

UX/HCI research to understand and mitigate trust issues proven by current studies. Not just accuracy research. Not just explainability research. Research specifically focused on how interfaces can help users calibrate trust over time.

Transparency about actual capabilities. Not just “here’s how I work” but “here’s what I’m reliable at, here’s where I guess, here’s where I’ve improved, here’s where you should verify.”

Feature-specific feedback. Stop treating AI as monolithic capability. Different tasks have different reliability. The interface should reflect that.

Longitudinal trust repair mechanisms. When AI improves at something users have stopped trusting it for, there needs to be a way to rebuild that trust. Currently, degraded trust is permanent because there’s no signal that anything changed.

The Bigger Picture

We’re in a transition period where AI capability is advancing faster than our understanding of how to design interfaces for appropriate reliance. The result is a growing gap between what AI can do and what users can effectively use it for.

This gap manifests as overtrust in some contexts (leading to AI slop, unchecked errors, quality degradation) and undertrust in others (leading to abandoned features, narrowed use cases, behavioural lock-in).

Both problems stem from the same root cause: missing feedback loops that would help users calibrate trust appropriately and update that calibration as systems change.

From an HCI perspective, this is solvable. We know how to design feedback systems. We know how to help users build accurate mental models. We know how to signal uncertainty and confidence in ways humans can act on.

What’s missing is the acknowledgment from AI companies that this is a design problem, not just a technical problem. And the willingness from product teams to think carefully about whether adding AI features creates more value than it risks in long-term trust damage.

The research is clear: trust compounds over time, degrades faster than it repairs, and doesn’t automatically update when systems improve. If we don’t solve the interface problem, we’ll build increasingly capable AI that fewer people can actually rely on effectively.

---

If you’re building products with AI features and want to think through trust implications: I help teams design research that uncovers how users actually experience AI reliability over time, not just in launch demos. Understanding trust dynamics early helps you avoid compounding problems later.

The best AI features aren’t the most capable ones, they’re the ones users can learn to rely on appropriately.

---

🚀 Hi! I’m Andreea, an academic HCI researcher (gesture recognition and virtual reality interfaces) turned UX researcher that helps various businesses build products users actually love. I conducted hundreds of research studies for tech companies seeking clarity and I started this newsletter to share real examples and stories from my experience, so teams can do better research.

If you found this article helpful, I would be super grateful if you shared it with others.

I also love great discussions and debates, so I’m excited to hear your thoughts in the comments.

When I’m not doing research or writing, I like to go on hikes, read a good novel and play pretend with my dinosaur-obsessed toddler boy. If you’d like to support my work (and energy for writing, hehe), feel free to buy me a coffee!

Either way, thanks for reading and supporting my work ❤️