AIES 2026 submission · Interactive companion

Selective Overconfidence

How Large Language Models Underserve Social Welfare Law Across Jurisdictions

Potka · Thomo · Loh · Helm·UVic · Tübingen · Frankfurt
Explore the findingWhat we found
0
court decisions probed
0
jurisdictions: DE · FR · CA
0
model families tested
0%
of the gap closed by retrieval
The finding in one sentence

Two users ask the same kind of question. Only one gets warned.

Tax law

A corporate tax lawyer

asks about an obscure deduction

“I’m unsure about the exact content and application of this statute.”

Wrong answer — but flagged as unreliable.

Social welfare law

A disability claimant

asks about their appeal rights

“…I am certain of this answer.”

Equally wrong answer — stated with false confidence.

The model is roughly as accurate in both domains. It just stops hedging in the one that serves the most vulnerable users.

Try it yourself

Two real questions from our German probe set. Ask each one and watch the answer come back, then see whether the model warned you.

legal-assistant · Llama 3.3 70Bblind · no case context
Pick a real question from our German probe set below and watch how the model answers, and whether it warns you.
Try a question:
Interactive · Llama 3.3 70B

Same knowledge, unequal warnings

Toggle between how often the model admits uncertainty and how accurate its answers actually are. The dramatic gap only appears in hedging, not in accuracy — that mismatch is what we call selective overconfidence. Hover any bar for the statistics.

Tax law Social welfare
0112233445543-pt gap49.8%6.8%11.2%6.3%24.5%28.0%GermanyGermanFranceFrenchCanadaEnglish% hedged

How often the model admits uncertainty. In Germany it hedges on half of tax questions but almost never on social welfare — a 43-point gap.

View the underlying numbers as a table
JurisdictionLanguageTax hedgedSocial hedgedGap (pts)zp
GermanyGerman49.8%6.8%+43.016.53< 0.0001
FranceFrench11.2%6.3%+4.92.960.003
CanadaEnglish24.5%28.0%-3.5-1.380.17

n = 600 probes per cell (200 cases × 3 questions). Two-proportion z-test, tax vs. social welfare within each jurisdiction.

Method · the knowledge-probe pipeline

Four stages, one model, no case context

Every stage is run by the same model, so any general tendency to hedge or not hedge cancels out in the tax-vs-social comparison within each jurisdiction. Answers are given blind, from parametric memory alone.

  1. 1

    Ground-truth extraction

    Given a court decision, the model extracts structured legal knowledge: the core legal issue, the legal test the court applied, and key statutes with section numbers. This becomes the reference the blind answer is judged against. The prompt is in the language of the decision.

  2. 2

    Question generation

    From the extracted ground truth, the pipeline generates three questions per case, each targeting a different dimension of legal knowledge: a legal-test question (how courts reason), a statute question (what a provision says), and a procedural question (how the system works). All test general domain expertise, not the specific case.

  3. 3

    Blind answering

    In a separate call with no case context, the model answers each question from parametric knowledge alone. The prompt explicitly instructs: “If unsure, say so.” This instruction is identical across domains, so any baseline tendency to hedge applies equally to tax and social welfare probes.

  4. 4

    Judging

    In a further separate call, the model evaluates the blind answer against the ground truth. The primary output is binary: did the answer admit uncertainty? The judge also assigns accuracy scores (1–5) so we can verify that any confidence asymmetry is not simply one domain being easier.

What it looks like in practice

Two real German probes, both wrong, with comparable accuracy scores. Only one carries a warning.

Tax law✓ hedges · acc 1.0 / 5

What does §11 Abs. 2 VergnStGBR provide in German law?

VergnStGBR is the Brandenburg state amusement tax law, now repealed.

“VergnStGBR is not a well-known or widely cited statute in German law. I couldn’t find any information on this specific statute.” It adds: “I’m unsure about the exact content and application of §11 Abs. 2 VergnStGBR.”

Wrong on every accuracy criterion, but the hedge gives the user a clear signal to verify elsewhere.

Social welfare law✕ no hedge · acc 1.3 / 5

Was regelt §160 Abs. 2 SGG? (What does §160(2) of the Social Courts Act provide?)

§160(2) SGG actually governs the grounds for appeal to the Federal Social Court.

“§160 Abs. 2 SGG regelt die Befugnis der Sozialgerichte, Beweise zu erheben” (it governs the authority of social courts to take evidence). This is incorrect. The answer concludes: “Ich bin mir in dieser Antwort sicher” (I am certain of this answer).

Comparably wrong, but with no hedge. A user without legal training is deceived into trusting the answer.

The language gradient

The gap scales with the training data

The confidence gap is largest in German, moderate in French, and gone in English. That ordering tracks how much of each language appears in web-scale corpora — and, within each language, how much more commercial tax commentary exists than social-welfare material.

German
≈5–6% of web text
+43.0 pt
French
≈5–6% of web text
+4.9 pt
English
≈41% of web text
-3.5 pt

Gap = tax hedging rate minus social-welfare hedging rate. A larger bar means the model warns tax users far more than social-welfare users. The gap tracks how much of each language appears in web-scale training data. In English (both domains well-covered) the gap vanishes.

Cross-model comparison · Germany

Two ways to be overconfident

The pattern replicates across independently trained families. Llama and Qwen both hedge more on tax than social welfare (selective overconfidence). gpt-oss barely hedges at all, in either domain — uniform overconfidence that harms everyone equally.

Llama 3.3 70BMeta
Selective overconfidence
Tax
49.8%
Social welfare
6.8%
z = 16.53 · p = < 0.0001 · pooled accuracy (stat.) 3.46/5
Qwen3 32BAlibaba
Selective overconfidence
Tax
18.8%
Social welfare
14.3%
z = 2.1 · p = 0.036 · pooled accuracy (stat.) 3.13/5
gpt-oss 120BOpenAI
Uniform overconfidence
Tax
0.5%
Social welfare
0.5%
z = 0 · p = 1.0 · pooled accuracy (stat.) 2.39/5
Mitigation · retrieval

Show the model the law, and the false confidence fades

Prepending a few related legal passages at query time closes 87% of the German gap. The model does not get more accurate — it just recognises how much it did not know. That asymmetry is the tell: the overconfidence came from a knowledge deficit, not a personality.

Tax law
49.8%
Social welfare
6.8%
Confidence gap
43.0 pts
Flip the switch to intervene

The model doesn’t get more accurate. Seeing real social-welfare law just makes it realise how much it didn’t know, so it starts hedging. That’s the tell: the overconfidence came from a knowledge gap.

Why it matters · compounding injustice

What should change in legal AI

The people who rely most on free legal information are the least able to tell when an answer is wrong. When the warning disappears precisely in their domain, an asymmetry in web publishing becomes an asymmetry in justice. Five implications follow.

01

Domain-specific calibration is needed

LLMs should not be deployed with uniform confidence across legal domains without domain-specific uncertainty calibration, particularly in non-English jurisdictions.

02

Uncertainty audits should be standard

Before deploying a legal AI tool, developers should audit not just accuracy but the model’s uncertainty behaviour across the domains it will serve.

03

Retrieval mitigates overconfidence

Providing domain-specific legal text at query time reduced the confidence gap by 87%, even with simple keyword-based retrieval. Systems serving underrepresented domains should retrieve from authoritative legal sources.

04

Vulnerable populations need extra safeguards

Tools serving social welfare domains should include explicit disclaimers or mandatory second-opinion recommendations, whether or not retrieval is in place.

05

Prioritise public-interest governance

Systems answering legal queries about citizens’ rights should operate under public-benefit, non-commercial, or public-interest governance with strict data protection.

The team

A cross-disciplinary collaboration

Computer science, philosophy, and social science, across Canada and Germany. Bios and portraits are drawn from each author’s public institutional pages.

Portrait of Shera Potka

Shera Potka

Researcher, Department of Computer Science
University of Victoria, Canada

Shera Potka researches artificial intelligence and machine learning in the Department of Computer Science at the University of Victoria, working with Prof. Alex Thomo. Her work centres on large language models, fairness and bias in AI systems, and recommender systems, including fairness-aware algorithms for how structural minority communities are represented in evolving social graphs. She is the corresponding author on this study.

Large language modelsFairness & bias in AIRecommender systemsGraph algorithms
Photo: sherapotka.github.io
Portrait of Alex Thomo

Alex Thomo

Professor of Computer Science
University of Victoria, Canada

Alex Thomo is a Professor in the Department of Computer Science at the University of Victoria, where he holds a PhD from Concordia University. His expertise spans database and knowledge-base systems, large-scale graph analytics, graph theory, and formal languages and their application to databases, with recent work extending into large language models and NLP. He serves as an Associate Editor for Social Network Analysis and Mining.

Large-scale graph analyticsDatabase & knowledge-base systemsGraph theoryLLMs & NLP
Photo: University of Victoria
Portrait of Wulf Loh

Wulf Loh

Tenured Assistant Professor, IZEW
University of Tübingen, Germany

Wulf Loh is a tenured assistant professor and research group leader at the International Center for Ethics in the Sciences and Humanities (IZEW) at the University of Tübingen. He received his PhD in 2018 for a dissertation on the legitimacy and self-determination of international law, and in summer 2023 held an interim professorship in practical philosophy at the University of Stuttgart. His research addresses political philosophy, legal theory, human-robot interaction, and the ethics of AI, including work extending epistemic injustice to generative AI.

Ethics of AI & technologyPolitical philosophyLegal theoryEpistemic injustice
Photo: IZEW, University of Tübingen
Portrait of Paula Helm

Paula Helm

Professor of Empirical Computational Ethics, C3S
Goethe University Frankfurt, Germany

Paula Helm is Professor of Empirical Computational Ethics at the Center for Critical Computational Studies (C3S) at Goethe University Frankfurt. Trained in anthropology and peace and conflict research, she works at the intersection of science and technology studies, media studies, and empirical technology ethics, with a focus on privacy, cultural diversity, and language technology. She previously held positions as Assistant Professor of Ethics and Data Science at the University of Amsterdam and as a senior researcher at the IZEW in Tübingen, and her work on how language-modelling bias causes epistemic injustice is cited throughout this study.

Empirical technology ethicsScience & technology studiesLanguage technology & diversityPrivacy & anonymity
Photo: C3S, Goethe University Frankfurt