Current Location: > Detailed Browse

Rewriting the Stakes: A Controlled Evaluation of Game-Theoretic and Philosophical Prompts on Self-Reported Confidence and Hallucination

请选择邀稿期刊:
Abstract: Prompt wording rivals mitigation methods in effect size, yet game-theoretic and philosophical prompts targeting self-reported confidence and abstention lack controlled evaluation. On four public datasets (TruthfulQA, FinQA, SelfAware, FinanceBench) plus a synthetic financial corpus, we test DeepSeek-R1, Qwen3-32B and four further families under seven prompt configurations in a paired design, measuring confidence level, dispersion, discriminability and hallucination, with tri-judge majority labelling (kappa = 0.57-0.95; alpha = 0.56-0.97) validated by 70 manual items (kappa = 0.96). We find: (1) game-theoretic prompts barely move mean confidence (delta <= 3.8 pt) but reshape its form (adversarial auditing switched the confidence line's language on 12/30 items and doubled its SD); at n=150 only the downward level shift survives FDR correction (2.5 and 5.7 pt, q = 0.0035), the discriminability gain gone (delta AUC = 0.000, p = 0.98); (2) the Brier prompt is strategically exploitable: closed-book hallucination rises from 6.7% to 53.3% through hedged guessing, while adding the philosophical frame removes hallucination at the cost of 76.7% abstention and zero accuracy; (3) under full evidence prompt differences vanish (24-27/30 vs 25/30), and hallucinated open-book answers still report confidence 87.5-95.0; (4) repricing abstention (free versus scored zero) moves closed-book hallucination from 6.7% to 100% (30/30) and numeric commitment from 3.3% to 83.3%, giving causal evidence for commitment-cost repricing; (5) on FinanceBench (n=100), neutral retrieval hallucinates on 31% of items; applying the philosophy+Brier prompt to the same evidence cuts this to 17%, raising accuracy from 69% to 80% (net +11, McNemar p = 0.027), whereas using it as a closed-book fallback when retrieval fails recovers nothing, and the gain vanishes where retrieval is reliable (FinQA, net -1). We propose the commitment-cost mechanism, a value-condition matrix, and a third domain type (zero closed-book precision). Claims are scoped to the models, datasets, temperatures and prompts tested.

Version History

[V1] 2026-09-29 02:41:21 ChinaXiv:202609.00518V1 Download
Download
Preview
Peer Review Status
Awaiting Review
License Information
metrics index
  •  Hits857
  •  Downloads190
Comment
Share
Apply for expert review