Systematic Falsification of Metacognitive-Gating Approaches to Large Language Model Hallucination: A Controlled Study Across Four Models and Three Methodological Warnings

Abstract:
[Objective] To test whether metacognitive-gating dynamic retrieval, a popular approach to mitigating hallucination in large language models, actually outperforms 
fixed retrieval policies, and to examine the three implicit assumptions on which it relies.
[Methods] Two corpora are constructed: a controlled synthetic corpus A (40 fictitious companies, 200 questions, gold labels identical to the corpus text) and a real decontaminated corpus B (SEC XBRL filing values of 12 U.S. listed companies, with three contamination tiers defined by first public disclosure date and an entity-substitution group serving as a guessing baseline). Ten experimental arms are run on four local 4-bit quantized instruction-tuned models, yielding 20,800 controlled generations; exact replay is performed over all 102 thresholds, with a same-trigger-rate random gate control and alternative signal channels (semantic entropy, self-consistency, logits, hidden-layer probe).
[Results] Relative to the full-retrieval baseline, gating changes hallucination rates by +37.0pt, 0.0pt, +6.5pt and 0.0pt across the four models — no improvement anywhere — and the two "ties" are in fact degradation at a 100% trigger rate; the optimum lies at an endpoint for all 102 thresholds. Using "whether the closed-book answer is correct" as the label, the discriminative power of self-reported confidence is 0.364, 0.500, 0.439 and 0.341, with two models' 95% intervals lying entirely below 0.5; switching to sampled semantic entropy raises within-stratum AUC to 0.886 (1.5B) and 0.750 (7B), showing that what fails is natural-language self-report, not the uncertainty information inside the model. Context anchoring is the only method that never falls below the retrieval baseline on any of the four models while adding no extra calls.
[Limitations] The scope is limited to 1.5B–7B open-source instruction-tuned models and the present corpus configuration; training-time algorithms are not tested; the 20B remote batch and the deployment-side observations (n=1 per cell) are small samples and do not support statistical inference.
[Conclusions] Metacognitive-gating dynamic retrieval does not constitute an improvement over fixed policies under the controlled conditions of this study, and its failure can be located precisely in the self-reported integer channel; the validity of such self-reported signals must be verified on the target model before use and cannot be treated as a default design assumption.