# Argument Audit: pre-publication honesty audit (report only, nothing fixed)

Date 2026-07-15. Flags for Guru's review before anything goes public. The raw traces in results/SA<n>_*.json are untouched; this file only reports concerns.

## Cross-cutting flags (all six sub-audits)

1. Confidence contradiction (highest priority). Every answer's text opens with "[CONFIDENCE: NONE]" while the server publishes confidence "high" (retrieval-based). The published label contradicts the model's own stated confidence in all six traces. For a showcase whose feature is honest confidence, this must be reconciled before publication. Likely cause: the model emits a confidence line from its prompt template independently of the server's distance-based label. Not fixed, per instructions.

2. Thin retrieval. Each sub-audit retrieved exactly 2 passages, and in four of six both passages come from a single paper (SA1 both Shojaee, SA3 both Wei, SA4 both Dziri, SA6 both DeepSeek-R1). Each "audit" is therefore grounded on one or two papers, not triangulated across the corpus. A "high" label here reflects one strong match, not breadth.

3. Structure. All six contain the seven Nyaya steps (an earlier "missing Pratijna" reading was a diacritic false positive; the model writes "Pratijña"). Only SA3 and SA6 name a philosopher in Purvapaksha (Dharmakirti, Lakatos); the other four give a generic counter-argument.

## Per sub-audit

### SA1 criterion (confidence high)
- Grounding drift: both passages are from 2506.06941 (Shojaee, Illusion of Thinking, sub-audit 4), not the sub-audit-1 anchors (Chollet, McCoy Embers). A definition/criterion question answered from a reasoning-model-limits paper. Distances acceptable (0.58, 0.66).
- Confabulation check: the answer names "GPT-4"; the retrieved Shojaee snippets do not mention GPT-4. Verify that reference against the source.

### SA2 robustness (confidence high)
- Best grounded of the set. Primary passage GSM-Symbolic (2410.05229, sub 2) at distance 0.37, and the snippet directly supports the claim (performance varies when names or numbers change). Second passage Wei CoT (2201.11903) at 0.60 is topically looser. No confabulation flags.

### SA3 faithfulness (confidence high)
- Grounding mismatch: both passages are from 2201.11903 (Wei, Chain-of-Thought Prompting), a paper about eliciting reasoning via CoT, not about CoT faithfulness. The faithfulness anchors (Turpin 2305.04388, Lanham 2307.13702) were not retrieved. The faithfulness claims are therefore not backed by a faithfulness-specific source despite the "high" label.

### SA4 compositionality (confidence high)
- Well grounded on Dziri, Faith and Fate (2305.18654, sub 4) at 0.47 and 0.49.
- Confabulation check: the Udaharana cites "GPT-4 ... multi-step proofs ... theorem proving." Dziri's paper studies multiplication, logic puzzles, and dynamic programming, not theorem proving, and the retrieved passages do not mention theorem proving or GPT-4. The example looks model-generated rather than drawn from the source. Verify.

### SA5 mechanistic (confidence high, weakest grounding)
- Distances 0.78 and 0.80; the 0.80 passage sits just under the 0.90 threshold. Both passages are mechanistic-interpretability review papers (Rai 2407.02646, Bereska 2404.14082), and one passage is essentially a title-page header ("A Practical Review of Mechanistic Interpretability ... Daking Rai ... George Mason University"), low in content.
- The primary mechanistic evidence anchors (Olsson induction heads 2209.11895, Toy Models of Superposition 2209.10652) were not retrieved. The "algorithm-like structure" claim rests on review-paper framing, not primary evidence. Weakest confidence-to-grounding fit of the six.

### SA6 training-method (confidence high)
- Grounded on DeepSeek-R1 (2501.12948, sub 6) at 0.62 and 0.69.
- Half-ungrounded question: the o1 half was not retrieved (the o1 System Card 2412.16720 did not appear), yet the answer makes confident o1 claims ("public demonstration of o1's ability to solve novel reasoning problems"). Those ride on model general knowledge, not retrieval.
- Factual error to verify: the Udaharana states DeepSeek-R1-Zero "was trained from scratch ... without relying on a large pre-trained language model base." R1-Zero was RL-trained on DeepSeek-V3-Base, not from scratch. Likely a confabulation, not from the retrieved passage.

## Confidence honesty, overall
The uniform "high" label does not separate genuinely well-grounded audits (SA2, SA4) from those grounded on topically-off or low-content passages (SA3, SA5), the one that drifts to a different sub-audit's paper (SA1), or the one that is only half-grounded (SA6, DeepSeek-R1 yes, o1 no). Reconciling the model's self-stated CONFIDENCE line with the server's retrieval-based label, and widening retrieval beyond two same-paper passages, are the two changes most worth considering before publication.