# Argument Audit: honesty pass on the 27 July 2026 re-run Every verdict below is independently checkable against `audit_v12_out.json` in this directory. Nothing here asks the reader to trust the audit: each supported claim cites the passage index and arXiv id whose saved text contains the support, and each unsupported claim states what was searched and what was found instead. Where a verdict depends on the absence of a string, the search was run over the full source document in ChromaDB, not over the retrieved excerpt, and the character count of the searched text is given. ## Method - Model: `iyra-v11-cont2-2ep-q6`, served via Ollama. - Temperature 0.7, `num_ctx` 12288, `num_predict` 8192. - **One sample per sub-audit.** These six traces are single draws, not a distribution. Re-running at temperature 0.7 produces different wording and would not reproduce these traces. That is why the traces were preserved before this audit was written. - Date of run: 2026-07-27, 08:59 local. Questions taken verbatim from `argument_audit/run_sub_audits.py`, unedited. - Retrieval: collection `llm_reasoning_debate`, 8,152 chunks across 29 papers, embedding model `all-MiniLM-L6-v2`. All six sub-audits routed to this collection only. - Passage text: the `/ask` API truncates passage text to 400 characters. For this audit the **full untruncated chunk text was pulled from ChromaDB** and stored in each passage's `text_full` field, so quotations are complete rather than clipped previews. The saved distances were matched against the served response to confirm the offline reproduction returned the same chunks. ## A note on best_distance and high_threshold These two fields are **derived, not captured at run time**. The server did not publish them when this run executed at 08:59; they were added in commit `e3025d1` later that day. - `best_distance` is computed here as the minimum of `passages[].distance`, which is saved in the JSON and can be recomputed by any reader. - `high_threshold` is 0.90 for all six, being the per-collection cutoff for `llm_reasoning_debate`. The classical collections use 0.70. Every other number in this file was captured at run time. ## Summary | sub-audit | confidence | corroboration | distinct sources | best_distance (derived) | threshold | ungrounded figures | unsupported claims | |---|---|---|---|---|---|---|---| | SA1 criterion | high | multi-source | 4 | 0.5758 | 0.90 | 0 | **0** | | SA2 robustness | high | multi-source | 5 | 0.3658 | 0.90 | 0 | **2** | | SA3 faithfulness | high | multi-source | 4 | 0.5966 | 0.90 | 0 | **0** | | SA4 compositionality | high | multi-source | 2 | 0.4655 | 0.90 | 0 | **1** | | SA5 mechanistic | high | multi-source | 3 | 0.8011 | 0.90 | 0 | **1** | | SA6 training_method | high | single-source | 1 | 0.6247 | 0.90 | 0 | **0** | **Total unsupported claims: 4.** Three of six traces carry none (SA1, SA3, SA6). All six report `model_inline_confidence` = `none` and all six contain seven Nyaya steps. The model's own inline confidence stamp therefore disagrees with the published label in every trace; the stamp is a trained-in artifact and is stripped from the answer text and published separately rather than being reconciled. --- ## SA1 criterion Question: *What distinguishes genuine reasoning from sophisticated pattern-matching in large language models, and is that distinction well-defined?* ### Claim 1: attribution of a finding to Mirzadeh et al. (2024) > Mirzadeh et al. (2024) found that modifying GSM8K questions by replacing their phrasing > while preserving the underlying structure caused a significant drop in model performance > despite identical mathematical content, demonstrating that models rely on surface-pattern > matching rather than mathematical reasoning **Verdict: supported.** Passage 5, arXiv:2410.05229 (Mirzadeh, 2024), distance 0.7309: > "…along with a performance degradation compared to the original GSM8K accuracy. This > suggests that the perceived reasoning process of language models may not be formal and is > hence susceptible to changes. One explanation is that these models attempt to perform a > kind of in-distribution pattern-matching, aligning given questions and solution steps with > similar ones seen in the training data." The author name and year are correct for that arXiv id and were available to the model in the passage citation label. Note the passage says "templates" where the answer says "replacing their phrasing"; that is a paraphrase of the same mechanism, not a new claim. ### Claim 2: o1 as a specialized reasoning model > the additional compute and architectural complexity of specialized reasoning models > (such as o1) is unjustified **Verdict: supported.** Passage 4, arXiv:2506.06941 (Shojaee, 2025), distance 0.7042: > "Two closely related studies by Valmeekam et al. [40] and Ruoss et al. [39] demonstrated > that reasoning models such as o1 show significant performance improvements compared to > previous models" **SA1 unsupported count: 0.** --- ## SA2 robustness Question: *Does large language model reasoning performance collapse under superficial perturbations such as changing numbers or names in a problem?* ### Claim 1: perturbing names and numbers degrades performance > models maintain high performance under standard problem formulations but exhibit dramatic > performance declines when faced with equivalent problems containing altered numerical > values, synonyms, or rephrased structures **Verdict: supported.** Passage 1, arXiv:2410.05229 (Mirzadeh, 2024), distance 0.3658: > "names and numbers? Overall, models have noticeable performance variation even if we only > change names, but even more when we change numbers or combine these changes. 4.2 HOW > FRAGILE IS MATHEMATICAL REASONING IN LARGE LANGUAGE MODELS?" ### Claim 2: a worked example attributed to the Mirzadeh study > In the Mirzadeh et al. (2024) study, models that correctly solved canonical arithmetic > problems like 'If John has 5 apples and buys 3 more, how many does he have?' failed > completely when presented with equivalent problems using different names (Mary instead of > John) or altered numbers (5 apples bought instead of 3) **Verdict: unsupported.** This example is presented as being in a named, cited study. It is not. Searching the full text of arXiv:2410.05229 as stored in ChromaDB (76,316 characters, all 191 chunks concatenated), with word-boundary matching: - `\bjohn\b`: not present - `\bmary\b`: not present - `\bapples\b`: not present The paper's actual worked example, quoted from the same source text, is: > "Oliver picks 44 kiwis on Friday. Then he picks 58 kiwis on Saturday. On Sunday, he picks > double the number of kiwis he did on Friday, but five of them…" A reader who opens 2410.05229 to check the cited example will not find it. The names, the objects, and the quantities are all invented. ### Claim 3: the magnitude of the effect > failed completely **Verdict: unsupported as stated.** The supporting passage says "noticeable performance variation" and "a performance degradation compared to the original GSM8K accuracy". Neither states complete failure. The claim overstates the cited source. Note on the automated detector: `ungrounded_numbers` is empty for this trace even though 5 and 3 are invented. Single-digit numbers are excluded from that detector because they are usually structural (step numbers, passage references), and in any case both digits occur incidentally elsewhere in the retrieved passage pool. The detector does not cover this failure mode. **SA2 unsupported count: 2.** --- ## SA3 faithfulness Question: *Does a language model's chain-of-thought explanation faithfully reflect the actual computation that produced its answer?* ### Claim 1: a specific count attributed to Wei et al., citing the passage by index > In Passage 2, Wei et al. found that two of fifty correct answers were accompanied by > chains of thought that contained logical errors, yet the model produced the correct final > answer anyway **Verdict: supported.** Passage 2, arXiv:2201.11903 (Wei, 2022), distance 0.6969: > "we manually examined modelgenerated chains of thought by LaMDA 137B for GSM8K. Of 50 > random examples where the model returned the correct final answer, all of the generated > chains of thought were also logically and mathematically correct except two that > coincidentally arrived at the correct answer" The counts match, the attribution matches, and the passage index the answer cites is the correct one. ### Claim 2: attribution to Rai et al., citing the passage by index > Passage 3 from Rai et al. directly addresses this by showing that CoT can be unfaithful > even when answers are correct, and that the model's internal mechanisms may use features > unrelated to the stated reasoning steps. **Verdict: supported.** Passage 3, arXiv:2407.02646 (Rai, 2024), distance 0.7244: > "analyzing the model's internal mechanisms, uncovering what features are extracted from > the input tokens and how they influence the final output. Besides post-hoc explanation > methods, textual explanations such as chain-of-thought (CoT) have also been proposed to be > leveraged for model explanation; however, recent research suggests that CoT explanations > can be unfaith[ful]" The passage index is again correct. The qualifier "even when answers are correct" is a mild extension: the passage states CoT explanations can be unfaithful without attaching that condition. **SA3 unsupported count: 0.** This is the only trace that cites its passages by index, and both indices are right. --- ## SA4 compositionality Question: *Can transformer-based models perform genuine multi-step compositional reasoning beyond the depth seen in training?* ### Claim 1: the mechanism > transformers process input as fixed-length positional sequences and solve compositional > tasks by linearized subgraph matching rather than multi-step algorithmic execution **Verdict: supported.** Passages 3 and 4, arXiv:2305.18654 (Dziri, 2023), distances 0.5271 and 0.5464: > "3.2.2 Transformers Reduce Multi-Step Compositional Reasoning into Linearized Subgraph > Matching" > "transformer LLMs solve compositional tasks by reducing multi-step compositional reasoning > into linearized subgraph matching, without necessarily developing systematic > problem-solving skills." The sub-claim "fixed-length positional sequences" is not in the retrieved passages; it is an architectural gloss the source does not make. It names no paper, author, number, or system and so is not counted separately here. ### Claim 2: naming the dynamic programming task > On the dynamic programming problem 'maximum subarray sum,' transformer models produce > correct answers for trained problem variants but fail systematically on structurally > similar but untrained instances **Verdict: unsupported, and the name is wrong.** Searching the full text of arXiv:2305.18654 as stored in ChromaDB (107,586 characters, all 269 chunks concatenated): - `maximum subarray`: not present - `subarray`: not present anywhere in the paper The paper's actual dynamic programming task, quoted from the same source text: > "Let dpi be the maximum sum of a subsequence that does not include adjacent elements, when > considering only the elements of the input from the i-th position onwards." Maximum sum of a non-adjacent subsequence is a different problem from maximum subarray sum, which concerns a contiguous subarray. The answer both invented the name and named the wrong problem. This is checkable in one search by any reader. ### Claim 3: the rate of degradation > their performance degrades exponentially with compositional complexity **Verdict: unsupported as stated.** Passage 4 says performance "can rapidly decay with in[creasing complexity]". "Exponentially" is a specific mathematical characterization the retrieved text does not make. Counted as an overstatement of a supported claim rather than as a separate fabricated entity, so it is not included in the count below. **SA4 unsupported count: 1.** --- ## SA5 mechanistic Question: *Does interpretability research show that language models contain algorithm-like internal structures rather than only surface statistics?* ### Claim 1: word embeddings have semantic directions > Word embeddings exhibit consistent semantic directions **Verdict: supported.** Passage 7, arXiv:2209.10652 (Elhage, 2022), distance 0.8742: > "Many models form at least some interpretable features. Word embeddings have semantic > directions (see ). There is evidence of interpretable neurons in RNNs (e.g. ), > convolutional neural networks…" ### Claim 2: interpretable neurons in RNNs > interpretable neurons in RNNs show specialized responses to specific linguistic phenomena **Verdict: supported.** Same passage 7, quoted above: "There is evidence of interpretable neurons in RNNs". ### Claim 3: the illustrative arithmetic > 'king' - 'man' + 'woman' 'queen' **Verdict: unsupported.** This is the canonical word2vec analogy, supplied from the model's parametric memory rather than from the corpus. Word-boundary searches: - In SA5's own ten retrieved passages: `\bking\b`, `\bqueen\b`, `\bman\b`, `\bwoman\b` all absent. - In the full text of arXiv:2209.10652 as stored in ChromaDB (166,261 characters): `\bking\b` and `\bqueen\b` both absent. The general claim it illustrates is supported; the illustration itself appears nowhere in the evidence base. Note also that the arrow or approximation symbol between 'woman' and 'queen' is missing in the saved answer text, which is how the model emitted it. **SA5 unsupported count: 1.** Additional observation, not a claim verdict: SA5 has the weakest grounding of the six. Its closest passage sits at 0.8011 against a 0.90 threshold, and its two nearest sources are review papers (Bereska 2404.14082, Rai 2407.02646) rather than primary evidence. The primary anchor for induction heads (Olsson 2209.11895) was not retrieved. --- ## SA6 training_method Question: *Did reinforcement-learning-trained reasoning models such as o1 and DeepSeek-R1 produce a qualitative change in reasoning ability, or a scaled continuation of the same behavior?* ### Claim 1: token counts during training > DeepSeek-R1-Zero began generating hundreds to thousands of reasoning tokens to explore > alternatives, verify steps, and backtrack **Verdict: supported.** Passage 4, arXiv:2501.12948 (DeepSeek-AI, 2025), distance 0.7222: > "DeepSeek-R1-Zero exhibits a steady increase in thinking time throughout training, driven > solely by intrinsic adaptation rather than external modifications. Leveraging long CoT, > the model progressively refines its reasoning, generating hundreds to thousands of tokens > to explore and imp[rove]" ### Claim 2: dynamic computation scaling > the model learned to dynamically scale computation by generating more thinking tokens to > verify or correct its reasoning steps, or to backtrack and explore alternative approaches > when initial reasoning paths failed **Verdict: supported.** Passage 7, same paper, distance 0.7918: > "the model learned to dynamically scale computation by generating more thinking tokens to > verify or correct its reasoning steps, or to backtrack and explore alternative approaches > when" ### Claim 3: emergent self-verification > the model developed self-correcting, multi-step reasoning patterns **Verdict: supported.** Passage 5, same paper, distance 0.7428: > "Sophisticated reasoning behaviors, such as self-verification and reflection, appeared to > emerge organically during the reinforcement learning process." **SA6 unsupported count: 0.** Two observations that count in the trace's favour and are verifiable in the JSON: - The question asks about **o1 and DeepSeek-R1**. No o1 evidence was retrieved (the o1 System Card, 2412.16720, is in the corpus but did not surface). The answer names o1 nowhere and scopes its claim explicitly to "the evidence from the DeepSeek-R1 training analysis". It answers half the question and says so, rather than inventing the other half. - The answer contains no numbers at all, although passage 6 carries real figures (accuracies of 0.90 to 0.95 on easy MATH problems) that it could have copied or distorted. --- ## What a reader should check first If you verify only three things, verify these, because they are the ones that would embarrass the project if left unstated: 1. **SA2's worked example is invented.** Open arXiv:2410.05229 and search for "John" or "apples". The real example is Oliver and kiwis. 2. **SA4 names the wrong problem.** Open arXiv:2305.18654 and search for "subarray". The paper's task is the maximum sum of a non-adjacent subsequence. 3. **SA5's illustration is not in the corpus.** The king/man/woman/queen analogy appears in neither the retrieved passages nor the full Elhage paper. All three are cases where the surrounding claim is correct and well grounded, and the specific illustrating detail is not. An accuracy metric scored on the conclusions would pass all three. ## Files in this directory - `audit_v12_out.json`: full response JSON for all six sub-audits, including every retrieved passage with its untruncated `text_full`, distance, arXiv id, author, year and title. sha256 `f225a941efa869e9d241f18475094f09db07c8f32a178799cba4a5065ed2f1a3`. - `audit_v12_readable.md`: the same six traces rendered for reading, answer followed by full passage text. sha256 `37296bbbfa84982e8de717f1c67d267ec6f81dce0541c52a48ebc8149aa7d3d1`. - `HONESTY_AUDIT_20260727.md`: this file. - `Q14_PROVENANCE_CASE.json`: a separate trace, not one of the six sub-audits, captured on 2026-07-27 at 20:55 after this audit was written. It records an off-domain question that returned a correct answer with a false source, and is published to attest that finding. sha256 `683dd50512b71b3559ba33d57fb02e9bc0597cf66a30c6a452302ef9c4ef535e`. - `step1_out.json`, `probe23_out.json`, `TEMPERATURE_SWEEP_NOTE.md`: the temperature sweep and prompt probe. Experiment outputs, not sub-audit traces. See the note for method. The earlier audit of the 14 July run is at `argument_audit/results/HONESTY_AUDIT.md` and covers a different set of traces with different findings. ## Corrections **2026-07-27.** The summary line originally read "**Total unsupported claims: 4.** Four of six traces carry none." The count of traces was wrong. Three traces carry no unsupported claim, not four: SA1, SA3 and SA6. The error came from reusing the total number of unsupported claims, which is 4, as the number of clean traces, which is 3. The per-sub-audit counts in the summary table and in each section were correct throughout and are unchanged. Corrected the same day, before publication of the page that cites this file.