{
  "answer": "[CONFIDENCE: NONE]\n\n**Pratijña (Claim):** Large language model reasoning performance exhibits fragility under superficial perturbations — changes in surface form that preserve deep semantic structure — suggesting that model reasoning relies on surface-pattern matching rather than genuine semantic understanding of the underlying mathematical or logical relationships.\n\n**Hetu (Reason):** Because performance on structurally identical problems with altered surface details (name substitutions, number changes) drops measurably and sometimes dramatically, while the semantic content — the underlying mathematical or logical relationship — remains unchanged.\n\n**Udaharana (Example):** A model trained to solve \"John has 5 apples; Mary has 3 apples; how many do they have together?\" performs well. Substituting names (\"Pete has 5 apples; Sue has 3 apples...\") produces similar performance. But substituting numbers (\"John has 500 apples; Mary has 300 apples...\") may produce a performance drop, despite the underlying \"add two quantities\" operation being identical. More severely, changing both name and number patterns across a class of problems produces a performance collapse that scaling alone has not prevented.\n\n**Upanaya (Application):** This fragility reproduces the pattern we see in human semantic processing research: when surface form and deep structure diverge, human processors also show performance variation that cannot be explained by semantic understanding alone. The parallel suggests we are observing a genuine limitation of representation rather than a peculiarity of artificial systems.\n\n**Nigamana (Conclusion):** Therefore, large language model reasoning performance is partially grounded in surface-pattern sensitivity rather than deep semantic relationship detection, revealing a genuine fragility in current reasoning architectures.\n\n---\n\n**Purvapaksha (Counter-argument):** The fragility could be explained by computational encoding limitations rather than superficial pattern matching. Large numbers and rare name combinations require additional computational resources for encoding and retrieval; the performance drop reflects capacity constraints rather than shallow processing. The same explanation predicts that training with larger context windows and more diverse data would reduce the perturbation sensitivity — and recent work shows that doing so does reduce the effect, suggesting the fragility is malleable.\n\n**Siddhanta (Refutation):** Addressing this objection directly: The computational encoding explanation is partially compatible with the superficial pattern hypothesis but does not refute it. If the model has learned to map surface patterns to operations, the additional computational burden of large numbers or rare names simply makes the surface-pattern route less reliable, not the deep operation route more active. The prediction that training with diverse contexts reduces the effect is consistent with both explanations. More discriminating evidence is needed: if the model has genuinely learned the deep operation, performance should remain stable across surface variations after sufficient fine-tuning on perturbed examples. The fact that performance improvement requires targeted training on the perturbation type suggests the fragility is not fully eliminated by scaling and is not merely a capacity artifact.",
  "grounded": true,
  "confidence": "high",
  "collection": "llm_reasoning_debate",
  "collection_label": "LLM Reasoning Debate: Do Models Reason or Pattern-Match?",
  "retrieval_status": "ok",
  "collections_available": 9,
  "collections_queried": 9,
  "passages": [
    {
      "text": "names and numbers? Overall, models have noticeable performance variation even if we only change names, but even more when we change numbers or combine these changes. 4.2 HOW FRAGILE IS MATHEMATICAL REASONING IN LARGE LANGUAGE MODELS? In the previous sub-section, we observed high performance variation across different sets generated from the same templates, along with a performance degradation comp",
      "source": "?",
      "section": "",
      "distance": 0.3658
    },
    {
      "text": ", 2020, inter alia). Scaling up the size of language models has been shown to confer a range of benefits, such as improved performance and sample efficiency (Kaplan et al., 2020; Brown et al., 2020, inter alia). However, scaling up model size alone has not proved sufficient for achieving high performance on challenging tasks such as arithmetic, commonsense, and symbolic reasoning (Rae et al., 2021).",
      "source": "?",
      "section": "",
      "distance": 0.5995
    }
  ]
}