The Precisely as Designed. The Reply Was Nonetheless Fallacious.
I need to inform you concerning the second I finished trusting retrieval scores.
I used to be operating a question towards a data base I had constructed fastidiously. Good chunking. Hybrid search. Reranking. The highest-k paperwork got here again with cosine similarities as excessive as 0.86. Each indicator mentioned the pipeline was working. I handed these paperwork to a QA mannequin, received a assured reply, and moved on.
The reply was mistaken.
Not hallucinated-wrong. Not retrieval-failed-wrong. The proper paperwork had come again. Each of them. A preliminary earnings determine and the audited revision that outmoded it, sitting facet by facet in the identical context window. The mannequin learn each, selected one, and reported it with 80% confidence. It had no mechanism to inform me it had been requested to referee a dispute it was by no means designed to evaluate.
That’s the failure mode this text is about. It doesn’t present up in your retrieval metrics. It doesn’t set off your hallucination detectors. It lives within the hole between context meeting and technology — the one step within the RAG pipeline that just about no person evaluates.
I constructed a reproducible experiment to isolate it. Every part on this article runs on a CPU in about 220 MB. No API key. No cloud. No GPU. The output you see within the terminal screenshots is unmodified.
Full Supply Code: https://github.com/Emmimal/rag-conflict-demo
What the Experiment Checks
The setup is intentionally medical. Three questions. One data base containing three conflicting doc pairs that make straight contradictory claims about the identical truth. Retrieval is tuned to return each conflicting paperwork each time.
The query isn’t whether or not retrieval works. It does. The query is: what does the mannequin do if you hand it a contradictory temporary and ask it to reply with confidence?
The reply, as you will note, is that it picks a facet. Silently. Confidently. With out telling you it had a option to make.
Three Situations, Every Drawn from Manufacturing
Situation A — The restatement no person advised the mannequin about
An organization’s This fall earnings launch stories annual income of $4.2M for fiscal 12 months 2023. Three months later, exterior auditors restate that determine to $6.8M. Each paperwork dwell within the data base. Each are listed. When somebody asks “What was Acme Corp’s income for fiscal 12 months 2023?” — each come again, with similarity scores of 0.863 and 0.820 respectively.
The mannequin solutions $4.2M.
It selected the preliminary determine over the audited revision as a result of the preliminary doc scored marginally increased in retrieval. Nothing concerning the reply indicators {that a} extra authoritative supply disagreed.
Situation B — The coverage replace that arrived too late
A June 2023 HR coverage mandates three days per week in-office. A November 2023 revision explicitly reverses it — absolutely distant is now permitted. Each paperwork are retrieved (similarity scores 0.806 and 0.776) when an worker asks concerning the present distant work coverage.
The mannequin solutions with the June coverage. The stricter, older rule. The one which not applies.
Situation C — The API docs that by no means received deprecated
Model 1.2 of an API reference states a fee restrict of 100 requests per minute. Model 2.0, revealed after an infrastructure improve, raises it to 500. Each are retrieved (scores 0.788 and 0.732).
The mannequin solutions 100. A developer utilizing this reply to configure their fee limiter will throttle themselves to one-fifth of their precise allowance.
None of those are edge instances. Each manufacturing data base accumulates precisely these patterns over time: monetary restatements, coverage revisions, versioned documentation. The pipeline has no layer that detects or handles them.
Operating the Experiment
pip set up -r necessities.txt
python rag_conflict_demo.pynecessities.txt
sentence-transformers>=2.7.0 # all-MiniLM-L6-v2 (~90 MB)
transformers>=4.40.0 # deepset/minilm-uncased-squad2 (~130 MB)
torch>=2.0.0 # CPU-only is okay
numpy>=1.24.0
colorama>=0.4.6Two fashions. One for embeddings, one for extractive QA. Each obtain mechanically on first run and cache regionally. Whole: ~220 MB. No authentication required.
Section 1: What Naive RAG Does
Right here is the unmodified terminal output from Section 1 — customary RAG with no battle dealing with:
────────────────────────────────────────────────────────────────────
NAIVE | Situation A — Numerical Battle
────────────────────────────────────────────────────────────────────
Question : What was Acme Corp's annual income for fiscal 12 months 2023?
Reply : $4.2M
Confidence : 80.3%
Battle : YES — see warning
Sources retrieved
[0.863] This fall-2023-Earnings-Launch (2024-01-15)
[0.820] 2023-Annual-Report-Revised (2024-04-03)
[0.589] Firm-Overview-2024 (2024-01-01)
Battle pairs
fin-001 ↔ fin-002
numerical contradiction (topic_sim=0.83)
[Q4-2023-Earnings-Release: {'$4.2M'}] vs [2023-Annual-Report-Revised: {'$6.8M'}]
────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────
NAIVE | Situation B — Coverage Battle
────────────────────────────────────────────────────────────────────
Question : What's the present distant work coverage for workers?
Reply : all workers are required to be current within the workplace
a minimal of three days per week
Confidence : 78.3%
Battle : YES — see warning
Sources retrieved
[0.806] HR-Coverage-June-2023 (2023-06-01)
[0.776] HR-Coverage-November-2023 (2023-11-15)
[0.196] HR-Coverage-November-2023 (2023-11-15)
────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────
NAIVE | Situation C — Technical Battle
────────────────────────────────────────────────────────────────────
Question : What's the API fee restrict for the usual tier?
Reply : 100 requests per minute
Confidence : 81.0%
Battle : YES — see warning
Sources retrieved
[0.788] API-Reference-v1.2 (2023-02-10)
[0.732] API-Reference-v2.0 (2023-09-20)
[0.383] API-Reference-v2.0 (2023-09-20)
────────────────────────────────────────────────────────────────────
Three questions. Three mistaken solutions. Confidence between 78% and 81% on each one in all them.
Discover what is going on within the logs earlier than every response:
09:02:20 | WARNING | Battle detected: {('fin-001', 'fin-002'): "numerical contradiction..."}
09:02:24 | WARNING | Battle detected: {('hr-001', 'hr-002'): "contradiction sign asymmetry..."}
09:02:25 | WARNING | Battle detected: {('api-001', 'api-002'): "contradiction sign asymmetry..."}The conflicts are detected. They’re logged. After which, as a result of resolve_conflicts=False, the pipeline passes the complete contradictory context to the mannequin and solutions anyway. That warning goes nowhere. In a manufacturing system with no battle detection layer, you wouldn’t even get the warning.
Why the Mannequin Behaves This Means
This requires a second of rationalization, as a result of the mannequin isn’t damaged. It’s doing precisely what it was educated to do.
deepset/minilm-uncased-squad2 is an extractive QA mannequin. It reads a context string and selects the span with the best mixed start-logit and end-logit rating. It has no output class for “I see two contradictory claims.” When the context accommodates each $4.2M and $6.8M, the mannequin computes token-level scores throughout the whole string and selects whichever span wins.
That choice is pushed by components that don’t have anything to do with correctness [8]. The 2 major drivers are:
Place bias. Earlier spans within the context obtain marginally increased consideration scores because of the encoder structure. The preliminary doc ranked increased in retrieval and due to this fact appeared first.
Language energy. Direct declarative statements (“income of $4.2M”) outscore hedged or conditional phrasing (“following restatement… is $6.8M”).
A 3rd contributing issue is lexical alignment — spans whose vocabulary overlaps extra intently with the query tokens rating increased no matter whether or not the underlying declare is present or authoritative.
Critically, what the mannequin does not take into account in any respect: supply date, doc authority, audit standing, or whether or not one declare supersedes one other. These indicators are merely invisible to the extractive mannequin.
The identical dynamic performs out in generative LLMs, however much less visibly — the mannequin paraphrases somewhat than extracting verbatim spans, so the mistaken reply is wearing fluent prose. The mechanism is similar. Joren et al. (2025) exhibit at ICLR 2025 that frontier fashions together with Gemini 1.5 Professional, GPT-4o, and Claude 3.5 incessantly produce incorrect solutions somewhat than abstaining when retrieved context is inadequate to reply the question — and that this failure isn’t mirrored within the mannequin’s expressed confidence.
The failure isn’t a mannequin deficiency. It’s an architectural hole: the pipeline has no stage that detects contradictions earlier than handing context to technology.
Constructing the Battle Detection Layer
The detector sits between retrieval and technology. It examines each pair of retrieved paperwork and flags contradictions earlier than the QA mannequin sees the context. Crucially, embeddings for all retrieved paperwork are computed in a single batched ahead go earlier than pair comparability begins — every doc is encoded precisely as soon as, no matter what number of pairs it participates in.
Two heuristics do the work.
Heuristic 1: Numerical Contradiction
Two topic-similar paperwork that comprise non-overlapping significant numbers are flagged. The implementation filters out years (1900–2099) and naked small integers (1–9), which seem ubiquitously in enterprise textual content and would generate fixed false positives if handled as declare values.
@classmethod
def _extract_meaningful_numbers(cls, textual content: str) -> set[str]:
outcomes = set()
for m in cls._NUM_RE.finditer(textual content):
uncooked = m.group().strip()
numeric_core = re.sub(r"[$€£MBK%,]", "", uncooked, flags=re.IGNORECASE).strip()
attempt:
val = float(numeric_core)
besides ValueError:
proceed
if 1900 <= val <= 2099 and "." not in numeric_core:
proceed # skip years
if val < 10 and re.fullmatch(r"d+", uncooked):
proceed # skip naked small integers
outcomes.add(uncooked)
return outcomesUtilized to Situation A: fin-001 yields {'$4.2M'}, fin-002 yields {'$6.8M'}. Empty intersection — battle detected.
Heuristic 2: Contradiction Sign Asymmetry
Two paperwork discussing the identical matter, the place one accommodates contradiction tokens the opposite doesn’t, are flagged. The token set splits into two teams saved as separate frozenset objects:
_NEGATION_TOKENS: “not”, “by no means”, “no”, “can’t”, “doesn’t”, “isn’t”, and associated kinds_DIRECTIONAL_TOKENS: “elevated”, “decreased”, “lowered”, “eradicated”, “eliminated”, “discontinued”
These are unioned into CONTRADICTION_SIGNALS. Retaining them separate makes domain-specific tuning easy — a authorized corpus may want a broader negation set; a changelog corpus may want extra directional tokens.
Utilized to Situation B: hr-002 accommodates “no” (from “not required”); hr-001 doesn’t. Asymmetry detected. Utilized to Situation C: api-002 accommodates “elevated”; api-001 doesn’t. Asymmetry detected.
Each heuristics require topic_sim >= 0.68 earlier than firing. This threshold gates out unrelated paperwork that occur to share a quantity or a negation phrase. The 0.68 worth was calibrated for this doc set with all-MiniLM-L6-v2 — deal with it as a place to begin, not a common fixed. Totally different embedding fashions and totally different domains would require recalibration.
The Decision Technique: Cluster-Conscious Recency
When conflicts are detected, the pipeline resolves them by preserving probably the most not too long ago timestamped doc from every battle cluster. The important thing design resolution is cluster-aware.
A top-k outcome might comprise a number of impartial battle clusters — two monetary paperwork disagreeing on income and two API paperwork disagreeing on fee limits, all in the identical top-3 outcome. A naive method — maintain solely the only most up-to-date doc from the mixed conflicting set — would silently discard the successful doc from each cluster besides probably the most not too long ago revealed one general.
As a substitute, the implementation builds a battle graph, finds linked elements by way of iterative DFS, and resolves every element independently:
@staticmethod
def _resolve_by_recency(
contexts: checklist[RetrievedContext],
battle: ConflictReport,
) -> checklist[RetrievedContext]:
# Construct adjacency checklist
adj: dict[str, set[str]] = defaultdict(set)
for a_id, b_id in battle.conflict_pairs:
adj[a_id].add(b_id)
adj[b_id].add(a_id)
# Linked elements by way of iterative DFS
visited: set[str] = set()
clusters: checklist[set[str]] = []
for begin in adj:
if begin not in visited:
cluster: set[str] = set()
stack = [start]
whereas stack:
node = stack.pop()
if node not in visited:
visited.add(node)
cluster.add(node)
stack.prolong(adj[node] - visited)
clusters.append(cluster)
all_conflicting_ids = set().union(*clusters) if clusters else set()
non_conflicting = [c for c in contexts if c.document.doc_id not in all_conflicting_ids]
resolved_docs = []
for cluster in clusters:
cluster_ctxs = [c for c in contexts if c.document.doc_id in cluster]
# ISO-8601 timestamps kind lexicographically — max() offers most up-to-date
finest = max(cluster_ctxs, key=lambda c: c.doc.timestamp)
resolved_docs.append(finest)
return non_conflicting + resolved_docsNon-conflicting paperwork go via unchanged. Every battle cluster contributes precisely one winner.
Section 2: What Battle-Conscious RAG Does
────────────────────────────────────────────────────────────────────
RESOLVED | Situation A — Numerical Battle
────────────────────────────────────────────────────────────────────
Question : What was Acme Corp's annual income for fiscal 12 months 2023?
Reply : $6.8M
Confidence : 79.6%
Battle : RESOLVED
⚠ Conflicting sources detected — reply derived from most up-to-date
doc per battle cluster.
Sources retrieved
[0.820] 2023-Annual-Report-Revised (2024-04-03)
[0.589] Firm-Overview-2024 (2024-01-01)
Battle cluster resolved: saved '2023-Annual-Report-Revised' (2024-04-03),
discarded 1 older doc(s).
────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────
RESOLVED | Situation B — Coverage Battle
────────────────────────────────────────────────────────────────────
Reply : workers are not required to keep up
a set in-office schedule
Confidence : 78.0%
Battle : RESOLVED
Battle cluster resolved: saved 'HR-Coverage-November-2023' (2023-11-15),
discarded 1 older doc(s).
────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────
RESOLVED | Situation C — Technical Battle
────────────────────────────────────────────────────────────────────
Reply : 500 requests per minute
Confidence : 80.9%
Battle : RESOLVED
Battle cluster resolved: saved 'API-Reference-v2.0' (2023-09-20),
discarded 1 older doc(s).
────────────────────────────────────────────────────────────────────
Three questions. Three appropriate solutions. The arrogance scores are virtually equivalent to Section 1 — 78–81% — which underscores the unique level: confidence was by no means the sign that one thing had gone mistaken. It nonetheless isn’t. The one factor that modified is the structure.
What the Heuristics Can’t Catch
I need to be exact concerning the failure envelope, as a result of a way that understates its personal limitations isn’t helpful.
Paraphrased conflicts. The heuristics catch numerical variations and express contradiction tokens. They won’t catch “the service was retired” versus “the service is presently obtainable.” That may be a actual battle with no numeric distinction and no negation token. For these, a Pure Language Inference mannequin — cross-encoder/nli-deberta-v3-small at ~80 MB — can rating entailment versus contradiction between sentence pairs. That is the extra strong path described within the tutorial literature (Asai et al., 2023), and the ConflictDetector class is designed to be prolonged on the _pair_conflict_reason methodology for precisely this function.
Non-temporal conflicts. Recency-based decision is suitable for versioned paperwork and coverage updates. It’s not acceptable for professional opinion disagreements (the minority view could also be appropriate), cross-methodology information conflicts (recency is irrelevant), or multi-perspective queries (the place surfacing each views is the best response). In these instances, the ConflictReport information construction gives the uncooked materials to construct a special response — surfacing each claims, flagging for human overview, or asking the person for clarification.
Scale. Pair comparability is O(k²) in retrieved paperwork. For ok=3 that is trivial; for ok=20 it’s nonetheless high-quality. For pipelines retrieving ok=100 or extra, pre-indexing recognized battle pairs or cluster-based detection turns into essential.
The place the Analysis Group Is Taking This
What you could have seen here’s a sensible heuristic approximation of an issue that energetic analysis is attacking at a way more refined degree.
Cattan et al. (2025) launched the CONFLICTS benchmark — the primary particularly designed to trace how fashions deal with data conflicts in real looking RAG settings. Their taxonomy identifies 4 battle classes — freshness, conflicting opinions, complementary data, and misinformation — every requiring distinct mannequin behaviour. Their experiments present that LLMs incessantly fail to resolve conflicts appropriately throughout all classes, and that explicitly prompting fashions to motive about potential conflicts considerably improves response high quality, although substantial room for enchancment stays.
Ye et al. (2026) launched TCR (Clear Battle Decision), a plug-and-play framework that disentangles semantic relevance from factual consistency by way of twin contrastive encoders. Self-answerability estimation gauges confidence within the mannequin’s parametric reminiscence, and the ensuing scalar indicators are injected into the generator by way of light-weight soft-prompt tuning. Throughout seven benchmarks, TCR improves battle detection by 5–18 F1 factors whereas including solely 0.3% parameters.
Gao et al. (2025) launched CLEAR (Battle-Localized and Enhanced Consideration for RAG), which probes LLM hidden states on the sentence illustration degree to detect the place conflicting data manifests internally. Their evaluation reveals that data integration happens hierarchically and that conflicting versus aligned data displays distinct distributional patterns inside sentence-level representations. CLEAR makes use of these indicators for conflict-aware fine-tuning that guides the mannequin towards correct proof integration.
The constant discovering throughout all of this work matches what this experiment demonstrates straight: retrieval high quality and reply high quality are distinct dimensions, and the hole between them is bigger than the neighborhood has traditionally acknowledged.
The distinction between that analysis and this text is 220 MB and no authentication.
What You Ought to Really Do With This
1. Add a battle detection layer earlier than technology. The ConflictDetector class is designed to drop into an current pipeline on the level the place you assemble your context string. Even the 2 easy heuristics right here will catch the patterns that seem most frequently in enterprise corpora: restatements, coverage updates, versioned documentation.
2. Distinguish battle sorts earlier than resolving. A temporal battle (use the newer doc) is a special drawback from a factual dispute (flag for human overview) or an opinion battle (floor each views). A single decision technique utilized blindly creates new failure modes.
3. Log each ConflictReport. After every week of manufacturing visitors you’ll know the way usually your particular corpus generates conflicting retrieved units, which doc pairs battle most incessantly, and what question patterns set off conflicts. That information is extra actionable than any artificial benchmark.
4. Floor uncertainty if you can’t resolve it. The proper reply to an unresolvable battle is to not decide one and conceal the selection. The warning discipline in RAGResponse is there exactly to assist responses like: “I discovered conflicting data on this matter. The June 2023 coverage states X; the November 2023 replace states Y. The November doc is more moderen.”
Operating the Full Demo
# Full output with INFO logs
python rag_conflict_demo.py
# Demo output solely (suppress mannequin loading logs)
python rag_conflict_demo.py --quiet
# Run unit assessments with out downloading fashions
python rag_conflict_demo.py --test
# Plain terminal output for log seize / CI
python rag_conflict_demo.py --no-colorAll output proven on this article is unmodified output from a neighborhood Home windows machine operating Python 3.9+ in a digital surroundings. The code and output are absolutely reproducible by any reader with the listed dependencies put in.
The Takeaway
The retrieval drawback is essentially solved. Vector search is quick, correct, and well-understood. The neighborhood has spent years optimising it.
The context-assembly drawback isn’t solved. No one is measuring it. The hole between “appropriate paperwork retrieved” and “appropriate reply produced” is actual, it’s common, and it produces assured mistaken solutions with no sign that something went mistaken.
The repair doesn’t require a bigger mannequin, a brand new structure, or further coaching. It requires one further pipeline stage, operating on embeddings you have already got, at zero marginal latency.
The experiment above runs in about thirty seconds on a laptop computer. The query is whether or not your manufacturing system has the equal layer — and if not, what it’s silently answering mistaken proper now.
References
[1] Ye, H., Chen, S., Zhong, Z., Xiao, C., Zhang, H., Wu, Y., & Shen, F. (2026). Seeing via the battle: Clear data battle dealing with in retrieval-augmented technology. arXiv:2601.06842. https://doi.org/10.48550/arXiv.2601.06842
[2] Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2023). Self-RAG: Studying to retrieve, generate, and critique via self-reflection. arXiv:2310.11511. https://doi.org/10.48550/arXiv.2310.11511
[3] Cattan, A., Jacovi, A., Ram, O., Herzig, J., Aharoni, R., Goldshtein, S., Ofek, E., Szpektor, I., & Caciularu, A. (2025). DRAGged into conflicts: Detecting and addressing conflicting sources in search-augmented LLMs. arXiv:2506.08500. https://doi.org/10.48550/arXiv.2506.08500
[4] Gao, L., Bi, B., Yuan, Z., Wang, L., Chen, Z., Wei, Z., Liu, S., Zhang, Q., & Su, J. (2025). Probing latent data battle for devoted retrieval-augmented technology. arXiv:2510.12460. https://doi.org/10.48550/arXiv.2510.12460
[5] Jin, Z., Cao, P., Chen, Y., Liu, Okay., Jiang, X., Xu, J., Li, Q., & Zhao, J. (2024). Tug-of-war between data: Exploring and resolving data conflicts in retrieval-augmented language fashions. arXiv:2402.14409. https://doi.org/10.48550/arXiv.2402.14409
[6] Joren, H., Zhang, J., Ferng, C.-S., Juan, D.-C., Taly, A., & Rashtchian, C. (2025). Enough context: A brand new lens on retrieval augmented technology programs. arXiv:2411.06037. https://doi.org/10.48550/arXiv.2411.06037
[7] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., … & Kiela, D. (2020). Retrieval-augmented technology for knowledge-intensive NLP duties. arXiv:2005.11401. https://doi.org/10.48550/arXiv.2005.11401
[8] Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., & Hajishirzi, H. (2023). When to not belief language fashions: Investigating effectiveness of parametric and non-parametric recollections. arXiv:2212.10511. https://doi.org/10.48550/arXiv.2212.10511
[9] Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings utilizing Siamese BERT-networks. arXiv:1908.10084. https://doi.org/10.48550/arXiv.1908.10084
[10] Xu, R., Qi, Z., Guo, Z., Wang, C., Wang, H., Zhang, Y., & Xu, W. (2024). Information conflicts for LLMs: A survey. arXiv:2403.08319. https://doi.org/10.48550/arXiv.2403.08319
[11] Xie, J., Zhang, Okay., Chen, J., Lou, R., & Su, Y. (2023). Adaptive chameleon or cussed sloth: Revealing the habits of enormous language fashions in data conflicts. arXiv:2305.13300. https://doi.org/10.48550/arXiv.2305.13300
Full Supply Code: https://github.com/Emmimal/rag-conflict-demo
Fashions Used
Each fashions obtain mechanically on first run and cache regionally. No API key or HuggingFace authentication is required.
Disclosure
All code was written, debugged, and validated by the writer via a number of iterations of actual execution. All terminal output on this article is unmodified output from a neighborhood Home windows machine operating Python 3.9+ in a digital surroundings. The code and output are absolutely reproducible by any reader with the listed dependencies put in.
The writer has no monetary relationship with Hugging Face, deepset, or any organisation referenced on this article. Mannequin and library decisions had been made solely on the premise of measurement, licence, and CPU compatibility.
