RAG Is Not Machine Studying, and the ML Toolkit Solves the Flawed Downside

Machine studying solves issues the place the true reply is unknown and must be predicted. Will this buyer churn? What’s the likelihood this transaction is fraud? Is that this picture a cat? You don’t know the reply upfront. That’s why you prepare a mannequin. The mannequin learns from labeled examples, generalizes to new inputs, and produces a prediction. Efficiency is measured in combination, throughout 1000’s of take a look at instances, as a result of particular person predictions might be fallacious whereas the mannequin continues to be helpful general.

RAG solves a distinct drawback. The reply to “what’s the efficient date of this contract?” exists, written on web page one of many doc, or it doesn’t exist wherever. There’s nothing to foretell. The system both finds the reply within the doc and stories it faithfully, or it fails and may say so. Efficiency is binary on the query stage (received it or didn’t) even for those who measure combination charges throughout many questions.

These variations are concrete:

• In ML, “the mannequin was fallacious on 8% of instances” is a function of the system. You construct redundancy, downstream checks, human assessment for the borderline instances. In RAG, “the system gave a fallacious reply 8% of the time” is a bug. Every of these 8% has a selected trigger: the fallacious passage was retrieved, the proper passage was retrieved however the mannequin paraphrased it badly, the reply wasn’t within the corpus and the system made one up. They aren’t statistical noise to optimize on common. They’re individually fixable failures.
• In ML, you may’t usually inform why the mannequin received a selected case fallacious. That’s why explainability is a analysis area. In RAG, you may at all times inform. The retrieval logs which passages it returned. The generator noticed precisely these passages. If the reply is fallacious, you stroll the chain backward and discover the damaged hyperlink. There’s nothing hidden.
• In ML, the mannequin improves by coaching on extra knowledge. In RAG, the system improves by indexing higher, parsing extra rigorously, retrieving extra exactly, prompting extra clearly. None of that’s coaching. It’s engineering.

That distinction adjustments which instruments you attain for when one thing breaks.

The instances catalogued in Article 2 fall precisely right here: negation, actual identifiers, inside acronyms, sign dilution in lengthy context, topical proximity outranking the precise reply. None of these transfer once you swap embedding fashions or sweep chunk sizes. They aren’t bugs a mannequin can study its method out of, as a result of there is no such thing as a labeled sign saying “that is the proper line” for the mannequin to coach on. The repair is structural (query parsing, skilled key phrases, retrieval that is aware of the doc’s construction), and the following sections stroll by means of the three ML reflexes that decide the fallacious instrument as a substitute.

Three ML strategies get imported into RAG tasks by default: hyperparameter optimization, analysis datasets with prepare/take a look at splits, and feature-attribution explainability. Every is affordable inside ML. Every misfires right here.

The commonest framing goes one thing like this: chunk measurement, overlap, top-k, similarity threshold. These are hyperparameters, and it is best to optimize them the best way you optimize ML fashions, utilizing instruments like Optuna or Ray Tune. Run a sweep, plot the curves, decide the perfect configuration.

In these setups, top_k is the variety of passages the retriever retains, and similarity_threshold is the minimal cosine rating a passage should attain to qualify. The code under declares all 4 as numbers to optimize:

# What groups sometimes write (and why it is the fallacious exercise)
import optuna
def goal(trial):
    chunk_size    = trial.suggest_int("chunk_size", 100, 2000)
    chunk_overlap = trial.suggest_int("chunk_overlap", 0, 200)
    top_k         = trial.suggest_int("top_k", 1, 20)
    threshold     = trial.suggest_float("threshold", 0.5, 0.95)
    accuracy = run_rag_pipeline_and_score(
        chunk_size, chunk_overlap, top_k, threshold
    )
    return accuracy
examine = optuna.create_study(route="maximize")
examine.optimize(goal, n_trials=200)  # two weeks of compute later...

There’s a grain of fact right here. These variables do have an effect on retrieval high quality, and they’re value tuning. The difficulty begins with the phrase “hyperparameter,” which brings in a metaphor with hidden assumptions.

In machine studying, a hyperparameter controls how a mannequin learns: studying fee, regularization power, variety of layers. The mannequin itself is what adjustments throughout coaching; the hyperparameter shapes that change. In RAG, there is no such thing as a studying. The chunk measurement doesn’t management how one thing learns. It controls how a operate splits textual content, the identical method each time, no matter what you’ve fed it earlier than.

What appears to be like like a hyperparameter is a configuration alternative, the sort you’d make when configuring a search engine. The experience wanted to tune it nicely isn’t statistical optimization. It’s understanding the construction of your paperwork and the form of your questions. Chunk measurement of 512 tokens may match superbly on dense educational papers and disastrously on insurance coverage contracts the place a single clause spans 800 tokens and breaking it in half loses the conditional that provides the clause its that means. No grid search will inform you that. It’s good to learn your paperwork.

This is the reason groups who grid-search chunk measurement typically discover a “finest” worth that performs marginally higher on the take a look at set and identically on manufacturing knowledge. The optimum on the take a look at set was an artifact of the take a look at set, not a real enchancment within the underlying system. They’ve optimized a quantity, not solved an issue.

Frequent pitfall: A group working Optuna over chunk_size, top_k, and similarity_threshold for 2 weeks, ending up at chunk_size=487 with no concept why. The sincere reply to “why 487?” is “as a result of Optuna mentioned so.” That reply doesn’t survive an actual manufacturing failure, and it doesn’t generalize when the doc distribution shifts. A bit measurement of 500 chosen as a result of that’s roughly the scale of a paragraph on this corpus is extra defensible than 487 chosen as a result of a sweep landed there.

The correct exercise isn’t tuning numbers. It’s deciding structurally methods to chunk. By part? By paragraph? By the desk of contents entries? By query kind, with completely different chunkers for brief lookups vs lengthy clauses? Answered by taking a look at paperwork and questions, not by optimization curves.

There’s a deeper purpose chunk measurement resists optimization: by development, no single chunk measurement can serve each query. Take two questions on the identical insurance coverage contract:

• “What’s the efficient date?” The reply is one line, someplace on web page one. It needs a bit sufficiently small to pin down a single line exactly.
• “What are the exclusions of the coverage?” The reply may be one web page, or three pages, relying on how the insurer wrote it. It needs a bit massive sufficient to seize a whole part.

There isn’t any quantity that satisfies each. A bit measurement of 200 tokens chops the exclusions part into incoherent fragments. A bit measurement of 2000 tokens buries the efficient date in surrounding noise.

Looking for “the perfect chunk measurement” is subsequently not a tuning drawback. The framing itself is damaged: no single quantity can serve a distribution of questions whose solutions have completely different lengths.

You can, in precept, make chunk measurement reply to the query by coaching a small mannequin that predicts the proper chunker from the query’s options: classify the intent, regress over the anticipated reply size, output a technique. That will be machine studying utilized legitimately, on an issue the place one thing is being discovered.

However you don’t have to. You’ll be able to write the rule down. Take a look at a query and you’ll inform whether or not it asks for a date, a piece, or a comparability. So can a website skilled. So can ten traces of Python with hand-written situations over key phrases. The deeper purpose RAG isn’t machine studying is that, for a lot of the choices contained in the system, you already know the reply, or somebody in your group does. Machine studying is the instrument for issues the place no one is aware of the reply upfront.

The correct strategy is to cease in search of one chunk measurement and begin routing completely different query sorts to completely different retrieval methods:

# What to do as a substitute: route by query kind
def chunk_for_question(query: str, line_df, toc_df):
    intent = classify_intent(query)
    if intent == "point_lookup":          # "what's the efficient date?"
        return chunk_by_line(line_df)
    elif intent == "section_retrieval":   # "what are the exclusions?"
        return chunk_by_toc_section(line_df, toc_df)
    elif intent == "comparability":          # "evaluate clauses A and B"
        return chunk_by_full_section(line_df, toc_df)

The 2 code blocks above are your entire argument of this part. The primary runs Optuna over 4 numbers for 2 weeks and produces a worth no one can defend. The second makes one structural determination per query kind and produces a system whose conduct anybody can clarify.

Later articles develop methods to classify intent (Article 6, on query understanding) and the way the completely different retrieval strategies and granularities are carried out (Article 7, on retrieval). The purpose right here is simply that the exercise isn’t tuning, it’s routing.

The subsequent ML import is analysis technique. The reasoning goes: RAG, like all ML system, wants a correct analysis dataset: questions paired with anticipated solutions, cut up into prepare and take a look at units, scored with precision and recall. Frameworks like RAGAS have made this much more tempting, providing metrics for faithfulness, reply relevancy, and context recall that look satisfyingly ML-ish.

Analysis is helpful. The problem isn’t whether or not to judge. It’s what the metrics imply. In machine studying, analysis tells you whether or not a mannequin has generalized from coaching knowledge to unseen examples. The prepare/take a look at cut up exists since you wish to detect overfitting: a mannequin that memorized the coaching set relatively than studying a transferable sample.

In RAG, there may be nothing to generalize. Overfitting (when a mannequin memorizes coaching examples relatively than studying a sample that transfers to new knowledge) can not occur right here: the system doesn’t change between queries. The retriever computes the identical cosine distances each time. The generator follows the identical immediate template. There isn’t any mannequin adjusting to knowledge.

What analysis measures in RAG is three issues, all of that are protection and high quality questions, not statistical generalization:

• Does my corpus include the reply? If not, the system can’t discover it. This can be a content material query, not a mannequin query.
• Does my retriever discover the proper passage? If the reply is within the corpus however the retriever missed it, the system fails. This can be a search query.
• Does my generator keep devoted to what was retrieved? If the proper passage was retrieved however the mannequin paraphrased it incorrectly or hallucinated extras, the system fails. This can be a era self-discipline query.

Every one factors to a selected repair. Mixing them up beneath an combination “accuracy” rating loses data. A 75% accuracy from “corpus is lacking 25% of the documented matters” calls for completely different motion than a 75% accuracy from “retriever misses the proper passage 25% of the time.” The primary requires ingesting extra paperwork. The second requires fixing the retriever. An combination metric that treats them the identical hides the diagnostic.

This additionally explains why groups utilizing RAGAS-style frameworks generally report nice metrics on a held-out take a look at set after which watch the system fail in manufacturing. The take a look at set lined matters the place the corpus had solutions and the retriever occurred to seek out them. Manufacturing has questions whose solutions are usually not within the corpus in any respect, and the system both hallucinates or fails to say “not discovered.” The metric was excessive on the take a look at set as a result of the take a look at set was pleasant. The system isn’t damaged. The analysis was.

What you want to consider, damaged down by query kind, takes about ten traces:

# Retrieval recall, per query, per intent
def evaluate_retrieval(reference_set, retrieve_fn):
    rows = []
    for ref in reference_set:
        retrieved_lines = retrieve_fn(ref.query)
        recall = len(set(retrieved_lines) & set(ref.expected_lines)) / len(ref.expected_lines)
        rows.append({
            "query": ref.query,
            "intent":   ref.intent,
            "recall":   recall,
            "hit":      recall > 0,
        })
    return pd.DataFrame(rows)
# At all times break down by query kind, by no means simply an combination
df.groupby("intent")["hit"].imply()
# point_lookup        0.92
# section_retrieval   0.41   <-- that is the true drawback
# comparability          0.55

A single combination accuracy of 63% would have hidden the disaster on section_retrieval. The per-intent breakdown reveals it immediately. Recall right here means: on questions the place the reply exists within the corpus, did the retriever discover the proper passage? Grouping by intent (point_lookup, section_retrieval, …) exhibits which type of query fails, and subsequently which half of the pipeline to repair.

RAG has two analysis surfaces with very completely different shapes.

The retrieval floor is a search drawback: did the proper passage land in entrance of the mannequin? Measuring this implies checking, on a reference set of questions, whether or not the related traces or pages had been retrieved in any respect. The metric is recall on the stage you care about (recall at line, at web page, at part) and it’s particular to your corpus. No person else can run this analysis for you. Your corpus is exclusive. That is the place the majority of analysis effort belongs.

The era floor is completely different. As soon as the proper passage has been retrieved, the query turns into: did the mannequin produce a devoted reply, in the proper format, with correct citations, and a clear “not discovered” when the passage didn’t include the reply? A few of this you do consider your self, however a big half is already evaluated by the LLM distributors. OpenAI, Anthropic, and Mistral spend huge assets testing whether or not their fashions comply with JSON schemas, refuse to invent, and respect immediate directions. These are the scale on which they enhance their fashions. As a RAG builder, you’re not coaching the generator. You’re consuming it. If the mannequin fails badly at returning structured JSON or stays untrue to its inputs, you’ll discover inside an hour of integration. That’s not a metric to arrange; it’s a sanity verify that’s both apparent or nice.

What this implies in follow: most of your analysis time ought to go into retrieval (which is corpus-specific and solely you are able to do it), not into era (which is generally the seller’s drawback, and which exhibits apparent failures quick). Groups that spend weeks constructing elaborate era analysis suites are normally pushing aside the more durable retrieval work that may enhance the consequence.

Going additional: Evaluating Your System (later within the sequence) walks by means of methods to construct a reference set in your particular corpus, the 4 metrics that matter, and why per-question-type metrics are important whereas combination metrics are deceptive.

Machine studying has its personal toolkit for explainability. SHAP values to attribute predictions to options. LIME for native approximations of advanced fashions. Consideration visualization for transformers. When folks begin asking for RAG explainability (“why did the system give this reply?”) they naturally flip to those instruments. They wish to rating retrieval relevance, weight doc contributions, visualize which tokens influenced the output.

The irony is that RAG is extra explainable by design than most ML fashions. There’s no want for SHAP. There’s no opacity to crack open. The system retrieved these particular passages from these particular sources, and the reply was constructed on prime of them. That is the reason. It’s documentary, not statistical.

This factors to a deeper asymmetry between machine studying and RAG. In machine studying, the human has instinct however can not quantify. Ask who survived the Titanic and folks say wealth, age, class: none fallacious, none exact. The mannequin has no such doubt: match a choice tree and the foundation cut up is intercourse, the following lower is an actual age threshold no one would have guessed, then class. Each cut up is a quantity instinct alone couldn’t have produced. The mannequin exists to place these numbers down.

For textual content knowledge, the route reverses. The person can learn the supply. A lawyer scanning a contract sees the situations, the exceptions, the dates. A compliance officer reads a coverage and is aware of whether or not a conduct breaches it. The textual content doesn’t disguise its that means, and the skilled is already a fluent reader.

There are exceptions: sarcasm and irony are the traditional ones, the place fashionable LLMs generally catch what a literal reader misses. However in enterprise contexts the person is the area skilled.

The mannequin isn’t there to elucidate the textual content. It’s there to do the studying at corpus scale, and a quotation is sufficient to let the skilled confirm any reply in seconds.

When a person asks “why this reply?”, the proper response isn’t a heatmap of consideration weights or a function attribution rating. It’s: “I checked out pages 12, 47, and 89 of this contract. Right here’s the precise textual content I used. The reply follows from that textual content.” If the person disagrees with the reply, they’ll learn the supply themselves and choose. They don’t want an explainability framework. They want a quotation.

The fifty-line pipeline from Article 1 already confirmed this. The immediate requested the mannequin to return the beginning and finish line numbers (with their pages) alongside the reply, in a structured JSON; the annotator then highlighted these actual traces on the PDF. No SHAP, no LIME, no consideration visualization, no specialised observability platform. The “rationalization” was a aspect product of how the immediate was written. The quotation is a part of the reply, not an evaluation layer added on prime.

The hint is the reason. Studying it requires no interpretation, simply studying.

Importing ML explainability into RAG is fixing an issue that doesn’t exist. SHAP on a retrieval rating is utilizing a scalpel to open a mailbox. The retrieval rating is already a quantity you computed on inputs you may learn. There’s nothing to attribute that you just don’t already see.

The deeper failure of the ML-explainability framing is that it makes you concentrate on the fallacious factor. You begin making an attempt to elucidate why a selected passage scored increased than one other in vector area, a near-impossible query that doesn’t matter. What issues is whether or not the proper passage was retrieved in any respect, and whether or not the reply faithfully displays it. These are questions you may reply by studying the logs and the supply. No tooling wanted.

When you cease treating RAG as ML, two issues change. The day-to-day instruments, metrics and folks reorganize round search relatively than coaching. And a deeper query (the place the intelligence sits) strikes from the mannequin to the group. Each come from the identical framing.

Three concrete issues change.

The instruments change: You don’t want PyTorch, or a coaching cluster, or hyperparameter optimization frameworks for the system itself. You want a great parser, a versatile retriever, cautious immediate engineering, and structured logging of every part that occurs. The parts that are ML (the embedding mannequin, the LLM) you devour as providers. They’re commodity inputs, not belongings you construct or prepare.

The metrics change: Mixture accuracy offers method to per-failure-mode metrics: retrieval recall (did we discover the proper passage?), reply faithfulness (did the mannequin follow it?), extraction accuracy (when extracting structured knowledge, did the values match?), not-found fee (when the reply isn’t within the corpus, did we are saying so cleanly?). Every measures one thing particular, every maps to a selected a part of the pipeline you may repair.

The folks change: A pure ML group making an attempt to ship a RAG system typically misses what makes it work, and what makes it fail. The talents that matter most are software program engineering (the system has many shifting elements that have to compose cleanly), area experience (somebody has to know what a great reply to a website query even appears to be like like), and data retrieval instinct (somebody has to suppose like a search engine designer, not a mannequin coach). ML experience is helpful, nevertheless it’s not the dominant ability. A group of ML researchers and no area skilled will produce a superbly tuned system that misses the purpose. A group with one ML-aware engineer, two software program engineers, and one area skilled will normally outperform it.

The shift in folks factors to a deeper query: the place does the intelligence of the system dwell?

In an ML system the intelligence lives within the mannequin. The mannequin holds the patterns. The group feeds it coaching knowledge and tunes the loss operate. In a RAG system the intelligence lives within the group. The lawyer is aware of which clauses to take a look at first. The underwriter is aware of what “deductible” means, and which web page normally carries it. The compliance officer is aware of which regulation applies to which product. None of that lives contained in the embedding mannequin. None of it comes out of a hyperparameter sweep. It already lives within the heads of people that have learn these paperwork for years.

Watch an underwriter open a brand new coverage. She doesn’t learn it linearly. She jumps to the exclusions part first as a result of she’s learn 5 hundred of those and is aware of that’s the place the lure normally lives. She checks the schedule of advantages for the deductibles and ceilings. She checks the territory clause. Three minutes in, she has a clearer view of the contract than any embedding mannequin would produce on a thousand of these contracts. That behavior is what the system has to amplify.

The job of an enterprise RAG system is to amplify that experience at scale, not change it. What that appears like relies on the brick.

Parsing comes first. If the parser turns a contract’s PDF into scrambled textual content, no downstream cleverness recovers it. If the doc has a working desk of contents, the parser has to extract it cleanly, as a result of the TOC is what the skilled depends on to navigate. When a doc has no TOC in any respect (scanned faxes, slide decks exported to PDF, outdated typewritten insurance policies), reconstructing one turns into a job in itself, typically extra helpful than any retrieval tweak.

Query understanding carries the group’s vocabulary throughout the hole between how a person phrases a query and the way the doc writes the reply. The pilot person sorts kettle, the contract says small electrical equipment. The compliance officer sorts knowledge breach, the coverage says unauthorized disclosure of private data. The skilled is aware of the mapping. The query parser turns that mapping right into a lookup desk: translations throughout languages, spelling variants, plural types, inside acronyms. None of it’s discovered from knowledge, it’s dictated by the skilled and written down.

Retrieval amplifies what the skilled already does by hand. The skilled searches key phrases; that half is already straightforward. What the skilled can not do at scale is run regex patterns over 1000’s of pages, verify whether or not two phrases co-occur inside the identical paragraph, or mix boolean situations throughout the entire corpus. The retriever does that work quick, then arms candidates again so the skilled can confirm.

Technology does the 2 issues the skilled would in any other case do by hand: cite the precise passage that helps the reply, and format the uncooked worth into one thing usable. The string 3455434 on the web page turns into €3,455,434 within the reply. 20260516 turns into Could 16, 2026. thirty days from the date of the loss stays verbatim, with a quotation again to the clause so the skilled can confirm in a single click on.

Articles 5, 6, 7, and eight develop every brick in flip: the parser that extracts TOC construction, the skilled dictionary that maps vocabulary, the TOC-aware retriever, the typed-answer generator. Identical precept each time: decide up a chunk of human experience and transfer the repetitive half to the machine.

That is additionally why the sequence is cautious with autonomous brokers. It prefers key phrase retrieval to embedding similarity by default. It treats reranker tuning as a final resort. Every of these defaults assumes there is no such thing as a skilled to seek the advice of. In enterprise contexts the skilled is at all times there. The system ought to take heed to them.

In the event you work in a setting with no skilled, with unbounded questions, with very completely different paperwork, this sequence is not going to be your finest information. Common-purpose retrieval and autonomous brokers are a greater match there.

A helpful method to image RAG is as a search engine, plus an LLM that writes the reply. Two elements, every with a transparent job, every with its personal method of breaking.

The search engine retrieves passages from paperwork. Given a query, return the traces, paragraphs, or sections more than likely to include the reply. This can be a pure search drawback: selectivity, recall, rating. A long time of knowledge retrieval concept apply. The truth that a part of it makes use of neural embeddings doesn’t change its nature; embedding similarity is only one rating sign amongst a number of.

The LLM takes a passage and a query and produces a natural-language reply with a quotation. The LLM doesn’t discover the reply. The search engine already did that. The LLM writes the reply from a passage that’s been positioned in entrance of it. It’s nearer to a translator or a scribe than to an oracle.

Mapping again to the 4 bricks from Article 1: parsing, query understanding, and retrieval collectively make up the search engine; era is the LLM. The brick view is the operational one (one field of code per brick); the two-part view is the psychological mannequin you carry in your head when one thing goes fallacious.

The 2 elements fail in several methods, and the prognosis begins on the seam between them. Pull the hint from a failing question: had been the retrieved passages in entrance of the mannequin, and did they include the reply?

If the reply wasn’t within the retrieved passages, the search engine is the offender, and the repair is upstream. Was the proper web page corrupted by the parser (OCR errors, multi-word phrases cut up throughout traces, two-column interleaving)? Did the query parser miss a synonym the skilled vocabulary ought to have expanded? Did the retrieval mechanism rank the proper web page out of top_k, or break on punctuation that wanted a regex? Or is the related doc simply not within the corpus? 4 very completely different fixes, all upstream. “Tune the retriever” is meaningless till you’ve localized which one. The identical 4 bricks that amplify the skilled when working (part 3.3) break in their very own methods right here, every with its personal deep-dive article (Articles 5, 6, 7).

If the reply was within the retrieved passages however the response is fallacious, the LLM is the offender, and the repair is downstream. Frequent patterns: the mannequin paraphrased and misplaced a conditional, returned the uncooked 3455434 as a result of the schema left the reply free-form, cited the fallacious line numbers, invented a worth not within the passage, or produced a solution when it ought to have mentioned “not discovered”. 5 era bugs, 5 completely different fixes, all within the immediate, schema, or post-validation layer (Article 8). None of them get higher by tuning the retriever.

A group at a mid-size enterprise is given six months to ship a RAG system over just a few thousand inside paperwork. They begin by constructing an analysis dataset of 500 questions, splitting it 70/30 into prepare and take a look at. They arrange Optuna to comb chunk measurement, overlap, top-k, and similarity threshold. The primary sweep takes every week of compute, comes again with a “finest” configuration, and the group ships it for inside testing.

The pilot customers complain instantly. The system solutions fluently however is fallacious half the time on questions that the evaluators clearly know: questions on particular clauses, particular dates, particular numerical limits. The group’s response is to develop the analysis dataset, run one other sweep, fine-tune the embedding mannequin on artificial question-document pairs, and add a reranker. Three extra months go by. Manufacturing accuracy doesn’t transfer.

What was fallacious: the parser was treating scanned pages with degraded OCR layers as in the event that they had been native textual content. About 30% of the corpus was successfully unreadable, however the group’s analysis set occurred to be drawn from the readable 70%. No quantity of chunk measurement optimization, embedding fine-tuning, or reranker integration might repair it: a 3rd of the paperwork had been producing rubbish. A two-day funding in checking every web page (the work of Article 5, on parsing) would have caught this on day one.

The group had spent six months in ML mode (sweeping hyperparameters, rising analysis units, fine-tuning fashions) when the repair was a parser change.

This story is composite, however each aspect of it has occurred in actual tasks. The sample is constant: ML reflexes drive the group towards optimization actions that really feel productive, whereas the structural issues sit untouched within the parser, the corpus, or the not-found logic. The primary intuition on a struggling RAG system shouldn’t be “let’s tune”. It needs to be “let’s hint what occurs to a failing question, finish to finish, and discover the damaged hyperlink.”

RAG appears to be like like machine studying. The resemblance is shallow. The reply exists within the doc or it doesn’t. There isn’t any statistical generalisation, no studying curve, no prepare/take a look at cut up that maps to actual failures. The correct framing is search engine meeting: a search engine plus an LLM, two elements you may repair independently, with per-failure-mode metrics changing combination accuracy.

The price of holding on to the ML framing just isn’t mental. It’s six months of cautious work on the fallacious drawback. Article 4 turns the proper framing right into a working diagnostic: RAG issues sit on a grid of doc complexity by query management, and every cell requires a distinct stack.