Agentic RAG: Let the Agent Search

software we construct is a RAG app.

The recipe is straightforward: chunk, embed, retrieve, then reply.

It seems clear on paper. However when you apply it to actual instances, issues get messy in a short time: similarity search finds comparable wordings however not essentially helpful chunks, the appropriate proof by no means reveals up within the retrieved context because it ranks too low, or essential context could also be cut up throughout chunk boundaries.

With inadequate context, the LLM has little room to get better.

So, how about we make retrieval iterative? What if the mannequin can search, learn, determine whether or not it has sufficient proof, and search once more when wanted? In all probability we don’t even want the vector embeddings within the first place.

That’s the premise of agentic RAG.

On this put up, we’ll construct a mini agentic RAG workflow with the OpenAI Brokers SDK. We’ll study how the agent iteratively searches, reads, and grounds its reply.

On the finish, we’ll take a step again and briefly talk about the concerns for constructing a sensible agentic RAG resolution.

1. Case Research: Answering a Coverage Query with Agentic RAG

For our case research, we’ll construct a coverage RAG agent over an organization coverage doc assortment.

1.1 Curating The Doc Assortment

Right here, I created six artificial firm coverage docs. They’re all markdown recordsdata. Each has a title, an efficient date, a brief abstract, and the coverage textual content.

To be lifelike, these docs cowl 6 frequent firm coverage areas:

1. approval_matrix.md, containing approval ranges for frequent enterprise journey selections, efficient on July 1, 2025.
2. conference_guidelines.md, containing guidelines for attending exterior occasions, efficient on Could 15, 2025.
3. faq.md, containing casual solutions to frequent journey questions, efficient on September 1, 2025.
4. policy_updates_2026.md, containing updates to lodging, convention journey, and approval timing for 2026, efficient on January 1, 2026.
5. remote_work_policy.md, containing guidelines for distant work, efficient on February 1, 2026.
6. travel_policy.md, containing normal journey reserving guidelines for flights, lodging, meals, and transportation, efficient on March 1, 2025.

We made it intentional that the reply to a coverage query could not stay in a single doc. This permits us to see the specified agentic conduct.

Yow will discover the complete artificial paperwork and the agentic RAG implementation pocket book right here.

1.2 Defining The Agent

Subsequent, we configure the agent. For that, we use the OpenAI Brokers SDK.

At a excessive stage, the agent is simply this:

# pip set up openai-agents
from brokers import Agent

agent = Agent(
    identify="Coverage analysis assistant",
    directions=INSTRUCTIONS,
    mannequin="gpt-5.4",
    instruments=[list_docs, search_docs, read_doc],
)

Two components we have to undergo: the agent instruction, and the instruments it has entry to.

First, the instruction. That is the place we outline the specified search conduct:

# Be aware: This instruction is iterated with AI
INSTRUCTIONS = """
[Role]
You're a cautious inside coverage analysis assistant.

[Research behavior]
Reply worker coverage questions utilizing the doc instruments.
Discover sufficient related proof to assist the reply.
Maintain conclusions grounded within the coverage paperwork.

[Expected output]
Give a direct reply first.
Then briefly clarify the proof.
Cite the doc filenames used for every essential declare.
""".strip()

For this case research, we require that the agent can solely contact the docs through three pre-defined instruments:

The primary one is a device that provides the agent a fast overview of what paperwork exist:

@function_tool
def list_docs() -> checklist[dict]:
    """Record out there coverage paperwork with out returning their physique textual content."""
    return [
        {
            "doc_name": doc["doc_name"],
            "title": doc["title"],
            "efficient": doc["effective"],
            "abstract": doc["summary"],
        }
        for doc in docs.values()
    ]

The second device is a keyword-search device. We preserve it easy right here: every doc is cut up into paragraph chunks, and every question is matched towards these chunks by token overlap:

@function_tool
def search_docs(question: str) -> checklist[dict]:
    """Search coverage paperwork and return the highest three brief snippets."""
    query_tokens = tokenize(question)
    scored = []

    for chunk in chunks:
        rating = len(query_tokens & chunk["tokens"])
        if rating:
            scored.append((rating, chunk))

    scored.type(key=lambda merchandise: merchandise[0], reverse=True)

    outcomes = []
    for rating, chunk in scored[:3]:
        snippet = chunk["text"].change("n", " ")
        if len(snippet) > 420:
            snippet = snippet[:417].rstrip() + "..."
        outcomes.append({
            "doc_name": chunk["doc_name"],
            "title": chunk["title"],
            "part": chunk["section"],
            "snippet": snippet,
            "rating": spherical(rating, 2),
        })

    return outcomes

The final device is what permits the agent to open one doc by filename:

@function_tool
def read_doc(doc_name: str) -> str:
    """Learn one coverage doc by filename."""
    if doc_name not in docs:
        legitimate = ", ".be a part of(sorted(docs))
        return f"Unknown doc: {doc_name}. Legitimate paperwork: {legitimate}"

    return docs[doc_name]["text"]

That’s the complete RAG agent.

1.3 Operating One Coverage Query

Now we take a look at the agent with one concrete query:

We run the agent with:

from brokers import Runner

end result = await Runner.run(agent, PROMPT, max_turns=12)

The agent produced the appropriate reply: sure, the worker can e book the official convention lodge if there’s a sensible enterprise cause. It bought that info from conference_guidelines.md.

For the approval half, the agent first recognized that approval is required because the lodge is above the traditional cap. Then it gave the corresponding approval situations. The agent used travel_policy.md, approval_matrix.md, and policy_updates_2026.md to assist its reply, which is precisely what we might anticipate.

The extra fascinating half is the hint, from which we are able to learn the way the agent thinks. We are able to present the hint within the following approach:

for merchandise in end result.new_items:
    print(kind(merchandise).__name__, merchandise)

end result.new_items incorporates the intermediate device calls and power outputs produced by the agent. In my run, I can see that the agent first known as search_docs() with key phrases like convention lodge, lodge cap, approval, and Berlin. Then, it known as list_docs() to examine the out there coverage paperwork. After that, it opened the related recordsdata with read_doc(). Solely then did it produce the ultimate reply.

That is precisely the agentic loop we needed to see.

3. What to Resolve Earlier than Constructing Agentic RAG

The case research we simply went by solely scratched the floor. To actually construct a sensible agentic RAG resolution, based mostly on my expertise, I recommend you reply the next 5 questions:

Q1: How a lot freedom ought to the agent have?

One frequent possibility is precisely what we’ve achieved within the earlier case research: we uncovered a few rigorously curated instruments, and the agent is barely allowed to make use of these instruments to do the investigation. That is simple when it comes to controlling, testing, and auditing.

However we are able to additionally give the agent broader entry, similar to shell and file system. This fashion, the agent can immediately run scripts to look and examine recordsdata, and perhaps even do additional knowledge processing to generate helpful artifacts, all by itself.

This sample will be far more highly effective, nevertheless it additionally will increase threat and makes conduct more durable to foretell.

So for many RAG purposes, I’d begin with curated instruments first, and solely add shell/file-system entry when the duty complexity justifies it.

Q2: Ought to the agent search uncooked textual content solely?

Most RAG initiatives would possibly begin with plain textual content like PDFs, wiki pages, manuals, and so on. That’s high-quality.

However in apply, we are able to usually make retrieval simpler by deriving a data layer on prime of the uncooked texts.

These derived data artifacts will be doc metadata, summaries, cross-document hyperlinks, or we are able to go additional and implement a correct data graph.

These derived data artifacts assist the agent navigate the corpus, whereas the uncooked texts stay because the supply of fact.

Q3: Will we nonetheless want embeddings?

Agentic RAG doesn’t essentially imply embeddings are gone.

Vector embeddings are nonetheless an environment friendly solution to discover semantically related texts, and it usually outperforms a pure key phrase search technique.

In agentic RAG, what modified primarily is that the retrieval turns into an “motion” the agent can take. Underneath this framing, “motion” can nonetheless be powered by an embedding-based retriever, a keyword-based one, or perhaps a hybrid one.

So embeddings can nonetheless be helpful. They’re only one potential solution to energy the agent’s search device.

This fall: Ought to one agent deal with all the pieces?

The only agentic RAG setup is only one agent that does the search, learn, and reply.

However as the duty will get extra complicated, you would possibly wish to cut up the work amongst a number of brokers. Extra concretely, you would possibly must undertake a multi-agent technique.

You possibly can cut up the work by position. For instance, the planner-retriever-writer cut up, the place the planner decides what proof is required, the retriever collects it, and the author produces the ultimate reply through the use of the collected proof.

You may as well cut up by supply kind, the place every agent is provided with custom-made instruments and focuses on one particular kind of supply.

Simply take note: A multi-agent setup provides coordination complexity, and there’s no assure that it’s going to carry out higher than a single-agent setup. Empirical testing is essential.

Q5: Ought to we at all times use agentic RAG?

Possibly not at all times.

Simply because agentic RAG turns into a classy subject doesn’t essentially imply you need to at all times default to it.

Agentic RAG offers extra flexibility, however that comes with prices. That value shouldn’t be solely about latency or token value, but in addition much less predictable agent conduct.

All the time begin easy, then add agentic loops when the query truly wants iterative retrieval.