Why CrewAI’s Supervisor-Employee Structure Fails — and Easy methods to Repair It

is without doubt one of the most promising functions of LLMs, and CrewAI has shortly turn into a well-liked framework for constructing agent groups. However one in all its most vital options—the hierarchical manager-worker course of—merely doesn’t perform as documented. In actual workflows, the supervisor doesn’t successfully coordinate brokers; as a substitute, CrewAI executes duties sequentially, resulting in incorrect reasoning, pointless instrument calls, and intensely excessive latency. This concern has been highlighted in a number of on-line boards with no clear decision.

On this article, I display why CrewAI’s hierarchical course of fails, present the proof from precise Langfuse traces, and supply a reproducible pathway to make the manager-worker sample work reliably utilizing customized prompting.

Multi-agent Orchestration

Earlier than we get into the main points, allow us to perceive what orchestration means in an agentic context. In easy phrases, orchestration is managing and coordinating a number of inter-dependent duties in a workflow. However have’nt workflow administration instruments (eg; RPA) been out there eternally to just do that? So what modified with LLMs?

The reply is the power of LLMs to grasp which means and intent from pure language directions, simply as folks in a crew would. Whereas earlier workflow instruments had been rule-based and inflexible, with LLMs functioning as brokers, the expectation is that they’ll be capable of perceive the intent of the consumer’s question, use reasoning to create a multi-step plan, infer the instruments for use, derive their inputs within the right codecs, and synthesize all of the completely different intermediate leads to a exact response to the consumer’s question. And the orchestration frameworks are supposed to information the LLM with acceptable prompts for planning, tool-calling, producing response and many others.

Among the many orchestration frameworks, CrewAI, with its pure language based mostly definition of duties, brokers and crews relies upon probably the most on the LLM’s capacity to grasp language and handle workflows. Whereas not as deterministic as LangGraph (since LLM outputs can’t be totally deterministic), it abstracts away many of the complexity of routing, error dealing with and many others into easy, user-friendly constructs with parameters, which the consumer can tune for acceptable conduct. So it’s a good framework for creating prototypes by product groups and even non-developers.

Besides that the manager-worker sample doesn’t work as supposed…

For instance, let’s take a use-case to work with. And in addition consider the response based mostly on the next standards:

1. High quality of orchestration
2. High quality of ultimate response
3. Explainability
4. Latency and utilization price

Use Case

Take the case the place a crew of buyer help brokers resolve technical or billing tickets. When a ticket comes, a triage agent categorizes the ticket, then assigns to the technical or billing help specialist for decision. There’s a Buyer Assist Supervisor who coordinates the crew, delegates duties and validates high quality of response.

Collectively they are going to be fixing queries similar to:

1. Why is my laptop computer overheating?
2. Why was I charged twice final month?
3. My laptop computer is overheating and likewise, I used to be charged twice final month?
4. My bill quantity is inaccurate after system glitch?

The primary question is only technical, so solely the technical help agent needs to be invoked by the supervisor, the second is Billing solely and the third and fourth ones require solutions from each technical and billing brokers.

Let’s construct this crew of CrewAI brokers and see how effectively it really works.

Hierarchical Course of

In accordance with CrewAI documentation ,“adopting a hierarchical strategy permits for a transparent hierarchy in activity administration, the place a ‘supervisor’ agent coordinates the workflow, delegates duties, and validates outcomes for streamlined and efficient execution. “ Additionally, the supervisor agent might be created in two methods, routinely by CrewAI or explicitly set by the consumer. Within the latter case, you will have extra management over directions to the supervisor agent. We’ll attempt each methods for our use case.

CrewAI Code

from crewai import Agent, Crew, Course of, Activity, LLM
from dotenv import load_dotenv
import os
from observe import * # Langfuse hint

load_dotenv()
verbose = False
max_iter = 4

API_VERSION = os.getenv(API_VERSION')
# Create your LLM
llm_a = LLM(
    mannequin="gpt-4o",
    api_version=API_VERSION,
    temperature = 0.2,
    max_tokens = 8000,
)

# Outline the supervisor agent
supervisor = Agent(
    position="Buyer Assist Supervisor",
    aim="Oversee the help crew to make sure well timed and efficient decision of buyer inquiries. Use the instrument to categorize the consumer question first, then resolve the following steps.Syntesize responses from completely different brokers if wanted to supply a complete reply to the shopper.",
    backstory=( """
        You don't attempt to discover a solution to the consumer ticket {ticket} your self. 
        You delegate duties to coworkers based mostly on the next logic:
        Word the class of the ticket first by utilizing the triage agent.
        If the ticket is categorized as 'Each', at all times assign it first to the Technical Assist Specialist, then to the Billing Assist Specialist, then print the ultimate mixed response. Make sure that the ultimate response solutions each technical and billing points raised within the ticket based mostly on the responses from each Technical and Billing Assist Specialists.
        ELSE
        If the ticket is categorized as 'Technical', assign it to the Technical Assist Specialist, else skip this step.
        Earlier than continuing additional, analyse the ticket class. Whether it is 'Technical', print the ultimate response. Terminate additional actions.
        ELSE
        If the ticket is categorized as 'Billing', assign it to the Billing Assist Specialist.
        Lastly, compile and current the ultimate response to the shopper based mostly on the outputs from the assigned brokers.
        """
    ),
    llm = llm_a,
    allow_delegation=True,
    verbose=verbose,
)

# Outline the triage agent
triage_agent = Agent(
    position="Question Triage Specialist",
    aim="Categorize the consumer question into technical or billing associated points. If a question requires each points, reply with 'Each'.",
    backstory=(
        "You're a seasoned knowledgeable in analysing intent of consumer question. You reply exactly with one phrase: 'Technical', 'Billing' or 'Each'."
    ),
    llm = llm_a,
    allow_delegation=False,
    verbose=verbose,
)

# Outline the technical help agent
technical_support_agent = Agent(
    position="Technical Assist Specialist",
    aim="Resolve technical points reported by clients promptly and successfully",
    backstory=(
        "You're a extremely expert technical help specialist with a robust background in troubleshooting software program and {hardware} points. "
        "Your major accountability is to help clients in resolving technical issues, guaranteeing their satisfaction and the sleek operation of their merchandise."
    ),
    llm = llm_a,
    allow_delegation=False,
    verbose=verbose,
)

# Outline the billing help agent
billing_support_agent = Agent(
    position="Billing Assist Specialist",
    aim="Deal with buyer inquiries associated to billing, funds, and account administration",
    backstory=(
        "You might be an skilled billing help specialist with experience in dealing with buyer billing inquiries. "
        "Your primary goal is to supply clear and correct info concerning billing processes, resolve fee points, and help with account administration to make sure buyer satisfaction."
    ),
    llm = llm_a,
    allow_delegation=False,
    verbose=verbose,
)

# Outline duties
categorize_tickets = Activity(
    description="Categorize the incoming buyer help ticket: '{ticket} based mostly on its content material to find out whether it is technical or billing-related. If a question requires each points, reply with 'Each'.",
    expected_output="A categorized ticket labeled as 'Technical' or 'Billing' or 'Each'. Don't be verbose, simply reply with one phrase.",
    agent=triage_agent,
)

resolve_technical_issues = Activity(
    description="Resolve technical points described within the ticket: '{ticket}'",
    expected_output="Detailed options supplied to every technical concern.",
    agent=technical_support_agent,
)

resolve_billing_issues = Activity(
    description="Resolve billing points described within the ticket: '{ticket}'",
    expected_output="Complete responses to every billing-related inquiry.",
    agent=billing_support_agent,
)

# Instantiate your crew with a customized supervisor and hierarchical course of
crew_q = Crew(
    brokers=[triage_agent, technical_support_agent, billing_support_agent],
    duties=[categorize_tickets, resolve_technical_issues, resolve_billing_issues],
    # manager_llm = llm_a, # Uncomment for auto-created supervisor
    manager_agent=supervisor, # Remark for auto-created supervisor
    course of=Course of.hierarchical,
    verbose=verbose,
)

As is clear, this system displays the crew of human brokers. Not solely is there a manger, triage agent, technical and billing help agent, however the CrewAI objects similar to Agent, Activity and Crew are self-evident of their which means and simple to visualise. One other commentary is that there’s little or no python code and many of the reasoning, planning and conduct is pure language based mostly which relies upon upon the power of the LLM to derive which means and intent from language, then cause and plan for the aim.

A CrewAI code due to this fact, scores excessive on ease of growth. It’s a low-code method of making a move shortly with many of the heavy-lifting of the workflow being achieved by the orchestration framework quite than the developer.

How effectively does it work?

As we’re testing the hierarchical course of, the method parameter is ready to Course of.hierarchical within the Crew definition. We will attempt completely different options of CrewAI as follows and measure efficiency:

1. Supervisor agent auto-created by CrewAI
2. Utilizing our customized supervisor agent

1. Auto-created supervisor agent

Enter question: Why is my laptop computer overheating?

Right here is the Langfuse hint:

The important thing observations are as follows:

1. First the output is “Primarily based on the supplied context, it appears there’s a misalignment between the character of the problem (laptop computer overheating) and its categorization as a billing concern. To make clear the connection, it will be vital to find out if the shopper is requesting a refund for the laptop computer because of the overheating concern, disputing a cost associated to the acquisition or restore of the laptop computer, or looking for compensation for restore prices incurred because of the overheating…” For a question that was clearly a technical concern, it is a poor response.
2. Why does it occur? The left panel exhibits that the execution first went to triage specialist, then to technical help after which surprisingly, to billing help specialist as effectively. The next graphic depicts this as effectively:

Trying carefully, we discover that the triage specialist accurately recognized the ticket as “Technical” and the technical help agent gave an important reply as follows:

However then, as a substitute of stopping and replying with the above because the response, the Crew Supervisor went to the Billing help specialist and tried to discover a non-existent billing concern within the purely technical consumer question.

This resulted within the Billing agent’s response overwriting the Technical agent’s response, with the Crew Supervisor doing a sub-optimal job of validating the standard of the ultimate response in opposition to the consumer’s question.

Why did it occur?

As a result of within the Crew activity definition, I specified the duties as categorize_tickets, resolve_technical_issues, resolve_billing_issues and though the method is meant to be hierarchical, the Crew Supervisor doesn’t carry out any orchestration, as a substitute merely executing all of the duties sequentially.

crew_q = Crew(
    brokers=[triage_agent, technical_support_agent, billing_support_agent],
    duties=[categorize_tickets, resolve_technical_issues, resolve_billing_issues],
    manager_llm = llm_a,
    course of=Course of.hierarchical,
    verbose=verbose,
)

For those who now ask a billing-related question, it’ll seem to offer an accurate reply just because the resolve_billing_issues is the final activity within the sequence.

What a couple of question that requires each technical and billing help, similar to “My laptop computer is overheating and likewise I used to be charged twice final month?” On this case additionally, the triage agent accurately categorizes the ticket kind as “Each”, and the technical and billing brokers give right solutions to their particular person queries, however the supervisor is unable to mix all of the responses right into a coherent reply to consumer’s question. As an alternative, the ultimate response solely considers the billing response since that’s the final activity to be known as in sequence.

Latency and Utilization: As might be seen from the above picture, the Crew execution took nearly 38 secs and spent 15759 tokens. The ultimate output is barely about 200 tokens. The remainder of the tokens had been spent in all of the considering, agent calling, producing intermediate responses and many others – all to generate an unsatisfactory response on the finish. The efficiency might be categorised as “Poor”.

Analysis of this strategy

• High quality of orchestration: Poor
• High quality of ultimate output: Poor
• Explainability: Poor
• Latency and Utilization: Poor

However maybe, the above outcome is because of the truth that we relied on CrewAI’s built-in supervisor, which didn’t have our customized directions. Subsequently, in our subsequent strategy we exchange the CrewAI automated supervisor with our customized Supervisor agent, which has detailed directions on what to do in case of Technical, Billing or Each tickets.

2. Utilizing Customized Supervisor Agent

Our Buyer Assist Supervisor is outlined with the next very particular directions. Word that this requires some experimentation to get it working, and a generic supervisor immediate similar to that talked about within the CrewAI documentation will give the identical faulty outcomes because the built-in supervisor agent above.

    position="Buyer Assist Supervisor",
    aim="Oversee the help crew to make sure well timed and efficient decision of buyer inquiries. Use the instrument to categorize the consumer question first, then resolve the following steps.Syntesize responses from completely different brokers if wanted to supply a complete reply to the shopper.",
    backstory=( """
        You don't attempt to discover a solution to the consumer ticket {ticket} your self. 
        You delegate duties to coworkers based mostly on the next logic:
        Word the class of the ticket first by utilizing the triage agent.
        If the ticket is categorized as 'Each', at all times assign it first to the Technical Assist Specialist, then to the Billing Assist Specialist, then print the ultimate mixed response. Make sure that the ultimate response solutions each technical and billing points raised within the ticket based mostly on the responses from each Technical and Billing Assist Specialists.
        ELSE
        If the ticket is categorized as 'Technical', assign it to the Technical Assist Specialist, else skip this step.
        Earlier than continuing additional, analyse the ticket class. Whether it is 'Technical', print the ultimate response. Terminate additional actions.
        ELSE
        If the ticket is categorized as 'Billing', assign it to the Billing Assist Specialist.
        Lastly, compile and current the ultimate response to the shopper based mostly on the outputs from the assigned brokers.
        """

And within the Crew definition, we use the customized supervisor as a substitute of the built-in one:

crew_q = Crew(
    brokers=[triage_agent, technical_support_agent, billing_support_agent],
    duties=[categorize_tickets, resolve_technical_issues, resolve_billing_issues],
    # manager_llm = llm_a,
    manager_agent=supervisor,
    course of=Course of.hierarchical,
    verbose=verbose,
)

Let’s repeat the take a look at instances

Enter question: Why is my laptop computer overheating?

The hint is the next:

A very powerful commentary is that now for this technical question, the move didn’t go to the Billing help specialist agent. The supervisor accurately adopted directions, categorized the question as technical and stopped execution as soon as the Technical Assist Specialist had generated its response. From the response preview displayed, it’s evident that it’s a good response for the consumer question. Additionally, the latency is 24 secs and token utilization is 10k.

Enter question: Why was I charged twice final month?

The hint is as follows:

As might be seen, the supervisor accurately skipped executing the Technical Assist Specialist, though that was earlier than the Billing agent within the Crew definition. As an alternative the response generated is of excellent high quality from the Billing Assist Specialist solely. Latency is 16 secs and token utilization 7,700 solely

Enter question: My laptop computer is overheating and likewise, I used to be charged twice final month?

The hint exhibits the Supervisor executed each Technical and Billing help brokers and supplied a mixed response.

Expensive Buyer,

Thanks for reaching out to us concerning the problems you're experiencing. We sincerely apologize for any inconvenience induced. Beneath are the detailed options to deal with your issues:

**1. Laptop computer Overheating Problem:**
   - **Verify for Correct Air flow**: Guarantee your laptop computer is positioned on a tough, flat floor to permit correct airflow. Keep away from utilizing it on mushy surfaces like beds or couches that may block the vents. Think about using a laptop computer cooling pad or stand with built-in followers to enhance airflow.
   - **Clear the Laptop computer's Vents and Followers**: Mud and particles can accumulate within the vents and followers, proscribing airflow. Energy off the laptop computer, unplug it, and use a can of compressed air to softly blow out mud from the vents. If you're snug, you possibly can clear the interior followers and elements extra completely, or take the laptop computer to an expert technician for inside cleansing.
   - **Monitor Operating Purposes and Processes**: Open the Activity Supervisor (Home windows: Ctrl + Shift + Esc, macOS: Exercise Monitor) and test for processes consuming excessive CPU or GPU utilization. Shut pointless functions or processes to scale back the load on the system.
   - **Replace Drivers and Software program**: Replace your working system, drivers (particularly graphics drivers), and some other vital software program to the most recent variations.
   - **Verify for Malware or Viruses**: Run a full system scan utilizing a good antivirus program to detect and take away any malware.
   - **Alter Energy Settings**: Alter your energy settings to "Balanced" or "Energy Saver" mode (Home windows: Management Panel > Energy Choices, macOS: System Preferences > Power Saver).
   - **Examine the Laptop computer's {Hardware}**: If the laptop computer remains to be overheating, there could also be a problem with the {hardware}, similar to a failing fan or thermal paste that wants substitute. Seek the advice of an expert technician to examine and exchange the thermal paste or defective {hardware} elements if vital.
   - **Environmental Elements**: Function the laptop computer in a cool, well-ventilated surroundings. Keep away from utilizing the laptop computer in direct daylight or close to warmth sources.
   - **Take into account Upgrading Elements**: If the laptop computer is older, take into account upgrading elements similar to RAM or switching to an SSD to scale back the pressure on the system and assist with warmth administration.
   - **Monitor Temperature Ranges**: Set up a temperature monitoring instrument (e.g., HWMonitor, Core Temp, or Macs Fan Management) to maintain observe of the CPU and GPU temperatures. This may help determine if the laptop computer is persistently working at excessive temperatures and supply insights into when the overheating happens.

If the issue persists after making an attempt all of the above options, please contact the laptop computer producer’s help crew or a licensed restore technician for additional help.

**2. Duplicate Billing Problem:**
   - **Verification of Account Particulars**: I reviewed your account info to make sure accuracy and confirmed that the account is energetic and matches the reported concern.  
   - **Billing Information Verify**: Upon analyzing the billing data for final month, I recognized two similar prices processed on the identical day. This confirms the duplicate billing concern.  
   - **Fee Methodology Particulars**: I verified the fee technique used for the transactions and confirmed that each prices had been processed utilizing the identical fee technique.  
   - **Decision**:  
       - For the reason that duplicate cost was confirmed, I've initiated a refund for the second transaction. The refund will likely be processed again to the unique fee technique inside 5-7 enterprise days.  
       - Moreover, I've utilized a credit score to your account as a gesture of goodwill for the inconvenience induced.  
   - **Buyer Communication**: You'll obtain an e-mail or cellphone name from us to tell you of the decision, together with the refund timeline and the utilized credit score.  

We sincerely apologize for the inconvenience attributable to these points. Please tell us when you want additional help or have any further questions. We're right here to assist

And our last, extra nuanced question is as follows:

Enter question: My bill quantity is inaccurate after system glitch?

The hint exhibits that the question was accurately categorised as ‘Each’, requiring responses from Technical and Billing brokers and a mixed response was supplied. Latency is 24 secs and tokens used are 12k.

Thanks for bringing this concern to our consideration. Primarily based on the technical investigation, listed here are the steps we're taking to resolve the system glitch that induced the inaccurate bill quantity:

1. We've verified the main points of the inaccurate bill quantity and recognized the problem within the system.
2. Our technical crew has analyzed the system logs and reviewed the bill calculation logic to pinpoint the foundation reason behind the error.
3. The required fixes have been applied to deal with the glitch, and the system has been completely examined to make sure the problem is resolved.
4. A corrected bill has been generated, and we have now verified its accuracy.

Moreover, right here’s the entire decision for the billing concern associated to the inaccurate bill quantity attributable to the system glitch:

1. **Corrected Bill Verification**: The technical crew has resolved the system glitch and generated a corrected bill. The corrected bill has been personally verified to make sure its accuracy, together with cross-checking the quantities, dates, and any relevant prices or credit.

2. **Buyer Communication**: We'll promptly notify you in regards to the decision of the problem. This communication consists of:
   - An apology for the inconvenience attributable to the inaccurate bill.
   - Affirmation that the system glitch has been resolved.
   - Assurance that the corrected bill has been completely reviewed for accuracy.
   - A replica of the corrected bill to your data.

3. **Extra Steps Taken**: To forestall related points sooner or later, the technical crew has applied measures to make sure system stability and accuracy in bill technology.

4. **Account Adjustment (if relevant)**: If the inaccurate bill resulted in any overpayment or underpayment, the mandatory changes will likely be made to your account. This consists of issuing a refund for any overpayment or offering clear directions for settling any excellent steadiness.

5. **Observe-Up**: We're right here to help you with any additional questions or issues concerning your account or billing. Please don't hesitate to succeed in out to us, and we will likely be completely satisfied to assist. To your comfort, we have now supplied direct contact info for additional communication.

We sincerely apologize for any inconvenience this may increasingly have induced and guarantee you that we're taking steps to forestall related points sooner or later. Thanks to your understanding and endurance.

Analysis of this strategy

• High quality of orchestration: Good
• High quality of ultimate output: Good
• Explainability: Good (we perceive why it did what it did)
• Latency and Utilization: Truthful (commensurate with the complexity of the output)

Takeaway

In abstract, the hierarchical Supervisor–Employee sample in CrewAI doesn’t perform as documented. The core orchestration logic is weak; as a substitute of permitting the supervisor to selectively delegate duties, CrewAI executes all duties sequentially, inflicting incorrect agent invocation, overwritten outputs, and inflated latency/token utilization. Why it failed comes all the way down to the framework’s inside routing—hierarchical mode doesn’t implement conditional branching or true delegation, so the ultimate response is successfully decided by whichever activity occurs to run final. The repair is introducing a customized supervisor agent with specific, step-wise directions: it makes use of the triage outcome, conditionally calls solely the required brokers, synthesizes their outputs, and terminates execution on the proper level—restoring right routing, enhancing output high quality, and considerably optimising token prices.

Conclusion

CrewAI, within the spirit of preserving the LLM on the heart of orchestration, relies upon upon it for many of the heavy-lifting of orchestration, utilising consumer prompts mixed with detailed scaffolding prompts embedded within the framework. In contrast to LangGraph and AutoGen, this strategy sacrifices determinism for developer-friendliness. And generally leads to surprising conduct for vital options such because the manager-worker sample, essential for a lot of real-life use instances. This text makes an attempt to display a pathway for attaining the specified orchestration for this sample utilizing cautious prompting. In future articles, I intend to discover extra options for CrewAI, LangGraph and others for his or her applicability in sensible use instances.

You should use CrewAI to design an interactive conversational assistant on a doc retailer and additional make the responses really multimodal. Refer my articles on GraphRAG Design and Multimodal RAG.

Join with me and share your feedback at www.linkedin.com/in/partha-sarkar-lets-talk-AI

_{All photographs on this article drawn by me or generated utilizing Copilot or Langfuse. Code shared is written by me.}