How you can Make the most of OKF Effectively to Allow Data Alternate Amongst LLMs

• The sample. That is Google’s Open Data Format skeleton — a Markdown file with a YAML frontmatter block — repurposed for agent hand-off. The repo’s frontmatter carries one further load-bearing area the overall OKF spec doesn’t outline: token_pointer, an absolute path to the pre-computed .npy array in shared reminiscence. Human-readable physique, machine-readable pointer.
• The mechanism. Three Qwen2.5-Coder fashions of various sizes (7B / 3B / 1.5B) can’t share a KV cache — they’ve totally different architectures. However they can share pre-computed token IDs, as a result of the entire Qwen2.5-Coder household ships one an identical BPE vocabulary. This repo tokenizes as soon as, arms off the integer array via /dev/shm/qwen_tokens/, and lets each downstream agent skip its personal tokenizer solely on the enter facet.
• The numbers. Median of seven trials per immediate, 3 blocks, grasping decoding, 64 new tokens: on the 3B mannequin, imply baseline TTFT drops from 69.3 ms to 49.9 ms — a 28.0% discount. On the 1.5B mannequin, from 49.6 ms to 30.9 ms — a 37.8% discount. Each fashions cross the coherence heuristic on each pattern. Full pipeline wall clock is 41.3 s finish to finish (Agent 1: 3.9 s, Agent 2: 18.7 s, Agent 3: 15.7 s).
• The guardrail. Feeding a downstream mannequin an integer array that meant a totally different subword below its personal vocabulary doesn’t crash something. It generates a fluent, coherent-looking, utterly improper report. So earlier than any agent trusts one other agent’s integers, this pipeline runs a full ~151,936-entry get_vocab() dict equality examine — not a vocab_size comparability, the true factor.
• What this does NOT declare. Brief-block regime (few-hundred-token blocks). No customized CUDA — that is orchestration on prime of transformers‘ current mannequin.generate(input_ids=...) API. Tokenizer equivalence is verified for the precise three checkpoints this repo pins, not a family-wide standing assure.

TL;DR up entrance, so you’ll be able to go away with the purpose: when you’ve got ever wired three or extra LLM-based brokers from the identical mannequin household right into a pipeline that followers out over one shared doc, your CPU is operating the very same Byte-Pair Encoding merges over the very same characters two or thrice in a row, as a result of every agent’s tokenizer is a stateless new child that has no concept the earlier agent already produced the identical integer array. This publish is a couple of small pipeline of three Qwen2.5-Coder fashions (7B, 3B, 1.5B) the place the upstream agent tokenizes as soon as, drops a NumPy array of int64 token IDs into /dev/shm/qwen_tokens/, and each downstream agent calls mannequin.generate(input_ids=...) instantly on that array. It additionally — and that is the place the really attention-grabbing engineering lives — refuses to let anybody else within the pipeline belief that array till it has confirmed, byte for byte, that each mannequin within the chain agrees on what these integers imply. That is orchestration, not a CUDA kernel. However when you’ve got ever debugged an LLM pipeline that produced fluent, on-topic, wrong-in-a-different-way-every-run output, you already know the form of the issue this piece of infrastructure is designed to stop.

Github repo: https://github.com/AnubhabBanerjee/inter-llm-tokf

---

1. A confession: your second agent is doing all your first agent’s homework, twice

Let me dramatise the second this entire repo is about.

Think about you will have three LLM brokers chained collectively. Agent 1 is an enormous mannequin, it reads a design doc. Agent 2 is a mid-sized mannequin, it evaluates a part of it. Agent 3 is a small mannequin, it writes the ultimate report. All three of them come from the identical mannequin household — identical tokenizer, identical vocabulary, identical the whole lot above the hidden layers — simply at three totally different sizes. Since you aren’t product of H100s, and operating a 7B mannequin thrice when a 1.5B mannequin will do for the final step can be, frankly, impolite to your GPU.

Now watch what occurs on a naive setup:

You: “Agent 1, please learn this design doc and cross the related sections to Agent 2.”

Agent 1 (7B): “On it. Loading tokenizer. Operating BPE over the entire doc. Sections break up. Handing off the attention-grabbing sections to Agent 2 as strings. ✅”

You: “Nice. Agent 2?”

Agent 2 (3B): “Whats up, I’m a ravishing, stateless new child. Loading my very own tokenizer. Operating BPE over the identical characters Agent 1 already ran BPE over three seconds in the past. Writing an analysis.”

You: “Wait, you will have the very same tokenizer as Agent 1.”

Agent 2 (3B): “I do?”

You: “Sure. You’re actually in the identical mannequin household. Similar vocabulary, identical subword IDs, identical the whole lot.”

Agent 2 (3B): “That’s good. Anyway, I’ve re-tokenized the enter from scratch and I’m able to generate. Please stand by. 🫡”

You: “…and Agent 3?”

Agent 3 (1.5B): “Loading tokenizer. Operating BPE over Agent 2’s output—”

You: “You understand what, overlook I requested.”

That’s the joke, and it’s the soiled secret of each multi-agent LLM pipeline that followers out over one shared piece of textual content utilizing fashions from the identical household. The tokenizer is just not the bottleneck — a quick Rust-backed BPE tokenizer is just not gradual, and I can’t misinform you and fake it’s. However the tokenizer is redundant work, and what number of instances you do redundant work is just not a operate of how briskly the redundant work is. It’s a operate of what number of downstream shoppers you fanned out to.

The purpose of this piece of infrastructure, and the entire cause it took greater than a fifteen-line patch, is that the second you resolve to skip the tokenizer on the downstream facet, you will have inherited a correctness drawback that the tokenizer was beforehand doing for you. The remainder of this publish is what that appears like once you draw it out actually, and the one runtime examine that’s doing all of the load-bearing work.

2. Why three sizes in any respect? (a one-minute crash course on which layer is definitely shared)

Skip this in the event you already know. For everybody else, right here is the quick model.

The three fashions on this pipeline are Qwen/Qwen2.5-Coder-7B-Instruct, Qwen/Qwen2.5-Coder-3B-Instruct, and Qwen/Qwen2.5-Coder-1.5B-Instruct. Similar structure household, identical tokenizer, three totally different sizes. The explanation they’re three totally different sizes and never one massive one is intentionally telecom-flavored, as a result of that’s the world I really got here from: the concrete instance this repo is constructed towards is a design doc proposing {that a} chain of LLM brokers assist a cell core community’s operations crew cause a couple of new control-plane function — particularly, bolting MCP (Mannequin Context Protocol) and A2A (Agent-to-Agent protocol) model orchestration onto the present 5G Service-Based mostly Interface. The plan requires a big “Architect” agent that buildings the doc, a mid-sized “Protocol Engineer” that evaluates the attention-grabbing sections, and a small “Edge Analyst” that produces deployment-ready latency steerage — sufficiently small to run at a far-edge website subsequent to a UPF.

Three sizes, three roles, one pipeline.

Now, one structural truth drives your entire design: you can not share a KV cache throughout these three fashions. Totally different sizes imply totally different hidden_size values — 3584 for the 7B, 2048 for the 3B, 1536 for the 1.5B. The form of a KV cache is derived instantly from that quantity, so there is no such thing as a reinterpreting one mannequin’s cache as one other’s. That door is closed, completely, by the mathematics.

What’s not closed is the tokenizer. Qwen2.5-Coder ships one BPE vocabulary throughout its whole dimension vary — the entire household is documented to agree on the identical integer-to-subword mapping. So whilst you can’t share activations between differently-sized fashions, you completely can share token IDs, supplied — and this “supplied” is doing a variety of work, extra on that in a minute — each mannequin within the chain actually does use that very same vocabulary.

You probably have learn sufficient distributed-systems papers to be harmful, this form is acquainted. Two community capabilities on the identical message bus don’t get to imagine they agree on message semantics simply because they’re each plugged into the identical bus. Two fashions in the identical household don’t get to imagine they agree on hidden states simply because they agree on vocabulary. Totally different layer, identical self-discipline: discover the precise layer of the stack the place interoperability is definitely assured, and refuse to imagine it holds one layer larger simply because the layers are adjoining.

The tokenizer is that layer. Every part above it’s a form mismatch. Every part at or beneath it, if we’re fortunate and if we examine, is a free integer array.

3. OKF: the “simply hand off the integers” sample

Right here is the pitch in 5 bullets:

1. Agent 1 hundreds solely the 7B mannequin’s tokenizer — by no means its weights. It splits the doc, tags every part, and tokenizes every part.
2. It saves every part’s token IDs as a NumPy int64 array into /dev/shm/qwen_tokens/. That may be a RAM-backed tmpfs mount, not disk, so studying it again is a memcpy, by no means a search.
3. It additionally writes one Markdown file per part into okf_workspace/. The Markdown physique is the part’s human-readable textual content. The YAML frontmatter carries the metadata — block_id, tags, token_pointer, token_count, tokenizer_model_id, and so forth.
4. Agent 2 (the 3B mannequin) reads the frontmatter, follows token_pointer into shared reminiscence, hundreds the .npy, and calls mannequin.generate(input_ids=...) instantly on the loaded tensor. No tokenizer name on the enter facet.
5. Agent 2 tokenizes its personal output (that textual content has, by definition, by no means been tokenized earlier than — nothing to reuse), saves that array to shm, writes one other OKF file, and Agent 3 (1.5B) does the identical trick once more.

A fast introduction on the “OKF” (for many who don’t know but)

OKF stands for Open Data Format, and earlier than you learn the frontmatter block beneath, one factor is price being trustworthy about.

The Open Data Format is a broadcast spec — Google Cloud shipped v0.1 in June 2026 and v0.2 is now the present model (see GoogleCloudPlatform/knowledge-catalog on GitHub). Its pitch is deliberately minimal: a bundle is a listing of UTF-8 Markdown information, every file is one idea, and every file carries a YAML frontmatter block plus a Markdown physique. The one frontmatter area the spec requires is kind — a brief human-readable string like BigQuery Desk, Playbook, or Attested Computation. Every part else is elective metadata. It’s a format, not a platform: no schema registry, no SDK, no central authority. For those who can cat a file, you’ll be able to learn OKF.

This repo’s okf/ reuses that actual skeleton — one Markdown file per unit of labor, YAML frontmatter plus a human-readable physique — however interprets it for a job the overall spec was not written for: an agent-to-agent hand-off of pre-tokenized integer arrays. So this repo’s required frontmatter fields will not be Google’s kind; they’re block_id, source_agent, stage, title, tags, token_pointer, token_count, tokenizer_model_id, and created_at (see utils/okf_parser.py‘s REQUIRED_FRONTMATTER_KEYS). The load-bearing one is token_pointer — an absolute path into /dev/shm/qwen_tokens/ — which has no equal within the normal OKF spec as a result of Google’s OKF was designed for sturdy information sharing, not for a shared-memory hand-off between short-lived agent processes on the identical GPU host. Put plainly: this repo’s information are not legitimate Google-OKF bundles as-is (they lack kind, they add token_pointer); the repo is conforming in spirit — identical Markdown+YAML aesthetic, identical “standardise the interoperability floor, not the content material mannequin” intuition — with one domain-specific required area bolted on. This publish retains the repo’s terminology as a result of that’s what the supply code and the generated information really use.

With that out of the best way, right here is the schema within the wild — the precise frontmatter block from okf_workspace/block_004_routing_and_signaling_integration_points.md, unedited:

---
block_id: block_004_routing_and_signaling_integration_points
source_agent: agent_1_architect
stage: 1
title: Routing and Signaling Integration Factors
tags:
- routing
- signaling
- safety
- deployment
token_pointer: /dev/shm/qwen_tokens/block_004_routing_and_signaling_integration_points.npy
token_count: 3437
tokenizer_model_id: Qwen/Qwen2.5-Coder-7B-Instruct
created_at: '2026-08-04T12:41:39.249368+00:00'
---

The load-bearing area is token_pointer. Every part else — source_agent, stage, tags, token_count, tokenizer_model_id, created_at — exists to assist routing and provenance choices round that one array. Agent 2 filters the workspace by tag (routing or signaling, each set off it). Agent 3 filters by supply agent (agent_2_protocol_eval, so it by no means unintentionally picks up its personal output on a re-run). The tokenizer_model_id area is there so a future audit can cross-check per-file which tokenizer really produced the bytes at that path, as a substitute of trusting one pipeline-start assertion for all eternity.

Yet another architectural element price calling out: every agent is a separate OS course of. src/run_pipeline.py launches them by way of subprocess.run, separately. That’s deliberate, not lazy: a CUDA context solely releases its VRAM again to the motive force when the method holding it exits. So operating three multi-GB fashions sequentially inside one course of would leak every prior mannequin’s VRAM into the subsequent agent’s reminiscence finances except each caller remembered to manually del mannequin; torch.cuda.empty_cache() — and even that isn’t at all times enough to completely reclaim CUDA context overhead. Subprocess isolation makes VRAM launch unconditional and automated. On a single-GPU field, that is what lets the 7B, then the 3B, then the 1.5B every get the entire card to themselves in flip, with out ever needing all three resident in reminiscence concurrently.

4. The precise save/load code, all six significant strains of it

Now the code that does the precise hand-off. From utils/token_manager.py, verbatim:

def save_token_array(token_ids: torch.Tensor, block_name: str) -> Path:
    ...
    token_ids_as_numpy_int64 = token_ids.detach().cpu().numpy().astype(TOKEN_ARRAY_DTYPE)
    destination_path = QWEN_TOKENS_SHM_DIR / f"{block_name}.npy"
    np.save(destination_path, token_ids_as_numpy_int64, allow_pickle=False)
    return destination_path

That’s the write half. Three strains that truly transfer knowledge. QWEN_TOKENS_SHM_DIR is /dev/shm/qwen_tokens, a RAM-backed tmpfs mount. TOKEN_ARRAY_DTYPE is np.int64, matching torch’s default torch.lengthy, particularly so the load facet by no means wants a casting step. And allow_pickle=False is there as a result of a .npy file with allow_pickle=True will fortunately deserialise and execute pickled Python objects from disk — pointless assault floor for an array that’s, by definition, pure numeric knowledge.

Right here is the learn half:

def load_token_array(pointer_path: Path) -> torch.Tensor:
    ...
    token_ids_as_numpy_int64 = np.load(pointer_path, allow_pickle=False)
    if token_ids_as_numpy_int64.dtype != TOKEN_ARRAY_DTYPE:
        increase TypeError(...)
    return torch.from_numpy(token_ids_as_numpy_int64)

Additionally three significant strains. np.load reads again the precise .npy header (which embeds dtype, form, and byte-order, all specific), the defensive dtype examine refuses to silently .astype() if some future code path ever writes one thing aside from int64 into this namespace, and torch.from_numpy(...) shares reminiscence with the NumPy array — zero-copy, since token IDs from this level ahead are by no means mutated in place by any agent.

That’s the whole on-wire format. A NumPy .npy file, int64, on a RAM-backed mount. For those who had been anticipating one thing unique, sorry to disappoint you.

The final piece of the puzzle is what a downstream agent really does with the loaded tensor. From utils/model_loader.py, the 2 entry factors that Agent 2 and Agent 3 can name — the naive baseline, and the optimized path. Take a look at them facet by facet, as a result of the entire optimization is one operate name’s price of distinction:

def generate_from_text(mannequin, tokenizer, prompt_text, max_new_tokens):
    ...
    wall_clock_start = time.perf_counter()

    encoded_prompt = tokenizer(prompt_text, return_tensors="pt")

    input_ids = encoded_prompt["input_ids"].to(mannequin.machine)
    attention_mask = encoded_prompt["attention_mask"].to(mannequin.machine)

    return _generate_and_measure_ttft(
        mannequin, tokenizer, input_ids, attention_mask, wall_clock_start, max_new_tokens
    )

Baseline. Clock begins earlier than tokenizer(...) runs, so the tokenizer-encode value this pipeline exists to skip is absolutely included within the reported TTFT. That’s not unintended — it’s intentionally trustworthy. If the baseline began its clock after tokenization, the comparability would understate the true financial savings and fake the tokenizer was free. It isn’t free. It’s quick, however it’s not free.

Now the optimized facet:

def generate_from_token_ids(mannequin, tokenizer, token_ids, max_new_tokens):
    ...
    wall_clock_start = time.perf_counter()

    input_ids = token_ids.unsqueeze(0).to(mannequin.machine)
    attention_mask = torch.ones_like(input_ids)

    return _generate_and_measure_ttft(
        mannequin, tokenizer, input_ids, attention_mask, wall_clock_start, max_new_tokens
    )

The clock additionally begins right here, with no tokenizer name previous it — the entire level of the comparability. token_ids was already produced by an upstream agent’s tokenizer, already saved into shm, already loaded off shm. All this operate does earlier than beginning the mannequin is unsqueeze a batch dimension and duplicate the array to the GPU. The tokenizer argument remains to be handed in, however solely as a result of _generate_and_measure_ttft wants it to provide pad_token_id and to decode the output tokens again to textual content — the enter facet genuinely by no means hits the tokenizer.

The one-line distinction between these two capabilities — one line, tokenizer(prompt_text, ...) — is your entire financial savings. It sounds virtually too small to write down an article about. Hold studying, as a result of the failure mode on the opposite facet of “virtually too small” is just not small in any respect.

5. The half the place I ended trusting the seller docs

Right here is the sentence from my very own challenge notes that made me nervous sufficient to write down code as a substitute of simply delivery the pipeline: “Qwen2.5-Coder is documented to share one tokenizer throughout the entire household.” Documented. By whom? Checked how not too long ago? What occurs to a few brokers’ price of generated textual content if that seems to be true for six of the seven sizes and subtly not true for the one I picked?

A tokenizer mismatch right here doesn’t crash something. That’s the scary half. mannequin.generate(input_ids=[1234, 5678, ...]) doesn’t know or care whether or not 1234 meant the identical subword to whoever produced it because it means to the mannequin about to embed it. It’ll fortunately run a ahead cross on integers that decode to finish nonsense below its personal vocabulary, and it’ll fortunately generate a fluent-looking continuation of that nonsense. You get a confidently improper report, not an error. Your tokenizer: not the bottleneck. Your assumptions about your tokenizer: solely the bottleneck.

So earlier than any agent is allowed to belief a token array it didn’t produce itself, this runs — from utils/env_checks.py:

def verify_tokenizer_equivalence(
    model_ids: tuple[str, ...] = PIPELINE_MODEL_IDS,
) -> None:
    ...
    loaded_tokenizers = {
        model_id: AutoTokenizer.from_pretrained(model_id) for model_id in model_ids
    }

    reference_model_id = model_ids[0]
    reference_tokenizer = loaded_tokenizers[reference_model_id]
    reference_vocab_size = reference_tokenizer.vocab_size

    reference_vocab = reference_tokenizer.get_vocab()

    for candidate_model_id in model_ids[1:]:
        candidate_tokenizer = loaded_tokenizers[candidate_model_id]

        if candidate_tokenizer.vocab_size != reference_vocab_size:
            increase RuntimeError(
                f"Tokenizer vocab_size mismatch: {reference_model_id} has "
                f"vocab_size={reference_vocab_size}, however {candidate_model_id} "
                f"has vocab_size={candidate_tokenizer.vocab_size}. Token IDs "
                "produced by one will not be protected to feed into the opposite's "
                "embedding layer."
            )

        if candidate_tokenizer.get_vocab() != reference_vocab:
            increase RuntimeError(
                f"Tokenizer vocabulary mismatch between {reference_model_id} "
                f"and {candidate_model_id}: at the very least one token string maps "
                "to a special integer id between the 2. Direct token "
                "injection throughout these fashions would silently corrupt "
                "downstream generations."
            )

        if candidate_tokenizer.special_tokens_map != reference_tokenizer.special_tokens_map:
            increase RuntimeError(
                f"Particular-tokens map mismatch between {reference_model_id} "
                f"({reference_tokenizer.special_tokens_map}) and "
                f"{candidate_model_id} ({candidate_tokenizer.special_tokens_map})."
            )

Three checks, intentionally layered.

The primary examine is vocab_size. It exists purely so a mismatch right here produces a brief, immediately-readable error naming the 2 integers that disagree, as a substitute of forcing whoever is debugging this to diff two ~151,936-entry dicts by hand to seek out that the sizes alone differ.

The second examine — the load-bearing one — is full dictionary equality on get_vocab(). Not a vocab_size comparability. A full dict != dict over your entire ~151,936-entry mapping of each subword string to each integer id. Two tokenizers can have an identical sizes and nonetheless disagree about what integer 42 means. That is the examine that may catch a “shuffled id project for even a single subword” mismatch, which is strictly the sort of failure that produces fluent nonsense downstream as a substitute of a loud error.

The third examine is special_tokens_map. A mannequin’s chat template and stopping conduct rely on these actual strings/ids matching too — an accurate principal vocabulary with a divergent EOS id, for instance, would make a downstream agent’s generate() name fail to cease on the boundary Agent 1 supposed.

I needed the precise assure, not a budget proxy for it. Ran it towards the true triplet earlier than writing one other line of pipeline code, and it held: Qwen2.5-Coder-7B-Instruct, Qwen2.5-Coder-3B-Instruct, and Qwen2.5-Coder-1.5B-Instruct all agree, byte for byte. Good. However “it held, this time, for this triplet” is a really totally different sentence from “it’s documented to carry,” and solely a type of two sentences belongs in a pipeline you’re going to run unattended.

6. The receipts

Similar 3 sections of the design doc (those Agent 1’s key phrase scan tagged routing or signaling — block_002 at 1948 tokens, block_003 at 2292 tokens, block_004 at 3437 tokens). Similar grasping decoding. Max 64 new tokens for the timed comparability. One throwaway warm-up name absorbed earlier than any timed measurement so cuBLAS’s first-call kernel choice doesn’t contaminate the numbers. Median of seven repeated trials per block, to clean out millisecond-scale scheduling and GPU-clock jitter.

Straight from scripts/benchmark.py‘s output:

=== Benchmarking Qwen/Qwen2.5-Coder-3B-Instruct ===
  Metric 1 (TTFT discount): mean_baseline=69.3 ms, mean_injection=49.9 ms, discount=28.0% -- PASS
  Metric 2 (semantic constancy): PASS

=== Benchmarking Qwen/Qwen2.5-Coder-1.5B-Instruct ===
  Metric 1 (TTFT discount): mean_baseline=49.6 ms, mean_injection=30.9 ms, discount=37.8% -- PASS
  Metric 2 (semantic constancy): PASS

[benchmark] ALL ACCEPTANCE METRICS PASSED

In desk kind:

The attention-grabbing bit is just not that each fashions obtained sooner — after all they did, they stopped doing redundant work. The attention-grabbing bit is why the 1.5B mannequin’s proportion discount is noticeably larger than the 3B mannequin’s, although absolutely the variety of milliseconds saved is roughly comparable. The reason is within the repo’s personal README, and it’s price quoting as a result of it’s the sort of factor that journeys folks up in the event that they solely learn the desk:

The tokenizer’s CPU value is similar string, tokenized as soon as, no matter which mannequin reads the end result — however GPU forward-pass latency scales with mannequin dimension. For the smaller 1.5B mannequin, that GPU-side flooring is decrease, so the (roughly fastened) tokenizer value it avoids is a bigger fraction of its complete time-to-first-token.

That can also be, by the way, why blindly rising the enter doc additional doesn’t push the discount towards 100%. Previous a sure enter size, GPU compute time itself begins rising too, and the share plateaus fairly than climbing indefinitely. The financial savings scale with how a lot textual content you’d in any other case redundantly re-tokenize, instances what number of downstream brokers share that very same enter, divided by how massive every downstream mannequin’s personal ahead cross is. On a brief single-hop demo, you get double-digit p.c. On a big supply doc fanned out to many downstream brokers of the identical household, you pay the BPE value as soon as as a substitute of N instances, which is strictly the regime the plan was constructed for.

The semantic-fidelity facet of the receipts is a heuristic, on goal. Two ratios: printable-character ratio ≥ 0.98, and unique-word ratio ≥ 0.25 throughout the pattern’s tokens. Low-cost sufficient to run on each era, calibrated to catch the precise “rubbish output” failure mode a tokenizer mismatch or byte-order bug produces — degenerate repetition of 1 token, or a wall of non-printable control-character noise — not a normal high quality judgment. Each pattern from each mannequin handed. Learn extra particulars concerning the outcomes right here.

7. Wrap: the really attention-grabbing half was the guardrail

The attention-grabbing a part of this challenge was by no means “skip the tokenizer, it’s gradual.” Tokenizers, particularly the quick Rust-backed variety, will not be the bottleneck anybody thinks they’re — the numbers above show that themselves. Saving 20 ms of TTFT is good. It isn’t the purpose.

The attention-grabbing half was constructing the one piece of infrastructure that makes skipping the tokenizer protected: a runtime examine that refuses to let one agent belief one other agent’s integers till it has really confirmed they converse the identical language, byte for byte, vocabulary entry for vocabulary entry. That examine is what turns “20 ms sooner” from a footgun right into a dependable engineering transfer. With out it, you will have a pipeline that’s quick when it really works and confidently improper when it doesn’t, and no clear solution to inform which one you’re presently dwelling in.

Each multi-agent pipeline that passes state between fashions is making an assumption like this someplace, normally silently. Typically it’s about tokenizer vocabularies. Typically it’s about hidden-state dimensions. Typically it’s concerning the that means of a specific chat-template string. Typically it’s about which facet of an RPC boundary the retries reside on. Mine simply occurs to be about BPE integer-to-subword mappings, as a result of that’s what this repo’s optimization technique leans on. Yours is someplace else. Go discover it. It’s most likely not documented both.

If you wish to reproduce the numbers, python scripts/benchmark.py on a CUDA GPU with sufficient VRAM for a bf16 3B checkpoint will do it. If you wish to reproduce the pipeline itself towards your individual enter, drop your doc into knowledge/raw_input.txt and python src/run_pipeline.py walks via the three levels, cleans up shm on the best way out, and leaves three OKF information behind in okf_workspace/.

Small pipeline. Modest numbers. One load-bearing examine. That’s the entire form of it.

Disclaimer: The illustrations on this article had been generated utilizing AI (Claude Opus 4.8). They’re illustrative, not photographic, and any labels seen inside the pictures are stylized fairly than authoritative — seek advice from the article physique and the code itself for exact operate names, metric values, and structure particulars.