In November 2016, three colleagues and I at IBM filed a patent utility with a title that sounds nearly quaint right now: “Actual-Time Evaluation of a Musical Efficiency Utilizing Analytics” (printed as US 2018/0137425 A1 [1]).
The situation we described was a rock live performance. A guitarist performs an instrument with an embedded sensor, cameras learn the viewers’s faces and physique language, followers fireplace likes and dislikes from their telephones, and a system fuses all of it, in actual time, into one judgment: is that this efficiency going effectively? If not, it tells the performer, whereas there may be nonetheless time to repair it. In a single embodiment, an LED on the neck of the guitar lights up inexperienced when the half is performed appropriately, flashes yellow when the tempo begins to waver, and turns crimson on a flawed word.
We referred to as the core of it an “artificially clever analytics engine,” as a result of in 2016 that was the vocabulary we had.
Ten years later I design knowledge architectures for agentic AI methods in manufacturing, and just lately I re-read our patent with right now’s eyes. The expertise was uncanny. What we specified, element by element and step-by-step, is the anatomy of an AI agent: multimodal notion, context aggregation, inference grounded in episodic and semantic reminiscence, an analysis perform, and actuation by means of a human within the loop, all beneath a tough real-time constraint. We didn’t have the language for it. We had the loop.
This text walks by means of that mapping actually: what a 2016 decision-support spec acquired structurally proper about brokers, and, simply as instructive, what it acquired flawed or couldn’t see in any respect.
What we really specified
Strip away the patent prose and the system has seven elements.
Notion, in three modalities. Sensors embedded in devices and microphones report goal efficiency parameters: timing, pitch, intonation, quantity (the spec calls them sensors 113a, and consists of MIDI, tactile, vibrational, ultrasonic, strain and infrared variants). Cameras seize the viewers’s involuntary alerts: facial expressions and physique language, explicitly configured for low gentle and statistically vital protection of the room. And a community path carries deliberate suggestions: scores, likes, dislikes and feedback submitted from private gadgets to an internet service.
A suggestions aggregator that receives all the pieces, tags every factor by supply and sort, types it into per-performer streams, and reformats it for evaluation. Studying it now, that is context engineering: the element that decides what the reasoning engine really sees, in what form and in what order.
An analytics engine that infers traits of the continuing efficiency. The claims are exact about what grounds the inference, and that is the sentence that stopped me on re-reading: the process is “a perform of a repository of archived details about previous performances by the a number of performers and of a knowledgebase from which the suggestions could also be given semantic that means.” In 2026 phrases: inference as a perform of episodic reminiscence (what occurred in previous performances, together with which tempos traditionally correlated with audiences dropping curiosity) and semantic reminiscence (what the area is aware of, together with the rating being carried out). We even specified the comparability of dwell sensor knowledge in opposition to the saved rating to detect {that a} guitarist isn’t taking part in just a few flawed notes however is within the flawed key.
A efficiency index: a home made analysis perform that fuses optimistic tweets, likes, facial sentiment, body-language sentiment and goal accuracy into one quantity, normalized, thresholded, and tracked over time.
A reporting module that closes the loop to the human: a graph, a named sentiment, a suggestion (“finish the bass solo”), or that LED on the guitar neck.
And wrapped round all of it, a real-time price range: the spec insists the loop should full quick sufficient for the performer to behave through the efficiency, ideally beneath a second, at worst inside just a few.
Sensors in, aggregation, memory-grounded inference, analysis, actuation, iteration. Steps 200 by means of 280, “carried out repeatedly, as an iterative process.” It’s a sense-think-act loop with reminiscence. In 2016 the taxonomy referred to as this choice help. Right now we’d name the identical diagram an agent, with one deliberate design selection: the actuator is a human.
The mapping, element by element
2016 spec, and its 2026 equal:
Embedded sensors, cameras, social scores: multimodal software and occasion ingestion. The perception that involuntary alerts (faces, physique language) and deliberate alerts (scores) are completely different lessons of proof, to be fused slightly than averaged blindly, is context-quality pondering that many agent pipelines nonetheless lack.
Suggestions aggregator: the context meeting layer. Tagging by provenance, per-entity streams, format normalization. Right now that is the retrieval and packing stage in entrance of the mannequin, and it’s nonetheless the place most manufacturing failures originate.
Historic database plus knowledgebase: episodic plus semantic reminiscence. The patent treats inference as explicitly a perform of each. That’s the memory-grounded structure that agentic methods are actually converging on, after a detour by means of stateless prompting.
Efficiency index: the eval. A linear, hand-weighted, interpretable scoring perform with thresholds and persistence circumstances. Primitive subsequent to a realized reward mannequin, and in addition auditable in a approach most realized rewards are usually not. Regulated industries are rediscovering the advantage of that trade-off.
Reporting module with suggestions: actuation with a human within the loop, and an autonomy dial. The spec distinguishes between displaying an index (pure data), naming an inferred state (“the viewers is bored”), and recommending a particular motion (“improve your quantity”). Three escalating ranges of company, with the human at all times executing. In 2026, selecting the best level on that dial per motion class is without doubt one of the central design selections in enterprise brokers.
Actual-time constraint: the latency price range. The patent’s insistence that suggestions should arrive whereas the efficiency can nonetheless be corrected is identical self-discipline that separates brokers that function in workflows from brokers that write reviews about them.
What we acquired structurally proper
4 issues maintain up, they usually maintain up as a result of they’re knowledge engineering, not mannequin science.
First, the loop is the product. The worth was by no means in any single inference however within the closed cycle of notion, analysis and correction. Groups that right now bolt a mannequin onto a workflow with out closing the loop rediscover this the arduous approach.
Second, inference have to be grounded in reminiscence. Our engine with out the historic repository and the knowledgebase would have been a sentiment classifier shouting into the void. The identical is true of a contemporary agent with out episodic and semantic shops: fluent, and structurally incapable of studying from Tuesday for Wednesday.
Third, proof has provenance and sophistication. Fusing goal telemetry, inferred sentiment and deliberate scores as distinct sign households, every with its personal reliability profile, is precisely how a critical agent ought to weigh software outputs in opposition to retrieved paperwork in opposition to consumer assertions.
Fourth, the human is an architectural element, not an afterthought. The performer is the actuator, and the system’s whole design respects the constraints of that actuator: consideration, timing, cognitive load. An LED that turns crimson is a masterpiece of interface economic system beneath these constraints.
What a decade exposes
Now the humbling half, as a result of the gaps train greater than the hits.
The intelligence was the bottleneck, and we couldn’t admit it. Our “artificially clever analytics process” was guidelines, ontologies and classifiers. Fusing frowning faces, sarcastic tweets and MIDI streams right into a judgment like “the viewers is dropping curiosity as a result of the tempo dropped” was, in 2016, largely aspiration. The loop was proper and the reasoner was weak. Basis fashions inverted that: right now the reasoner is powerful and it’s all the pieces round it, the loop, the reminiscence, the context, that’s weak in most deployments. The lesson cuts each methods: structure with out intelligence stalls, intelligence with out structure flails.
Reminiscence was append-only, and forgetting didn’t exist. Our historic database solely grew. Nothing decayed, nothing expired, nothing could possibly be provably deleted. In 2016 that appeared like a function. Right now, an agent reminiscence with out designed forgetting is a legal responsibility with wonderful recall: stale details poisoning selections, and private knowledge accumulating with out lifecycle. Retention, decay and provable erasure are architectural capabilities, not cleanup scripts.
Permissions had been nowhere. Re-read with 2026 eyes, essentially the most placing absence in your entire doc: we cheerfully specified cameras performing sentiment evaluation on the faces of particular person viewers members, with zoom lenses to pick out particular folks, and steady per-member sentiment streams, and there may be not one paragraph about consent, entitlement or entry. We filed in November 2016; the GDPR turned relevant eighteen months later [2] and would have had robust opinions about each determine in that patent. Right now I spend a big a part of my working life on precisely this layer: what an AI system is allowed to look at, retain and act upon, on whose behalf, enforced within the knowledge layer itself. The 2016 blind spot is the 2026 self-discipline.
Nothing realized. The system inferred and really helpful, however no final result ever up to date the system itself. The weights, in each sense, had been frozen. The hole between reminiscence as accumulation and studying as enchancment remains to be one of many trustworthy open issues in agentic AI, and pretending retrieval alone closes it repeats our 2016 optimism in new vocabulary.
The lesson for agent builders
Here’s what I take from my very own decade-old doc.
The agentic sample isn’t new. Notion, aggregation, memory-grounded inference, analysis, bounded actuation: engineers have been drawing this loop for a very long time, in choice help, in management methods, in monitoring. What modified within the final three years is that the reasoning element lastly works. That could be a real revolution, and it is usually exactly one element of seven.
Which suggests the scarce talent in agentic AI isn’t novel. It’s the knowledge architect’s self-discipline utilized to a brand new reasoner: design the reminiscence with lifecycle and forgetting, engineer the context with provenance, make the analysis auditable, put entitlements within the knowledge layer, respect the latency price range, and select intentionally the place the human sits within the loop. Each a type of considerations was seen in 2016, not less than in embryo, to 4 engineers serious about rock live shows. Each considered one of them is the place manufacturing brokers succeed or fail in 2026.
We didn’t lack the concept. We lacked the mannequin. Now the business has the mannequin, and what it lacks, deployment after deployment, is the self-discipline round it. The LED on the guitar neck nonetheless appears to me the best ambition for the entire subject: a system that is aware of sufficient, remembers sufficient, and is allowed to know sufficient, to inform you one true and well timed factor, whereas there may be nonetheless time to behave on it.
References
[1] S. D’Alo’, M. Lerro, M. Noioso and N. Piazza, Actual-time evaluation of a musical efficiency utilizing analytics (2018), US Patent Software US 2018/0137425 A1
[2] European Parliament and Council of the European Union, Regulation (EU) 2016/679 (Normal Knowledge Safety Regulation) (2016), Official Journal of the European Union
