Multi-agent architectures that actually survive production

Here is the test we apply to every step in a workflow. If you can state the rule completely — every input, every case, every exception — write it as ordinary code. Is the temperature above the threshold? Is this equipment under warranty? Has an operator approved this action? Those are lookups and comparisons. Code answers them in milliseconds, for free, with the same answer every time, and a unit test proves it. Putting a language model in charge of a question like that adds cost, latency, and a small but real chance of a confidently wrong answer — while removing your ability to guarantee the behaviour.

An AI agent earns its place only where the rules run out. Interpreting a messy, ambiguous situation: three sensors drifting in a pattern that does not match any alarm definition, but that an experienced operator would find suspicious. Turning unstructured input into something structured: a technician's free-text fault note, a rambling alarm log. And producing language for humans: an incident summary a facility manager can read at shift handover. These are judgement tasks. Nobody can enumerate the cases in advance, which is exactly why a model — trained on patterns rather than rules — is the right tool.

Apply that test to facility monitoring and the system splits cleanly. Deterministic code gets: threshold checks, alarm routing, warranty and work-order lookups, deciding whether a step already ran, whether a retry is safe, whether approval was given, and — without exception — every action that touches physical equipment. Agents get: interpreting multi-signal anomalies, proposing a likely root cause, drafting the work order, and writing the plain-English briefing. Notice the pattern: agents interpret and propose; code verifies and acts.

The most common failure we see is the line drawn in the wrong place — usually an agent doing a rule's job. An agent asked to decide whether a reading breaches a threshold will get it right nearly every time, and 'nearly' is the problem. The reverse mistake is subtler: teams write thousands of lines of brittle if-statements trying to classify anomalies that genuinely need judgement, and the rules rot as the facility changes. Both mistakes come from not asking the question explicitly, step by step.

The architecture's job is to make the dividing line structural rather than aspirational. AWS Step Functions is the deterministic backbone: it starts when a sensor event arrives, runs checks in parallel, enforces waiting periods, retries failed steps, and stops — indefinitely if needed — until a human approves. Every question of sequencing and certainty lives here, in a workflow you can read as a diagram. The agents sit inside individual steps as pockets of judgement, and the workflow treats their outputs as proposals to be checked, never as instructions to be obeyed.

Inside those pockets, the Strands SDK — AWS's open-source toolkit for building agents — defines what each agent can actually do. The sensor analyst can query sensor history; the maintenance advisor can draft a ticket. Neither can change a piece of equipment, because neither is handed a tool that can: an agent cannot misuse a capability it was never given. Amazon Bedrock runs the models behind them, using the access controls and logging the rest of AWS already uses — and since many models sit behind one interface, routine classification runs on small, cheap models while the genuinely ambiguous incidents get the expensive ones.

Watch the line at work in a single incident. A compressor draws unusual current. Code detects the anomaly — a statistical check, no model involved — and Step Functions saves the raw event before any AI sees it. Now judgement is needed: the sensor analyst agent reads 24 hours of related signals and returns a structured verdict — severity, likely cause, confidence. Back to code: the policy check against the site's rules is lookups, not opinion. Is there an open work order? Does the fix require shutdown? If action is needed, the workflow stops and waits for an operator — approval is a fact, recorded deterministically, never inferred by a model. Judgement again: the advisor agent drafts the work order a technician will read. And finally code: a small, tightly restricted step carries out the approved action, built so running it twice by accident has the same effect as running it once.

Code, agent, code, human, agent, code. Every handoff is a checkpoint where a structured result gets saved to a database — so retries read the record of what actually happened instead of a model's reconstruction, and so an auditor can see months later exactly which facts were computed, which were judged, and who approved what.

The boundary is not fixed forever, and testing is how you find out it has drifted. Each agent runs against a library of known scenarios — real fault patterns, rule-book cases — and the results tell you where the line belongs. If the sensor analyst gives effectively the same answer to the same class of input every time, that judgement has become a rule: replace the agent with code and bank the cost, latency, and reliability. If a block of classification rules keeps sprouting exceptions and still misses cases, that rule has revealed itself to be judgement: consider an agent. The line moves in both directions, and it should.

The discipline this demands is the one demos never teach: default to deterministic code, add an agent only when you can point to a judgement the code demonstrably cannot make, and keep checking that the judgement is still needed. The architectures that survive production are not the ones with the most agents. They are the ones where a tired engineer at 2 a.m. can tell, for every step, whether the answer came from a rule or a judgement — and trusts the difference. That, in the end, is what the pump was asking.