Pulling the Andon Cord in Network Automation – Part 2

If Andon is going to be real, meaning it can outrun defect propagation, then it can’t live as a feature buried inside a single tool. It has to live where work becomes action. It has to sit in the same place a circuit breaker sits in an electrical system: upstream of the load, authoritative over what flows, and fast enough to matter.

In a hyperscaler-grade automation ecosystem, that usually converges to a simple architectural primitive: an Andon Policy Service.

Think of it as a small, boring service with an outsized responsibility: it owns the current “permission posture” for automation work. When the cord is pulled, nothing mystical happens. The posture changes, and the rest of the ecosystem is obligated to obey.

That obligation only holds if the Andon service is engineered like a control plane, not like a dashboard. It needs to be highly available, low-latency, and resistant to partial failures, because you will rely on it precisely when the world is unstable.

The Andon Policy Service is the source of truth for policy state: which scopes are frozen, which work types are disabled, which exceptions exist, and why. That “central authority” phrase scares people because it sounds like you’re introducing a single point of failure. The way hyperscalers avoid that trap is by separating authority from runtime dependency.

Authority is centralized: there is one canonical policy state.

Enforcement is decentralized: every executor can make a decision locally, quickly, and safely, even when the policy service is briefly unreachable.

That split is the heart of the architecture.

When a defect is spreading, your enemy is not just the defect: it’s time. If your automation fleet can push changes in seconds, an Andon that takes minutes to propagate is ceremonial, not protective.

So policy updates must propagate like a live control signal. The common pattern is:

The cache is not optional. It’s the mechanism that makes policy checks fast enough to sit inline with real workflows. When a deployer is about to push, it should not wait on a network round-trip to a central service. It should consult its cached policy snapshot in microseconds, decide “allowed/denied,” and move on.

This also gives you a crucial property during turbulence: bounded staleness. Even if the policy service becomes briefly unreachable, components can continue making safe decisions based on the last known policy for a defined time window.

A robust Andon design enforces policy in two places, because defects propagate through multiple paths, and you don’t want a single missed check to become an incident.

The first place is admission control, as early as possible. If you have a shared orchestration layer, such as a job scheduler, workflow engine, or campaign coordinator, this is the ideal choke point. Work requests enter the system here. If Andon says “deployments to eu-west border devices are frozen,” the orchestrator refuses to enqueue those jobs. Propagation is stopped before it becomes side effects.

The second place is execution-time enforcement, right before an action touches the world. Even with strong admission control, jobs can already be queued when the cord is pulled. Executors must re-check policy at the moment they’re about to act. That turns Andon into a true circuit breaker: even if the job exists, the action does not happen.

This “check twice” pattern is what makes the system resilient under race conditions. It acknowledges reality: distributed systems are always mid-flight.