High Availability for Network Automation Systems

Most teams learn “high availability” by protecting what is obviously critical: spines, leafs, border routers, firewalls, links, power feeds, and route reflectors. Then they automate the network, often successfully, and quietly introduce a new single point of failure: the automation and control systems that now define how the network is operated.

That blind spot only shows up when the pressure is real.

A fabric controller becomes unreachable right in the middle of a production incident. A change must be made now. The on-call engineer asks two deceptively simple questions:

This is the deeper truth: modern networks are not just routers and switches. They’re a coupled system of data plane + control plane + management/automation plane + observability plane. If you want genuine resilience, you must engineer HA across all of them, especially the software stack that governs intent, change, and evidence.

Let’s start with the cleanest case, because it clarifies the entire problem space:

Juniper Apstra is management and assurance, not the traffic path. If Apstra becomes unavailable, the spine–leaf fabric keeps forwarding with the last applied configuration; what you lose is orchestration, intent validation, and Apstra’s own analytics/UX, unless you designed around that.

That model is not unique to Apstra. It’s the right baseline for most automation systems:

So the network may stay “up,” but your ability to operate it may degrade sharply.

And in 2025 realities, “operate” is often the difference between a 10-minute incident and a 6-hour outage.

It’s tempting to lump all controllers into one box and call it “the orchestrator.” But the failure blast radius depends on what the system actually controls.

Apstra is designed to manage intent, generate configurations, deploy them, and continuously validate state against intent. It can be run as a VM cluster with a controller node and worker nodes to scale resources and improve resilience. Juniper Networks

Cisco APIC is the point of policy configuration and the place where operational stats/telemetry are processed for visibility and health; APIC is deployed as a cluster, with Cisco recommending a minimum of three controllers for HA and scale. Cisco+1

ACI is still not your traffic path either, but it is far more operationally central because the entire fabric’s policy model lives there. You can often keep forwarding during APIC outages, but your ability to make safe policy changes, troubleshoot via the platform, or restore certain workflows can become constrained.

In hyperscalers, the “controller” is rarely a single product. It’s usually a distributed automation platform made of:

These are software services, meaning they inherit every HA problem of modern distributed systems: leader election, databases, queues, consistency, idempotency, backpressure, partial failures, and degraded modes.

That is why hyperscalers treat network automation as a first-class production platform, owned with SRE rigor rather than “a set of scripts.”

Technically: yes. Operationally: it depends on how mature your system is.

In a traditional enterprise (or most organizations), a switch is still a switch: you can reach it via OOB/console/SSH and change configuration. The danger is not whether you can, it’s what happens after the controller returns:

This is why this article emphasized the real contract:

That reconciliation step is the difference between “incident mitigation” and “planting the seed of the next incident.”

If your observability depends on the orchestrator’s data stores and UI, then yes: that platform’s dashboards and validations go dark.

But the devices do not stop emitting signals.

Switches continue generating syslog, responding to SNMP, exporting streaming telemetry, producing gNMI/NETCONF state, exporting flow logs, and so on. What changes is whether you still have a separate ingestion + storage + alerting pipeline that continues functioning when the controller is down.

So the right question becomes:

Did you build observability as an independent plane, or did you outsource “evidence” to the controller UI?

“The world doesn’t stop,” but you should not depend exclusively on Apstra (or any controller) as your only source of monitoring; you need an independent stack for logs/metrics/alerting.

Break-glass is widely misunderstood. Many teams treat it as “SSH in and do what you must.”

That is not break-glass. That’s just “normal operations with heroics.”

In most environments, the organization relies on policy and culture:

That’s a human trust chain. It works… until it doesn’t.

And when it fails, it fails in the worst possible way: silent drift, untracked configuration deltas, compliance gaps, and the next automated push “healing” the network back into the broken state.

In most companies, preventing unauthorized change is largely trust-based because the environment does not technically prevent direct SSH changes.

At hyperscale, the philosophy flips:

The exact approval chain varies by company and system, but the consistent pattern is that access is gated by tooling (ticket metadata, limited scope, strong identity, full audit). The reason is simple: in a fully automated environment, out-of-band changes are not just “risky” (no… scrap that… they are very dangerous!), they are incompatible with the operating model because automation will continuously attempt to converge state to intent.

Hyperscalers also evolve mechanisms to prevent automation from blindly overwriting emergency changes until humans reconcile them, which we describe as “override prevention mechanisms” and “Andon cords.”

This matters because it’s the missing engineering layer in traditional orgs: they have “a controller,” but they don’t have the socio-technical safety system around it.

If you take nothing else from this article, take this:

The HA goal is not “controller uptime.” The HA goal is “operational continuity under partial failure.”

That means designing the automation ecosystem so that:

Let’s translate that into concrete architecture.

Cisco ACI APIC: run it as a proper cluster (minimum three controllers), size it for transaction rate, and treat APIC health as a first-class SLO because it is the policy brain. Cisco+1

Juniper Apstra: use VM clustering where appropriate, adding worker nodes to scale/absorb load (especially for analytics/offbox agents/IBA). Juniper Networks

But clustering alone isn’t enough.

Controllers are software. Software fails in messy ways: database corruption, broken upgrades, disk fills, container failures, expired certs, bad NIC bonding, you name it.

So you need tested backup/restore and off-box copies:

A mature design has:

This is where many organizations lose the plot.

If the controller provides assurance dashboards, that’s great, but you still need a parallel pipeline:

The point is not duplicating UIs. The point is preserving signal continuity when the controller UI is unavailable.

This is your “break-glass engineering.”

You need three things, and they must be designed before the outage:

The reconciliation workflow is the “closing of the loop” that prevents the next automation run from undoing your emergency fix.

Here’s what a grown-up break-glass process looks like in reality, whether you run Apstra, ACI, or an in-house stack.

If engineers routinely bypass the controller because it’s “faster,” your automation platform will never become reliable. You’ll be permanently stuck in drift and fear.

Even if you cannot implement hyperscaler-grade gating immediately, you can move beyond the trust model:

The source document explicitly highlighted the operational need to define who can do it, how changes are recorded, and how reconciliation happens.

Not “later.” Not “when we have time.”

Reconciliation is part of mitigation completion:

This is how hyperscalers stay sane: they treat the automation platform as the system of record, and everything else is either forbidden or reconciled quickly.

Once you accept that automation is production software, the conclusions become obvious:

Also, rigorous postmortems and engineering follow-through so the same class of failure does not require break-glass again.

That is the real maturity curve:

Break-glass is not a shameful workaround, and it’s not a sign that automation failed.

Break-glass is proof that you’re operating a real system under real failure.

The difference between hyperscalers and traditional organizations is not that hyperscalers never need break-glass. It’s that hyperscalers engineer break-glass as a controlled, auditable, time-bound safety valve, and then they build the platform improvements so they need it less over time.

If you want high availability in modern networks, you must design HA not only for switches and links, but for:

Because in the incident that matters most, the question won’t be “is the fabric forwarding?”

It will be: can you still see, act, and recover (safely) when your automation brain is unavailable?

That's the whole point!

Leonardo Furtado