Skip to main content

High availability

A single proxy installation is enough for most environments, but the proxy can also run redundantly, across machines or even across data centres. This page explains how work is distributed over multiple proxy instances and which failover setups are supported.

High availability builds on the pull transport mode (see Transport modes): the automator queues all work centrally, and every connection is opened outbound by the proxy instances themselves. The push transport mode sends work to a single endpoint URL, so redundancy there is limited to what a load balancer of your own in front of that URL provides.

How work is distributed​

When a package calls executeOnProxy, the automator queues work for the Proxy account; the work contains the encrypted command to execute. Every proxy instance whose profile is configured with the account's Automator URL polls this queue, and each unit of work is claimed by exactly one instance: once claimed, other instances can no longer pick it up. The claiming instance executes the command and posts the encrypted result back, which completes the executeOnProxy call in the package.

Instances do not need to know about each other, and the automator does not need to know how many instances exist. Running a redundant setup is therefore purely a matter of running more than one instance with the same profile configuration.

Active/active​

Run two or more proxy instances with the same profile configuration (the same Automator URL, token and key). All instances poll for work:

Active/active setup: two proxy instances inside the customer network both poll the automator for work through the firewall, and each unit of work is claimed by exactly one instance
  • Work is distributed automatically: whichever instance claims the work executes its command, and every command is executed exactly once.
  • No load balancer, virtual IP or other shared infrastructure is needed.
  • When an instance fails, the remaining instances keep claiming work and processing continues without intervention.

Make sure every instance can execute all configured work: deploy the same scripts on every machine, and give each machine the same access to the internal systems those scripts reach.

Active/passive​

In an active/passive setup a second installation stands ready but does not run during normal operation. This is a warm standby: the proxy is installed and configured on the standby machine, but its service is stopped:

Active/passive setup: the active proxy instance polls the automator for work while the standby instance stays idle until failover, which needs no DNS, firewall or load balancer changes
  • During normal operation only the active instance polls for and executes work.
  • On failover, start the proxy on the standby machine and it takes over polling. Because every connection is outbound, failover needs no DNS, firewall or load balancer changes.
  • A hot standby, an instance that is running and polling, is simply an active/active setup. Choose active/passive only when commands must run on one designated machine at a time; otherwise prefer active/active, which fails over without any manual step.

When an instance fails​

  • A command that was being executed by the failed instance does not return a result. The executeOnProxy call fails with a timeout error, five minutes after the work was queued.
  • Work that was queued but not yet claimed is unaffected: it stays in the queue and is picked up by the remaining instances. When no instance is polling at all, queued work waits until the same five minute timeout expires.
  • Work is executed at most once. A claim is never given to a second instance, not even when the claiming instance fails, so a single executeOnProxy call can never run its command twice. Scripts do not need to protect against duplicate execution; a failure always surfaces as a timeout error, never as a repeated run.
  • An instance that comes back simply resumes polling; there is nothing to clean up or re-register.