Failover: when a server stops

Failover on a cluster - what moves, what waits, how long it takes, and what happens when the server comes back.

When a server of a cluster stops answering, its apps start on another server of the cluster. Nobody clicks anything.

  • Failover starts at two servers. On a cluster of one, nothing moves.
  • It is on by default. An admin can turn it off per cluster: then everything waits for its own server to come back.
  • With two servers, failover works only while Deplo is up.
  • With three or more, it also works while Deplo is down, as long as more than half of the cluster's servers are up and the leader runs on one of them.

See When a cluster is cut in two for what that means.

When a server stops

AfterWhat happens
About 5 seconds of silenceThe cluster's leader starts moving that server's apps without stored data
About 10 secondsThe server reads Offline on the Servers page, and its apps read Offline too
About 25 secondsThe silent server has stopped, by itself, everything that must never run twice: its databases, its apps with stored data and its running cron jobs. The leader starts moving those

Nothing that keeps data moves earlier than that, on purpose. A server that is only cut off from the network is still running. The leader waits until that server has stopped its own databases before it starts them somewhere else. That is the first promise, "Never two writers". Apps without stored data do not wait: two copies of those harm nothing, and the server removes its own when it is back.

Nothing moves at all when the trouble looks like the cluster's own, not the server's:

  • the leader lost its own connection
  • too few servers answer the leader: half of them or fewer, unless it is exactly half and Deplo answers too. See When a cluster is cut in two

25 seconds is when moving starts for anything with data, not when visitors are served again. That also takes the app's own start and a restore. See How long it takes.

How long it takes

Measured on a cluster of two servers, counting from the moment one of them lost its power:

What ran on the stopped serverMeasured
Replicas of an app on a Cloudflare domain, spread over both serversNo downtime. Cloudflare stopped sending visitors to the stopped server within about 5 seconds: one request in that time failed
An app without stored data, on that server onlyRunning on the other server after about 10 seconds
An app with a Volume, with replication onVisitors served again after about 46 seconds, putting its copy in place included
A database with a 4 MB backupIts restore took 17 to 24 seconds, on top of the wait
The server comes backAbout 20 seconds after it answers, its old copies stop, and what its database still held is saved as a backup

A bigger database takes longer to restore. Its card says how long, from the speed of its last restore. See Databases.

What moves, and what waits

Order: databases first, then apps with a domain, then the rest, oldest first.

What ran on the serverWhat happens
An app with no stored dataStarts on another server, exactly as its last deploy left it: same version, variables, files, ready check and automatic restart
Replicas of an appStart on the other servers. See Replicas across servers
A databaseRestored on another server from its latest backup, or its standby copy takes over. See Databases
An app with a Volume or FilesWith replication on, starts on another server with the copy of its data kept there. Otherwise it waits for its server. See Apps with stored data
An app with a BindNever moves: the folder lives on that one machine
A pull request previewNever moves. See Pull request previews
A cron job that was runningMarked Lost: its server stopped, and never run again on its own. See Cron jobs

An app keeps answering under its domain only if the domain follows it. See Addresses and Cloudflare.

What cannot move yet

Activity says why, once a day, until it is fixed:

  • An app that was never deployed since it joined the cluster. The leader has nothing to start it from: web cannot fail over yet: Deplo has no saved copy of its last deploy to give the cluster's leader. It is sent with the next deploy. Cluster eu will not move it to another server until that is fixed. Deploy it once.
  • A server whose server agent is too old never takes work over: The server agent on server-b is too old to take over Apps when another server in cluster eu stops. Deplo updates it on its own. Nothing to do: wait.
  • A Compose app that builds its own images: web does not fail over yet: its compose file builds images, and they exist only on the server that built them. A compose file that names published images does.

Waiting for room

What finds no server waits, and Deplo tries again every 30 seconds.

  • The app's header and its Settings -> General -> Server card say Waiting for room in eu. A database says it on its Failover card.
  • Inside the cluster on the Servers page, the banner 2 Apps waiting for room lists them, with "Deplo tries again every 30 seconds."
  • The fix: add a server to the cluster, or lower the app's memory.

One alert per failover that left something waiting, on by default: Apps waiting for room after a failover. Its title is web waiting for room in eu, or 3 Apps waiting for room in eu, and it says: "server-a stopped answering. No other server has room for web yet. Deplo tries again every 30 seconds."

Room is not the only reason to wait. Activity and the alert name the real one:

Activity saysThe alert says
web is waiting in eu: server-a stopped answering and it did not start on another server"web did not start on another server."
web is waiting in eu: server-a stopped answering and no other server has its image yet"No other server has the image of web yet."
web is waiting in eu: server-a stopped answering and no other server is online and ready"No other server is online and ready for web."
web is waiting in eu: server-a stopped answering and no other server has taken it yet"No other server has taken web yet." Read back after Deplo reconnected to the leader

With any of those the alert's title reads web waiting to move in eu, or 2 Apps waiting to move in eu.

What Activity says

Every entry is by Deplo, in the team that owns the app or database:

EntryWhen
Moved web to server-b: server-a stopped answeringAn app moved
Started a replica of web on server-b: server-c stopped answeringA lost replica was replaced
Restored db on server-b from the backup of 14:15A database came back from a backup
Moved web to server-b with its data replicated there: server-a stopped answeringAn app with stored data moved with its replicated copy
web is waiting for room in eu: server-a stopped answering and no other server has room yetNothing had room. See Waiting for room for the other reasons
web moved to another server of its cluster, so its generated address is now web-calm-fox-cb007118.deplo.siteIts deplo.site address changed

If Deplo was down during the failover, it catches up when it is back: the same entries, the new address and the alert.

The slower case

The times above are for a leader that was already leading. When the leader ran on the server that stopped, Deplo's own machine first takes over as leader, in 25 to 40 seconds (see The leader). Then the same waits apply:

What ran on itStarts elsewhere after
Apps without stored dataAbout 40 seconds: Deplo's machine takes over, then 5 seconds
Databases, and apps with a VolumeAbout a minute, then the restore. Measured: an app with a Volume and replication on was served again 82 seconds after its server lost its power

About 7 minutes in two cases. Just before a planned update, a leader gives every server 5 more minutes, and tells every server first. The leader that takes over asks the servers that are left whether that happened. It waits those 5 minutes, instead of risking the same database running on two servers, when:

  • the leader had given those 5 minutes, and its server never came back
  • the servers that are left cannot tell yet: they joined the cluster, or got a server agent that keeps count, less than about 6 minutes before. Deplo updates the server agent on its own

"Bring it back" from a backup is refused for the same time. A server that has answered the new leader even once is back on the usual times.

When a cluster is cut in two

A network cut can split a cluster. Neither side can tell whether the other is down or only out of reach, so only one side may go on. That is how nothing ever runs twice.

The rule. The leader keeps leading only while it is answered by:

  • more than half of the cluster's servers, or
  • exactly half of them and Deplo itself

Otherwise it moves nothing, and the servers it still reaches stop their databases and their apps with a Volume within about half a minute. The side that goes on is the one with more than half of the servers. When the sides are equal, it is the side Deplo is on.

What it means for your cluster

ClusterWhat to expect
One serverNothing changes
Two serversFailover needs Deplo up. If one server stops while Deplo is down too, or restarting for longer than about 25 seconds, the other server's databases and apps with a Volume stop as well. They start again by themselves within seconds of Deplo being back, and the stopped server's work moves then
Three servers or moreFailover works with Deplo down as long as more than half of the servers are up: one of three may stop, two of five
More than half of the servers lost at once (two of three)With Deplo up or not, the servers that are left stop their databases and apps with a Volume within about half a minute, until one of the others is back. If they are gone for good, the rest start again once those servers are removed from the cluster

Apps without a Volume keep running the whole time, in every row.

For failover that does not depend on Deplo being up, use three servers or more.

What a split looks like

WhereWhat happens
The smaller sideIts servers stop their databases and apps with a Volume within about half a minute
The larger side, its leader already leadingBrings them up from the latest backup, or promotes the standby copy, after about 25 seconds
The larger side, when Deplo's machine had to take over as leader firstThe same about 25 seconds after Deplo's machine started leading, except in the cases of The slower case
Both sidesApps without a Volume keep running. The server that was cut off removes its copy when it is back

When the leader is on the larger side and Deplo on the smaller one, Deplo shows its own side's servers as offline and cannot see the other side until the split heals.

What you read meanwhile

  • Inside the cluster, the banner Failover is not working right now: "Only 1 of 3 servers answers this leader, so nothing moves. Failover works again once 2 do."
  • Activity, one silent server: server-b is not responding in cluster eu., then server-b has not responded for a minute in cluster eu, so it stopped the databases and Apps with files that must never run twice.
  • Activity, more than half of the servers silent: Too few servers answer in cluster eu, so nothing moves and server-b no longer counts as answering., then Too few servers answer in cluster eu, so server-b stopped the databases and Apps with files that must never run twice.
  • Bringing an app back from a backup is refused meanwhile: "The cluster's leader cannot reach its servers right now. Try again in a minute."

When a server comes back

A restart, or a network cut that healed. The server:

  1. Starts nothing of the cluster's until it has heard from the leader. Anything that started by itself at boot is stopped again first.
  2. Stops and removes its own copy of every app and database that moved away while it was gone, the app's Cloudflare connector included.
  3. Starts again what is still its own. Work that found no room elsewhere goes back to it.

Until it has heard from the leader, a deploy or a start of a cluster app on it fails with: "this server has not heard from its cluster's leader, so it starts nothing of the cluster's yet".

One Activity entry and one alert per return, per team, naming only that team's own apps and databases:

  • server-a is back in cluster eu. Its copy of web was stopped: web runs on another server now.
  • Several: Its copies of api, web and worker were stopped: they run on other servers now.
  • The alert is Server back in its cluster, on by default.
  • A return that stopped nothing only says server-a answers again in cluster eu.

Replicas go back to an even spread at the app's next deploy, not at once.

After a power loss, the old copies can run for a few seconds at boot, before the server stops them. Nothing reaches the live database from there: what an old copy wrote in that window stays with the data described below.

Data left on the server

A server that comes back may still hold data that the moved database or app wrote before it stopped and that never reached a backup. Deplo does not throw it away silently.

Before its copy is removed, what the server still held is saved as a backup. This covers a database and an app with a Volume. The server does it by itself, also while Deplo is down.

It goes where the backups of that database or app already go, a bucket or another server's disk:

Where its backups are keptWhat happens
A bucketSaved, and listed with its backups as Left on server-a after the failover at 14:32
Another server of the clusterSaved the same way. The returning server sends it to that server directly
A server that is not in the clusterNot saved. The data stays on the returning server. Keep the backups in a bucket, or on a server of the cluster
Nowhere (no backup schedule)Not saved. The data stays on the returning server

Without a bucket, Deplo keeps a cluster's backups on another server of the cluster, so the default setup is covered.

  • It is encrypted before it leaves the returning server, with the destination's key. The server that stores it never sees the data in the clear and holds no key to open it.
  • A server only accepts such a backup from a server of its own cluster, and only the one file the cluster expects from it.
  • The data is removed from the returning server only once the destination has confirmed it holds the whole backup, and only while the database or app still runs on another server. One that was moved back to that server meanwhile keeps its data there, untouched, and nothing is saved.

It keeps trying

If the destination cannot be reached when the server comes back, the data stays on the server and the server tries again on its own: after 1 minute, then 2, 4, 8 and so on, never more than an hour apart, for as long as it holds the data. A restart of the server does not stop it. Nobody has to press anything.

  • When a later try succeeds, the backup appears with the same label, and Activity gets one line by Deplo. There is no second alert.
  • A repeat never leaves two files: a try that was cut short is written over by the next one.

What Activity and the alert say

SentenceWhen
What it still held of db was saved as a backup, labelled "Left on server-a".Saved
What it still held of db could not be saved as a backup, so it stays on server-a: the backup failed, then the server's own error in bracketsThe try failed
What it still held of db could not be saved as a backup, so it stays on server-a: its backups are kept on server-b, and server-a could not send it there.The other server could not be reached
server-a tries again on its own, and Activity says when it is saved.Added once after either of the two above
What it still held of db could not be saved as a backup, so it stays on server-a: its backups are kept on server-x, which is not in cluster eu.Its backups are outside the cluster. Nothing is retried
What it still held of db could not be saved as a backup, so it stays on server-a: the cluster had no backup destination for it.No backup schedule. Nothing is retried
server-a saved what it still held of db as a backup, labelled "Left on server-a".A later try succeeded
server-a saved what it still held of db, but Deplo could not match the file to db's backups, so it is not listed.Saved, but not where that database's backups go

A failed try can name the other server: reach the server that holds the backups, the server that holds the backups did not confirm it, or the server that holds the backups refused it: this server was not told to keep that. The last one means the two servers have not both heard from the cluster yet. The next try passes.

What you can do with it

The backup is in the Backups tab of the database or app, with Download, Restore and Delete in its menu.

  • It is never deleted by "keep the last N", and it does not count towards N.
  • Deplo never restores it on its own. A later failover restores the newest ordinary backup.
  • Restoring it costs everything written since the failover. The dialog says: "This overwrites db in place with what server-a still held when it came back." and "The database is stopped and its current data is wiped: everything written since the failover is lost."
  • Deplo does not merge it into the live data. Download it to look at what is in it.
  • It is a copy of the data folder as the old server left it, not a dump. A PostgreSQL database restored from it recovers by itself, as after a crash, and takes a new base backup before its change log counts again.
  • Restore needs Restore backups, Delete needs Delete backups, and downloading an app's also needs Reveal secret values.

The row's date is when it was saved. The label carries when the failover happened: the time alone when it was today, 3 Mar, 14:32 otherwise, and "Left on a removed server" once that server is gone.

An app with only a Bind, and an app that keeps nothing, have nothing to save.

Over the API: App.placementNote, Cluster.waiting, BackupRun.leftAt and BackupRun.leftOnServer. Over MCP: get_app, get_cluster and list_backup_runs return the same.

Did this page help you?

On this page