Replicated VM Storage: Keeping Disks Alive When Hosts Fail

Compute can be replaced in seconds. Storage cannot. If the only copy of a filesystem was on the local SSD of a reclaimed Spot host, there is nothing left to recover.

The guide to preemptible VMs names local-disk loss as a primary risk. This article covers why ephemeral storage is incompatible with reliable use of Spot capacity, how replication differs from backing up to S3, and how volume replication works in microsrv.

In short: a replicated volume syncs continuously to another node and survives a host failure. A backup answers a different question: rolling back logical mistakes.

1. Ephemeral vs persistent: different failure semantics

An ephemeral (local) disk is tied to the lifecycle of an instance or physical host. Terminate the instance or lose the hardware, and its contents disappear with the server. This storage is inexpensive and often fast (high IOPS), but it suits only:

temporary build caches;
scratch data that can be recomputed;
layers already externalized to remote storage.

A persistent, replicated volume exists independently of any single compute node. Stop the VM, migrate it, boot it on another host: the volume stays with it. For stateful services this is a baseline requirement, not an optimization.

With raw Spot capacity, teams often pair cheap compute with local disks. The first eviction wipes the data, and the vCPU savings evaporate with it.

2. Replication is not the same as backup

Confusing backups with replication gets expensive:

	Backup (e.g. snapshots to object storage)	Continuous replication
When it writes	On a schedule or manually	Continuously or with a short lag
RPO	Minutes to hours between points	Seconds or less
Purpose	Roll back logical mistakes, ransomware, human error	Survive failure of the current host
Recovery time	Create a volume from a snapshot, attach, start the service	Rebind an already synchronized volume

Backups are always necessary. But when an eviction notice gives you 30–120 seconds of warning, rushing to restore the latest snapshot from S3 is a bad bet. Fail over instead to a volume that has been replicating alongside your writes the whole time.

3. Sync, async, and the cost of RPO

Replication comes with different guarantees:

Synchronous. A write completes only after the replica acknowledges it. Minimal RPO (often ≈ 0 for acknowledged writes), higher I/O latency.
Asynchronous. The write is acknowledged locally and the replica catches up afterward. Less impact on write latency, but a sudden primary failure can lose the most recent unreplicated writes.
Periodic snapshots. Simplest to operate, weakest RPO, essentially classic backup.

Before building orchestration on Spot capacity, decide what RPO you can accept after a hard failure. A CI cache can lose minutes. PostgreSQL with user transactions needs seconds or less. And remember: uncheckpointed in-memory state can still vanish during a cold restart, no matter how good your replication is.

4. Coupling with live migration and cold restart

Live migration moves RAM and processes. The disk is not copied from scratch at cutover: the target host must already see a current volume, through networked or distributed storage or a warmed-up replica. Otherwise the guest resumes after stop-and-copy with no root filesystem.

When a hard failure interrupts pre-copy:

compute on the source host is gone;
the orchestrator boots the VM on a new node;
the volume attaches from the replica;
networking (see the eBPF layer) restores a stable IP and routes.

Disk data survives and the application goes through a normal boot. You get controlled recovery: the instance no longer vanishes together with its disk.

5. How microsrv preserves volumes

microsrv treats replicated storage as part of the base model, not an enterprise add-on:

disk volumes are continuously synchronized and survive Spot host reclamation;
during a graceful migration the volume is available on the target alongside the migrated RAM;
on sudden failure the VM restarts automatically from the replicated copy;
applications don’t need to push every piece of state to S3 just because the compute is interruptible.

Together with the orchestrator and eBPF networking, this turns interruptible capacity into VMs that can run more than batch jobs, including services with a local filesystem, at about 25% savings versus standard cloud rates.

6. Practical checklist

Never keep the only copy of your data on the ephemeral SSD of a Spot instance.
Separate backup (logical recovery) from replication (surviving host failure).
Set an explicit RPO target for hard failures and choose synchronous or asynchronous replication deliberately.
Test both paths: eviction with live migration, and cold restart from disk.
On microsrv, volume replication is built in. Application-level checkpointing still makes sense for business logic, but it should not be the only thing standing between you and filesystem loss.

Sources and further reading

SNIA — Online Dictionary: replication, see also Data replication (Wikipedia)
Linux block layer — DRBD and Ceph RBD as reference replicated-volume designs
KVM storage — QEMU block jobs / live storage migration
Companion: live migration (why the target must already see the volume)

How we think about RPO: under sync replication, acknowledged PostgreSQL writes have RPO≈0; under async, the last few seconds can be lost on hard failure. Test both paths: graceful eviction and cold restart.

FAQ

Is replication a backup? No. Replication survives host loss (RPO seconds); backup (snapshots to S3) rolls back logical errors. You need both.

Do I still need to checkpoint to S3 on Spot? Yes, for application-level invariants, but not as the only defense against filesystem loss.