Stateful OverlayFS & Persistent Storage
How Stateful OverlayFS works in microsrv: persisting root filesystem modifications directly onto replicated NVMe volumes.
The Ephemerality Challenge in Kubernetes & Docker
In conventional container orchestrators (Kubernetes, Docker Swarm, Nomad), a container’s root filesystem is strictly ephemeral:
- Containers run on top of an immutable OCI image.
- Newly created files, logs, configuration modifications, and runtime package installations are written to a temporary writable layer stored locally on the physical host.
- When the container restarts, encounters a crash, or gets rescheduled (Pod Eviction / Reschedule) onto another host, this writable layer is permanently destroyed.
To preserve state, engineers are forced to provision external Persistent Volumes (PV/PVC), configure CSI storage plugins, and ensure application data is written strictly to designated mount paths (e.g., /var/lib/data).
Architecture of Stateful OverlayFS in microsrv
In microsrv, containers are architected as first-class stateful workloads. The container root filesystem is backed by Stateful OverlayFS, attached to a persistent, distributed NVMe block device (Volume):
1. UNIFIED MERGED ROOTFS (/)
[ROOTFS] The container sees a standard POSIX filesystem at / • Transparent overlay composition
Layer Composition:
- Lowerdir (Read-Only): The immutable layers of the OCI container image, cached locally on compute nodes.
- Upperdir & Workdir (Read-Write): Reside directly on the first attached volume (
volume_ids[0]). All write operations, file mutations, and deletions (whiteout markers) are persisted to the replicated NVMe volume. - Merged View: The virtual unified filesystem exposed to the gVisor Sentry kernel and guest application.
What is Preserved on Disk?
Thanks to Stateful OverlayFS, all filesystem modifications are preserved without manual volume mount configurations:
- Installed Packages & Dependencies: You can run
apt-get update && apt-get install -y ffmpegorpip installinside a running container—changes persist across container reboots. - Embedded & Local Databases: SQLite, DuckDB, RocksDB, LevelDB files, or PostgreSQL data directories (
/var/lib/postgresql/data) remain intact. - System & App Configurations: Edits in
/etc/nginx/,/etc/environment, or custom.envfiles are never wiped. - Application Caches: Disk caches survive container restarts and host migrations.
[!IMPORTANT] When provisioning a container in microsrv, attaching at least one volume (
Volume) is mandatory, as the first volume is used as the backing store for the Stateful OverlayFSUpperdir.
Persistence Model Comparison
| Scenario | Kubernetes Pod | microsrv Containers |
|---|---|---|
| Crash & Process Restart | Rootfs resets to base image | All files and modifications in rootfs survive |
| Stop and Start Cycle | Pod recreated from base image | Container resumes with identical disk state |
| Live Migration | ❌ Not supported | Volume lock atomically transferred to target host |
| Hardware Node Failure | Clean pod started with empty local rootfs | Cold restart reattaching the same replicated volume |
| Runtime Package Installation | Lost upon container reboot | Persisted on disk permanently |
Attaching Additional Storage Volumes
Beyond the primary volume allocated for Stateful OverlayFS, you can attach additional volumes for dedicated storage:
- Create a volume under Console → Volumes.
- Specify volume IDs in the
volume_idsarray during container creation or updates. - Additional volumes are exposed to the sandbox for high-throughput or isolated data workloads.