Hybrid Workloads in a Single VPC: Combining Containers and VMs in microsrv

Engineering teams are often handed a false choice: bet everything on Kubernetes and absorb its complexity and container-sandboxing limits, or run classic virtual machines for everything and pay for idle memory and virtualization overhead.

In practice, most real-world stacks want neither extreme. They want a heterogeneous setup where each component runs where it fits best:

Lightweight web backends and databases live in containers that boot in a split second with a minimal RAM footprint.
Build runners, test suites, and system tools need full virtual machines with kernel access, hardware virtualization, and root control.

On microsrv, containers and virtual machines are first-class peers in one platform. They share the same private network (VPC), mount the same block volumes, and talk to each other at eBPF speed.

Let’s look at this hybrid stack through a concrete case: a self-hosted Git and CI/CD platform.

In short: in microsrv, containers and KVM VMs are peers in one private VPC. The database and web service run as containers, the CI/CD runner as a VM, connected over eBPF with no public IPs.

1. The Architecture Scenario: Self-Hosted DevStack

Suppose a team wants its own independent source-code and CI/CD service. The standard three components:

Git Web Interface (Gitea / Forgejo): serves the web UI, handles Git HTTP/SSH traffic, and triggers webhooks.
Relational Database (PostgreSQL): stores repository metadata, user accounts, pull request states, and audit trails.
CI/CD Build Agent (Gitea Actions Runner): executes pipeline jobs, compiles source code, builds container images (docker build), and runs test suites.

Running this stack solely inside containers or solely inside separate VMs forces trade-offs either way. Here is what those trade-offs are and how microsrv removes them.

2. Containers: Instant Startup & Disk Persistence for Services

For the Gitea web service and PostgreSQL database, containers are the natural choice: they start in 1–2 seconds and carry no idle hypervisor overhead.

Databases inside standard Kubernetes, however, are a chronic headache, because container filesystems are ephemeral. Every node reschedule demands Persistent Volume Claim (PVC) bindings and CSI plugin drivers, or the written data is gone.

In microsrv, containers run on top of Stateful OverlayFS:

Immutable Base Image: OCI image layers for PostgreSQL or Gitea are pre-cached across compute nodes.
Persistent Root Writes: all database writes, table updates, and configuration edits go directly to a replicated NVMe volume.

The database container restarts or migrates across physical nodes, and 100% of its data stays intact. No Persistent Volume Claims to configure: disk persistence is a platform guarantee.

3. KVM Virtual Machine: Full Hardware Parity for CI/CD

Web apps and databases run happily in containers. Build runners want the opposite.

Building containers inside another container (Docker-in-Docker) on standard container platforms is a fight with security constraints:

it needs risky --privileged flags that expose the host kernel;
emulators (QEMU, KVM, nested virtualization) and custom cgroups v2 namespaces often refuse to run;
untrusted build scripts can reach the shared host container runtime.

In microsrv, the build runner gets a dedicated KVM Virtual Machine.

Inside the VM you have an uncompromised Linux environment:

native access to /var/run/docker.sock and the Docker daemon for fast image builds and caching;
freedom to install kernel modules, manage systemd services, run nested hypervisors, and execute privileged tests;
hardware-enforced isolation: failed build jobs or rogue processes can’t touch neighboring services.

4. Unified Private VPC: Security with Zero Public IPs

The strongest part of the hybrid setup on microsrv is network connectivity:

+-----------------------------------------------------------------------+
|  Private VPC: 10.42.0.0/24 (Project Network)                          |
|                                                                       |
|   Gitea (Container)          Postgres (Container)   Runner (KVM VM)   |
|   IP: 10.42.0.11             IP: 10.42.0.10         IP: 10.42.0.20    |
|   Port: 3000                 Port: 5432             eBPF Mesh         |
+-----------------------------------------------------------------------+

Database hidden by design. PostgreSQL binds to port 5432 exclusively on its private IP 10.42.0.10. It has no public IPv4 address and can’t be scanned or reached from the public internet; only Gitea talks to it over the private interface.
Zero-latency eBPF mesh. Traffic between containers and VMs inside the VPC is processed directly inside the Linux kernel via eBPF maps, with no intermediate bridge proxies or routing bottlenecks.
Secure edge gateways:
- The Gitea web interface is published through the HTTPS gateway with automated TLS certificates on git.msrv.space.
- Administrative SSH access to the runner VM goes through the SSH gateway with Ed25519 authentication (ssh -A runner@msrv.space).

None of your services need an expensive public IPv4 address, which cuts costs and shrinks the attack surface at the same time.

5. Spot Economics with Live Migration

This architecture also has a pleasant bottom line (see Spot vs On-Demand for a real-world savings model).

All three services, the web application, the database, and the build runner, can run on interruptible Spot capacity at up to 25% savings, without giving up the conveniences of a regular cloud.

In traditional clouds, a database or long-running CI build on Spot instances is a gamble: when the host is reclaimed, active builds fail and database connections drop.

In microsrv, automated live migration protects your workloads:

For containers: the orchestrator pre-copies sandbox memory and atomically updates eBPF socket maps. Active Git push sessions and web requests continue uninterrupted.
For the runner VM: RAM state migrates smoothly to a standby server while compilation and container builds keep running, no restart.

Summary

Containers and virtual machines in one microsrv VPC, side by side:

Workload	microsrv Solution	Key Advantage
Web Service (Gitea)	microsrv Container	Sub-second startup, low RAM overhead, strong sandbox security
Database (PostgreSQL)	microsrv Container + Stateful OverlayFS	NVMe-persisted data without external PVC configurations
Build Agent (Runner)	KVM Virtual Machine	Native Docker-in-Docker, full root control, isolated builds
Networking	Private VPC (eBPF)	Zero public IPs, secure perimeter, edge gateway routing
Resilience	Live Migration on Spot	Cost savings with no dropped TCP connections or interrupted jobs

Run containers where you need density. Run virtual machines where you need freedom. Both live in the same microsrv cloud.

Sources and further reading

gVisor — gVisor docs (userspace kernel for containers)
KVM — KVM documentation
OCI — Image Specification
microsrv pillars: live migration, replicated storage, eBPF networking, and cost model

How we tested: Gitea + PostgreSQL as gVisor containers (Persistent OverlayFS on a replicated NVMe volume) + a KVM runner VM, all in one VPC 10.42.0.0/24; 10.42.0.10 (Postgres) is VPC-only, Gitea via HTTPS gateway <vm>.msrv.space, runner via ssh -A runner@msrv.space. Migration keeps all three peers on the same private subnet.

FAQ

Why not run the runner in a container? Docker-in-Docker needs --privileged and breaks cgroups v2/KVM nesting; a KVM VM gives you a real /var/run/docker.sock and hardware isolation.

Does this need a public IP? No. Outbound NAT for pulls, HTTPS/SSH gateways for ingress, and WireGuard for your laptop to join the VPC.