Immutable Servers
You build immutable servers by baking app files and OS dependencies into one read-only image. When you need an update, you deploy a fresh machine and destroy the old one. Most teams start this setup by turning off remote shell logins and blocking package updates. Still, teams still need ways to check live bugs or run one-off database tasks. When engineers keep wide admin roles for those tasks, one bad command can break state. How immutable servers work in production In a standard setup, engin
SSH Bastion Hosts / Jump Servers + Immutable Backups: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
You build immutable servers by baking app files and OS dependencies into one read-only image. When you need an update, you deploy a fresh machine and destroy the old one. Most teams start this setup by turning off remote shell logins and blocking package updates. Still, teams still need ways to check live bugs or run one-off database tasks. When engineers keep wide admin roles for those tasks, one bad command can break state.
How immutable servers work in production
In a standard setup, engineers log in to live machines and change files in place. Over time, each machine looks a bit different from the rest of the fleet. This shift is called drift, and it causes surprise crashes during code rollouts. With immutable servers, no one changes a machine after it boots up in production.
The workflow starts in your build pipeline when a developer merges new code. An automated build tool takes a clean base OS image and adds your app code. It also bakes in system libraries, runtime tools, and static setup files. The build pipeline then saves this finished image as a versioned snapshot in a registry.
Next, your deployment tool boots new virtual machines or containers from that exact image. Load balancers run health checks against the new fleet before sending live traffic over. Once the new machines handle all traffic, the platform deletes the old instances right away. If a release fails, you boot the prior image version and swap traffic back.
Because the host itself never holds unique state, you store all data outside the machine. Databases, object stores, and log streams live on separate managed services over the network. You pass secret keys and network paths into the machine at boot time.
What image baking solves and what it leaves open
Baking images stops host drift and makes every code rollout repeatable and fast. You know that staging and production run the exact same OS files and binaries. If a hacker alters a local file, replacing the instance wipes out that change.
Yet this build model only controls the machine image, not live admin actions. Engineers still need to inspect memory, run database migrations, or fix broken customer records. Cloud roles and network rules decide who can start a debug session or database tool. They do not inspect what individual commands or queries do once that session starts.
SSH Bastion Hosts / Jump Servers + Immutable Backups: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
Even if the local disk is read-only, a live session holds real app permissions. An engineer running a diagnostic command can still read raw customer emails or card numbers. A bad query from an admin container can still drop a live production table. Native cloud roles govern initial login, not the safety of each command in flight.
Managing stateful data and secrets alongside read-only hosts
To keep hosts stateless, you must separate code from data and runtime secrets. Never bake API keys, database passwords, or TLS certificates into your machine image. Anyone who pulls that image from your registry could read those secret values.
Instead, let the machine fetch secrets from a secure store when the app starts. Keep local disks read-only so stray scripts cannot write sensitive files to the host. Send all app logs and metrics out to a central log service right away. If a host crashes and disappears, your team still keeps the full log trail. This clean split lets you destroy and replace any compute node at any time.
Check database and API queries on the wire
Identity tools, roles, and network rules decide who opens a session. Setup alone cannot check what each query does or mask sensitive values in a response once a session starts. When workflows touch Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, or internal HTTP APIs, hoop.dev runs beside those services as an open-source proxy, a sidecar for your Postgres or API. It inspects requests coming in and results going back on the wire.
Your login details and client tools stay as they are while hoop.dev guardrails refuse bad deletes or drops before they run. On responses, hoop.dev data masking rewrites matched sensitive values in memory. Furthermore, hoop.dev holds SOC 2 Type II, is MIT licensed, and writes every query and decision to an audit trail that generates evidence for HIPAA, GDPR, and PCI DSS. See the source on GitHub.
Frequently asked questions
How do you apply security patches to immutable servers?
You never run patch tools on live machines in production. Instead, you update the base image in your build pipeline and bake a new version. Then you boot new instances from that patched image and terminate the old fleet.
Where do logs and user data go if the server gets destroyed?
You stream all system and app logs to an external log service in real time. User data and app state live in external managed databases or cloud object storage. This ensures zero data loss when an instance shuts down.
Do you still need remote access for immutable servers?
You should turn off routine shell logins so engineers do not edit live files. Still, teams still need controlled ways to run live diagnostics or query attached databases. Using command-level checks and data masking keeps those rare debug sessions safe.