Using Kafka As a Database
Using Kafka as a database works by storing every state change in an event log. You then build local tables from those logs so apps can read current values. Many teams try this pattern to keep data pipelines simple and fast. Still, teams often grant broad topic permissions so developers can debug state issues. When roles are too broad, anyone with topic access can read sensitive records or break state. How using Kafka as a database works in practice A standard database overwrites old rows when
Authorization as a Service + Database Access Proxy: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
Using Kafka as a database works by storing every state change in an event log. You then build local tables from those logs so apps can read current values. Many teams try this pattern to keep data pipelines simple and fast. Still, teams often grant broad topic permissions so developers can debug state issues. When roles are too broad, anyone with topic access can read sensitive records or break state.
How using Kafka as a database works in practice
A standard database overwrites old rows when new updates arrive from an app. Event streams take a very different path to store and manage data. Every insert, update, or delete writes a new message to an ordered log. To avoid running out of disk space, teams turn on log compaction for key topics. Compaction keeps only the newest message for each unique record key in the log.
Logs alone are hard to query when an app needs one specific record right now. Reading a whole topic from the start takes too much time for user requests. To fix this, teams pair compacted topics with an embedded stream processor in the app. The app reads the topic on startup and builds a fast local state store. When new events arrive, the app updates its local table right away.
What native stream permissions solve and what they miss
Stream clusters include native access control lists to lock down topics and consumer groups. Platform teams use these rules to decide which service accounts can read or write events. These native rules work well for service-to-service traffic in a steady state. They stop unknown apps from reading private topics or writing bad messages to the log.
The gap appears when engineers or support tools need to inspect the data or companion stores. Native topic rules only check who logs in and which topic they ask to read. They do not inspect what individual queries or topic reads return in flight. If a topic holds emails, phone numbers, or card numbers, readers get every raw field. Native cluster rules cannot rewrite matched fields in results or require approval for one read.
Authorization as a Service + Database Access Proxy: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
When to pair event logs with relational stores
Storing state in event logs fits workloads that process events by a single key. Payment routers and fraud checks run well on local state tables built from streams. Still, complex ad-hoc queries across many fields remain hard to run on stream stores. You cannot join five large tables on the fly without writing new stream code first.
Most teams end up running both event streams and standard relational databases side by side. The stream acts as the central log of changes for all backend services. Downstream connectors then sink those events into relational databases for reporting and support queries. This hybrid setup means teams must secure human sessions across both streams and SQL databases.
Govern statements and responses across databases and APIs
Good config and scoped roles are needed setup. They do not check single queries in flight or mask sensitive fields in returned rows. Across Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, and internal HTTP endpoints, hoop.dev runs beside each supported service as an open-source proxy, a sidecar for your database or API.
First, hoop.dev guardrails block banned queries before they run. Then hoop.dev data masking rewrites matched sensitive values in responses in memory. The audit trail records every statement and verdict per resource. Furthermore, hoop.dev holds SOC 2 Type II and is open source on GitHub.
Frequently asked questions
Can an event stream replace my primary relational database?
It can replace a database for simple key-value lookups and event-driven apps. If your team needs complex multi-table joins or ad-hoc reporting, keep a relational database too.
How does log compaction keep storage sizes under control?
Background threads scan the topic log and remove older messages that share the same key. Only the latest value for each key stays in the log over time.
How do you delete a record when using an event log as storage?
You write a message with the record key and an empty value, called a tombstone. Compaction removes both the old values and the tombstone after a set wait period.