Cephfs S3
A cephfs s3 setup runs file storage and object storage on one Ceph cluster. You deploy CephFS for shared directories and the RGW daemon for S3 buckets. Many teams build this dual layout so old apps and new apps share disks. Engineers often create broad admin keys to test both storage pools fast. When those broad keys stay in use, one leaked secret exposes every bucket. How a cephfs s3 cluster works under the hood Ceph stores all data as raw objects inside a common storage layer called RADOS.
S3: the complete guide
Architecture patterns, implementation strategies, and security best practices. Delivered to your inbox.
A cephfs s3 setup runs file storage and object storage on one Ceph cluster. You deploy CephFS for shared directories and the RGW daemon for S3 buckets. Many teams build this dual layout so old apps and new apps share disks. Engineers often create broad admin keys to test both storage pools fast. When those broad keys stay in use, one leaked secret exposes every bucket.
How a cephfs s3 cluster works under the hood
Ceph stores all data as raw objects inside a common storage layer called RADOS. CephFS builds a POSIX file tree on top of those raw RADOS objects. It uses a metadata server to track file names, folders, and lock states. At the same time, the RADOS daemon serves an S3 API for web apps. Each service uses its own separate pool of disks inside the same cluster.
CephFS and S3 do not read the same files out of the box. A file written to a CephFS mount sits in the file data pool. An object uploaded through the S3 endpoint sits in the object data pool. If teams need both views on one file, they run a sync tool. Some teams also mount an NFS export that talks to the S3 pool.
Auth rules also split between the two storage types inside the cluster. CephFS uses native CephX keys that lock a client to specific folder paths. The S3 interface uses standard access keys, secret keys, and bucket policies. Admins must manage two sets of keys and rules across the storage cluster.
What native storage roles solve and what they miss
Native CephX keys and S3 policies control who can reach a storage pool. You can restrict a client key so it only mounts one folder path. You can also write a bucket policy that allows read calls from one app. These rules stop unapproved apps from reaching storage pools they do not own. They give your team a solid base for basic network and pool isolation.
Still, static storage keys stay valid until an admin rotates or deletes them. Engineers often copy S3 keys or CephX secrets onto laptops to debug issues. Once a key sits on a laptop, the cluster cannot tell who uses it. Two engineers using the same service key look identical in the cluster logs. Native rules check the key at login, not the person running the command.
S3: architecture patterns and best practices
Continue reading? Get the full guide.
Free. No spam. Unsubscribe anytime.
Native bucket and pool rules also lack fine checks for live data queries. A valid read key lets an engineer pull every object inside an S3 bucket. Those objects might hold raw emails, phone numbers, card numbers, or national IDs. The storage daemon sends the full file because the key has read rights. Native controls decide who logs in, not what individual requests return in flight.
Keeping shared storage keys off engineer laptops
Running two storage views on one cluster multiplies the secrets your team tracks. Every app, script, and admin needs either a CephX key or an S3 key. Storing these long-lived secrets in local text files creates a major risk. If an attacker steals one config file, they can read or wipe whole pools.
Teams reduce this risk when they stop handing out raw storage keys to people. Instead of sharing static secrets, you tie storage actions to a human login. You also check each command and mask sensitive fields like emails in results. This keeps raw card numbers and phone numbers safe during routine support tasks.
Check database and API queries on the wire
Identity tools, roles, and network rules decide who opens a session. Setup alone cannot check what each query does or mask sensitive values in a response once a session starts. When workflows touch Postgres, MySQL, SQL Server, MongoDB, ClickHouse, Kubernetes APIs, SSH, or internal HTTP APIs, hoop.dev runs beside those services as an open-source proxy, a sidecar for your Postgres or API. It inspects requests coming in and results going back on the wire.
Your login details and client tools stay as they are while hoop.dev guardrails refuse bad deletes or drops before they run. On responses, hoop.dev data masking rewrites matched sensitive values in memory. Furthermore, hoop.dev holds SOC 2 Type II, is MIT licensed, and writes every query and decision to an audit trail that generates evidence for HIPAA, GDPR, and PCI DSS. See the source on GitHub.
Frequently asked questions
Can S3 apps read CephFS files directly from the same pool?
No, CephFS and the S3 service write data into separate storage pools by default. To share files, you must export the S3 pool over NFS or sync data.
How do CephX keys differ from S3 keys in a Ceph cluster?
CephX keys let Linux clients mount CephFS folders and talk to cluster daemons. S3 keys work only with the HTTP object service to read or write buckets.
Why are shared S3 admin keys risky during storage debugging?
Shared S3 keys hide which person ran a delete or read call in logs. They also stay valid forever until an admin finds every copy and rotates them.