Rewriting Ceph in Rust: the first month, built by humans steering AI agents

By Martin Verges

Ceph calls itself "the future of storage". We have believed that for a long time. croit is a founding member of the Ceph Foundation, we have built our company on Ceph for ten years, and I have been working with Ceph for more than fifteen. We have developers who work on Ceph all day, and a support team that looks after hundreds of customer clusters around the clock.

That is exactly why we started this project. When you run that many clusters, you learn what Ceph does well, and you learn, ticket by ticket, where it hurts. Large omap objects that bring RocksDB to its knees. CPU budgets that grow with every NVMe generation. Tuning that only a handful of experts understand. A code base that has grown for almost twenty years, in a language that makes memory safety the developer's job.

In short: we want Ceph to stay the right choice for the next twenty years, not just the last twenty. On the current code base we no longer believe that is possible. So we started rewriting it, in Rust, from the bottom up.

This post is about where we stand after the first month, and about how we built it: with a strongly human-driven, AI-agentic workflow.

Why a rewrite, and why Rust

Three reasons.

Performance on modern hardware. BlueStore was designed when NVMe was the exception. Today a single server has more NVMe bandwidth than the network that connects it, and the OSD's CPU cost per operation decides how much of that hardware you can actually use.

Memory safety. Security agencies and regulators increasingly push software vendors away from C and C++ and toward memory-safe languages. For a storage system that holds customer data, we think that is the right direction anyway. Rust gives us memory safety without a garbage collector and without giving up the control we need on the data path.

Understanding. Patching a twenty-year-old code base gets you local fixes. Writing it again, from our own specifications, gets you a design you can reason about, test and change.

We did not want a fork. We wanted something that drops into existing clusters. So the rule from day one: wire-compatible with Ceph, today Ceph 21.2.0. Monitors, managers, clients and classic OSDs talk to our components without any change. You can mix them in one cluster and move over one drive at a time: each drive is re-created as a FlashStore OSD and Ceph recovery fills it again.

Start with the hardest part: the OSD

We started with the OSD, the biggest and hardest single component of Ceph. If the foundation does not get faster and safer, nothing above it will.

The result is croit FlashStore OSD, a drop-in replacement for ceph-osd with its own object store for NVMe. Thread per core, placement groups owned by cores instead of locks, no locks on the hot path, acknowledgements only once data is durable on power-loss-protected NVMe.

The interesting part is how many roads we tried before settling on today's design:

• Network. Kernel TCP, driven through io_uring with one ring per core, works with any NIC, bond or VLAN the kernel drives. RDMA (RoCEv2) is built in as an in-place upgrade of a TCP session for clusters that have the network for it.
• Storage. The kernel NVMe tier (io_uring on the kernel NVMe driver) is the production tier: it is what our customers can run without giving whole devices to a userspace driver. SPDK remains an optional build.
• omap, the key-value side of every object, which RGW bucket indexes and CephFS directories live on. This is where support tickets hurt most, and where we experiment most. We made the omap engine pluggable and measure the engines against each other and against BlueStore on the same hardware:
  • a B+tree in our own store, today's default, which now beats BlueStore on average in our omap test set;
  • an LSM engine of our own, opt-in, with very cheap writes but harder read and delete paths;
  • RocksDB, as BlueStore uses it, and SplinterDB, both in development and not yet in production builds.

Each has trade-offs. The B+tree reads well and is simple; the LSM writes fast but needs careful compaction; RocksDB is proven but brings back exactly the stalls our customers open tickets about. We decide with measurements, not opinions, and we test with workloads taken from those tickets: bucket index shards with hundreds of thousands of keys, resharding under load, bulk deletes followed by listings, CephFS stray directories with a million entries.

Where we are after one month

These are the numbers from our latest benchmark run: 30 NVMe OSDs, AMD EPYC 9455P servers, 2 × 200 Gbit/s, kernel TCP, no RDMA, unmodified Ceph clients against Ceph 21.2.0 BlueStore on the same hardware. Averages are geometric means over the test family. Every test ran once per engine.

The full method and every single test are in our launch post and the benchmark report (PDF).

We also rewrote the client side. librustyrados and librustyrbd are drop-in replacements for the librados and librbd C API: existing binaries and QEMU/KVM pick them up without a rebuild. Against stock librados and librbd on the same cluster they are up to 2.1x faster for 4 MiB RBD writes, 1.9x for 4 MiB RBD reads and 1.7x for 4 MiB RADOS reads; mid-size reads still have a gap we are closing. Optional RDMA is built in, and a KVBM/NIXL plugin for GPUDirect RDMA straight to the OSD exists in code. End-to-end tests on GPU hardware start in the coming weeks.

It is not finished. Some omap tests are still below BlueStore, a few latency tails need work, and many features are preview quality. That is the honest state after month one.

croit FlashStore OSD is available as a preview for testing and evaluation. It is not yet recommended for production data. librustyrados and librustyrbd are available as a preview for testing and evaluation in the croit frontier release. They are not yet recommended for production data.

How we built it: humans in the driver's seat

Now the part people ask about most.

The first commit is from 9 October 2026. Since then: more than 8,000 commits on the main branch, in 33 days. Three full-time developers drove the project, and each of them kept agents working around the clock: 33 days × 24 hours × 3 is 2,376 hours of engineering time, the equivalent of 297 eight-hour working days, and with many agents working in parallel most of the time that is a lower bound. Nobody builds this in one human working month.

The workflow is strongly human-driven and AI-agentic:

• Humans decide. A small senior Ceph team sets the design, decides what gets merged, and validates every result on real hardware. Igor Fedotov, a long-time BlueStore developer and a well-known member of the Ceph developer community, and Md Mahamudur Rahaman Sajib, who has contributed many patches that made Ceph better upstream, work on the storage engine with me. Matthias joins as a fourth developer part time, and support engineers, consultants and marketing all feed in.
• Agents execute. AI agents write, review and test the code under that direction. Every change has to pass an extensive test suite and benchmarks against BlueStore and stock Ceph clients before it lands.
• The right model for each job. 27 different models have worked on this project so far. Large reasoning models do architecture, durability and root-cause hunts. Mid-size models implement and review. Small, fast models run lab jobs, scripts and checks. Some of them are our own models, running on our own GPU systems through AIplane, the same setup we described in How We Built AI Into the Way croit Works.

Some scale: the agents processed about 93 billion tokens, most of them re-read context, and generated about 290 million. Including people, AI, our own test labs and rented bare-metal servers, the first month cost roughly EUR 100,000. We expect two or three more cycles like it before general availability.

What a day looks like

My day is a conversation with several long-running agent sessions at once. Each of them is a manager for one topic: the OSD, the client libraries, CephFS, benchmarks. They plan, split the work and hand single tasks to worker agents, each in its own branch. They review what comes back, run the tests, and report to me.

I set the direction. Which design, which trade-off, what to focus on, what must never happen. The rules are written down and enforced in code: no copying of Ceph code (Ceph source is turned into our own specification first, implementation agents only see the specification); every fix names the layer of its root cause and checks sibling paths; no workarounds, no silent timeouts; and "never worse than BlueStore": any benchmark test where we lose is treated as a bug, profiled and fixed.

AI is the tool. We are in the driver's seat.

What did not work

Plenty.

• Agents report success too early. We had helpers report green on tests that were hanging. Since then, the only accepted evidence is the real exit code of a foreground run, and the lead agent re-runs everything itself before a merge.
• omap took several redesigns. Local patches on the first LSM design did not close the gap to BlueStore. We stopped tuning knobs, went back to the design, made the omap engine pluggable, and measure the engines against each other.
• Benchmarks find what tests miss. Just this week, per-thread CPU sampling in our benchmark harness showed one client thread spinning at 100% whenever an RBD image was open. Unit tests were green; the benchmark was not. Now a dedicated gate checks idle CPU.

None of this is magic. It is a lot of engineering discipline, applied by people who know Ceph, at a speed that only agents make possible.

Why we do it in the open

We want this to become part of the Ceph ecosystem, and we will contribute upstream. Our goal is that everyone who runs Ceph today has a future with Ceph tomorrow: faster, safer, and ready for AI workloads with RDMA and GPUDirect. The code is published as open source at github.com/croit/ceph-rust.

General availability is planned for the first quarter of 2027. We will release it when our Ceph engineers are confident it is ready for production data, not by a fixed date. All dates are indicative and subject to change based on croit customer demand.

Try it

First update croit to the croit frontier release. Then enable it (Settings, Add: croit enable frontier image and croit enable flashstore, value on). Then, per drive: Disks, Set to OSD, Implementation: FlashStore. You need NVMe with power-loss protection, formatted to 4 KiB sectors without metadata or protection information, and Linux 5.14 or newer with io_uring enabled.

Or let us show you: tell our engineers what to test, and they run it live in front of you.