We are rebuilding Ceph in Rust. Here is the first part.

On flash, Ceph is limited by the CPU time its storage daemon spends per operation, not by the drive. We decided to fix that at the root: we are rebuilding the performance-critical parts of Ceph in Rust. Today we publish the first of them as open source: croit FlashStore OSD, a new OSD for NVMe, together with Rust client libraries for RADOS and RBD and an RDMA transport. They work with the Ceph you already run: monitors, managers and classic OSDs stay as they are.

croit FlashStore OSD is available as a preview for testing and evaluation. It is not yet recommended for production data.

The croit Rust stack for Ceph

Four components are in the public repository today.

The components are available as public previews for testing and evaluation. Ceph's monitors and managers stay the classic daemons. The rest of this article goes deep on croit FlashStore OSD, the core of the stack and the part that carries the numbers.

Where the time goes

A current datacenter NVMe drive serves several hundred thousand 4K IOPS. The classic OSD was designed when the drive was the limit, so on flash the ceiling moves to the CPU time spent per operation: shared operation queues with locks and wake-ups between threads, RocksDB and BlueFS under the metadata of every object, network paths with their system calls and copies.

For capacity-first clusters none of this matters much. But a growing share of the flash clusters we deploy and support at croit carry active workloads: virtual machines moved off VMware, databases, ML pipelines that have to keep GPUs fed, HPC scratch space. There, bandwidth and IOPS per core decide whether Ceph fits at all, and the usual answer is more cores, faster cores or a faster network. We wanted an answer that works on the hardware people already own.

What we built

croit FlashStore OSD is a Ceph OSD daemon written from scratch in Rust for NVMe flash, without reusing Ceph's C++ code. It is open source under LGPL-3.0.

What does not change. On the outside it is an ordinary OSD: the same messenger protocol (msgr2), cephx authentication, the same OSDMap and CRUSH rules. Monitors, managers, clients and classic OSDs talk to it without modification, and a single pool can hold classic and croit FlashStore OSDs at the same time. Replicated and erasure-coded pools are supported; RBD and CephFS run on top, and the object classes RGW needs are implemented.

What changes. Inside, almost everything:

• One pinned thread per core. Every placement group belongs to exactly one core, so a PG never waits for a lock. Cores hand work to each other over lock-free single-producer, single-consumer rings.
• Its own storage engine. Each core owns a shard of the drive with its own redo write-ahead log, copy-on-write B+tree, allocator and device queue. The B+tree holds only metadata: data goes to fresh 4K extents, is never overwritten in place and is written in parallel with its log record. By default there is no RocksDB, no BlueFS and no LSM compaction.
• Short I/O paths. NVMe access runs through each core's own io_uring ring, or through SPDK in user space, with one queue pair per core, direct I/O and 2 MiB hugepages: the OSD leaves the host page cache alone, so the memory it takes is its own process memory. Every core runs its own network connections, kernel TCP over io_uring; RDMA is optional, per client.
• One copy of the payload. A write is copied once, on network receive, and with RDMA not even then. From there it travels by reference to the drive and to every replica.
• Durable before acknowledged. A client is acknowledged only once data, metadata and log entry are on media on every replica. croit FlashStore OSD therefore requires drives with power-loss protection.

A deterministic multi-OSD simulation ships with the code and checks every message against a model of classic Ceph peers, so mixed-cluster behavior can be verified without a cluster.

How we measured

We compare against BlueStore, the production default of Ceph today, on the same cluster.

• Cluster: 30 NVMe OSDs on five AMD EPYC 9455P servers, 6 dedicated cores per OSD, 2 × 200 Gbit/s, kernel TCP, no RDMA.
• Baseline: stock Ceph 21.2.0 with BlueStore.
• Clients: unmodified. fio with the rados engine on stock librados. Every cluster figure is measured on RADOS.
• Tests: a replicated pool (size 3) and an erasure-coded pool (2+2), each with 10 benchmark tests at 48 clients × QD16: 64K, 4K and mixed. On top, 8 replicated tests at queue depth 1, 28 tests in all. 4 MiB objects ran on 5 and 10 OSDs, key/value metadata (OMAP) on one OSD and on 30 OSDs.
• Averages: geometric means of the per-test ratio. "Up to" is the best single test.
• Not measured: RGW and CephFS end to end.
• Cold start: every croit FlashStore OSD test started cold, after an OSD restart with the page cache dropped. The BlueStore reference tests did not. If anything, this favors BlueStore.
• Runs: each test ran once per engine.

The RDMA transport is part of today's preview. All cluster figures in this article are over kernel TCP; RDMA figures from our lab are shown separately below.

Results

Preview figures, subject to change: croit FlashStore OSD is still being optimized, so newer builds will measure differently, and we expect them to improve. Every figure in this article comes from the current benchmark report; we will publish updated results with their date and test conditions.

Throughput rises most on small I/O, the tests where the classic OSD spends the most CPU per operation: +123% for 4K sequential reads on the replicated pool, +187% for 4K random writes on the erasure-coded pool. Tail latency falls furthest for 4K random reads on the replicated pool, where the p99 drops by 94%.

croit FlashStore OSD reached or exceeded BlueStore throughput in 27 of 28 measured tests. The exception, 64K random reads on the replicated pool, is on par at 0.99x, and the former weak spot, 4K sequential writes on the erasure-coded pool, is now at 1.54x. p99 latency was lower in 25 of 28 tests, and all 20 tests at queue depth 16 used less OSD CPU per operation. On hyperconverged nodes that CPU goes back to the virtual machines; on dedicated storage nodes it is headroom, or fewer cores in the next purchase.

Queue depth 1 and large objects

At queue depth 1, replicated IOPS rose by 9% on average over 8 tests, up to 18%. With 4 MiB objects, writes gained 14% on 5 OSDs and 28% on 10 OSDs, reads 1% and 4%. On 30 OSDs the 4 MiB tests were limited by the client network, so they say nothing about the OSD.

Key/value metadata

The S3 bucket index and CephFS metadata live in OMAP, the key/value side of an object. On 30 OSDs croit FlashStore OSD was 22% faster on average, with 19 of 26 tests at or above BlueStore, and 23% faster on a single OSD. Scans were 39% faster. The lowest tests reached 0.70x.

Memory

BlueStore reads its RocksDB metadata through the kernel page cache, which grows with the data set: up to 26 GiB per OSD in the erasure-coded 4K tests. Because croit FlashStore OSD uses direct I/O, the memory an OSD occupies is its process memory, and you can plan with it: a median of 2.2 to 4.8 GiB per OSD in every test. That process memory is about 13% below BlueStore's. For sizing a hyperconverged node, that predictability may matter more than the throughput figures.

Capacity

The same data takes the same space: 18.93 TiB raw used with croit FlashStore OSD against 19.09 TiB with BlueStore.

The client libraries and RDMA

Measured with a single client against stock librados and librbd on the same cluster, librustyrbd writes 4M blocks 2.1 times as fast and reads them 1.9 times as fast, and librustyrados reads 4M objects 1.7 times as fast. RBD 4K random writes at queue depth 1 gain 56%, and 4K reads of one object at queue depth 32 run 1.6 times as fast. The Rust libraries were faster than or equal to stock in all 18 measured tests; 4K random writes over 256 objects are at parity, because there the OSD cluster sets the limit for both. Applications, QEMU/KVM and the Ceph tools pick them up without being rebuilt, and the stock libraries stay installed, so you can switch back at any time.

In the croit lab, on separate setups, the RDMA transport delivered 46% more 4 MiB write throughput than kernel TCP at 44% less client CPU, and KVM over RDMA 94% more read bandwidth (1 MiB, QD4).

Where the gain is, and where it is not

The gain is in the OSD. The cluster results measure the OSD data path through RADOS. A workload whose time is spent in RGW or in the MDS will not move because the OSD underneath it got faster.

NVMe only. croit FlashStore OSD is designed for flash with power-loss protection. There is no HDD path and none is planned. On spinning disks the drive remains the limit, which is where the classic OSD already belongs.

No cluster gets faster on its own. The figures above require croit FlashStore OSDs in the pool that serves the workload. An untouched cluster performs exactly as it did yesterday.

Where it does not win yet. In this benchmark:

• 64K random reads on a replicated pool: on par at 0.99x.
• p99 latency: not lower in 3 of 28 tests.
• Some OMAP tests: down to 0.70x.

Trying it

croit FlashStore OSD runs next to the classic OSDs of a cluster that already runs Ceph, on the hardware it already has. The on-disk format differs, so there is no in-place conversion: a drive is redeployed as a croit FlashStore OSD, one drive or one whole server at a time depending on your failure domains, and Ceph backfills it from the other copies exactly as it does when a disk is replaced, on replicated and erasure-coded pools alike. Classic and croit FlashStore OSDs serve the same pool throughout, so you can compare both side by side and stop after any step. As a preview, it belongs on a test system for now. croit customers get supervised testing: you tell our engineers what to test, and they run it live in front of you.

Requirements: Ceph 21.2.0, 20.2.5 or 20.2.4 (the benchmark ran on 21.2.0), NVMe drives with power-loss protection in 4 KiB sector format (512 B namespaces must be reformatted), and Linux 5.14 or newer (RHEL or Alma 9) with io_uring enabled.

In the croit frontier release you enable croit FlashStore OSD in the croit management software and choose it per drive, at no extra cost.

Who builds it

croit FlashStore OSD and the croit Rust stack are built by croit's Ceph engineers Igor Fedotov, BlueStore developer, and Md Mahamudur Rahaman Sajib, both regular contributors to the Ceph project, supported by Martin Verges, CEO of croit. The team uses AI as a development tool to turn deep Ceph expertise into working code quickly. Design, review and testing stay with the engineers, and every change is benchmarked against BlueStore.

croit is a founding member of the Ceph Foundation, and croit engineers contribute to the Ceph project; croit's Igor Fedotov is a member of the Ceph Steering Committee. The croit Rust stack is built to work with Ceph: monitors, managers and classic OSDs keep running unchanged. croit will offer its work to the upstream Ceph project. How and when it is merged is up to the community.

What comes next

• Available today as public preview: croit FlashStore OSD, librustyrados, librustyrbd, RDMA transport (RoCEv2) between clients and OSDs.
• Fourth quarter of 2026, public preview: deduplication for replicated and erasure-coded pools.
• First quarter of 2027: general availability of croit FlashStore OSD, the Rust client libraries, the RDMA transport and deduplication; public previews of a KVBM/NIXL plugin for GPUDirect RDMA into GPU memory, a new transaction-based file system compatible with libcephfs with its Rust client library, and a Rust implementation of the RADOS Gateway (S3).
• Second quarter of 2027: general availability of the KVBM/NIXL plugin, the new file system and the Rust RADOS Gateway.

All dates are indicative and subject to change based on croit customer demand. General availability of croit FlashStore OSD is planned for the first quarter of 2027. croit will release it when its Ceph engineers are confident it is ready for production data, not by a fixed date.

Try it, review it, talk to us

The code is at github.com/croit/ceph-rust, together with the deployment guide (croit FlashStore OSD under LGPL-3.0, the client libraries under MPL-2.0). The benchmark report (PDF) has every test with its measured values; the raw fio data of every test is available on request. Please review them, try the preview on a test system and tell us what you find. Questions, setups that behave differently, ideas: send them to the team with the Questions and feedback button below. croit customers get croit FlashStore OSD at no extra cost in the croit frontier release.

Downloads

• Benchmark report (PDF): every test with its measured values, the test conditions and the method.
• Fact sheet (PDF): the key figures, requirements and roadmap on five pages.
• Press Q&A (PDF): background and answers for editors.
• Raw data: on request, from the downloads on the croit FlashStore OSD page; we email you the link.