Part of the croit Rust stack for Ceph
croit FlashStore OSD:
a Ceph OSD rebuilt in Rust for NVMe
- +50%* Throughput Replicated pools, average of 10 tests, 30 NVMe OSDs, kernel TCP
- −52%* p99 latency Replicated pools, 48 clients × QD16, 30 NVMe OSDs, kernel TCP
- −36%* CPU per operation Replicated pools, less in all 20 tests, 30 NVMe OSDs, kernel TCP
*Based on the benchmark of 9 November 2026, build 422be26.
On flash, Ceph waits for the CPU
A datacenter NVMe drive serves several hundred thousand 4K IOPS. The classic OSD was designed when the drive was the limit. Now the ceiling is CPU time per operation, and a faster drive no longer helps.
- 01
Built for active workloads
Virtual machines moved off VMware, databases, ML pipelines that keep GPUs fed, HPC scratch. Here, bandwidth and IOPS per core decide whether Ceph fits.
- 02
Same drives, servers and network
The gain comes from the OSD data path. On hyperconverged nodes the freed CPU and memory go back to the VMs; on storage nodes they are headroom, or fewer cores in the next purchase.
- 03
Open source, with or without croit
croit FlashStore OSD is licensed under LGPL-3.0, the Rust client libraries under MPL-2.0. There is no license fee and no capacity-based pricing. It runs in existing Ceph clusters with or without croit.
Same Ceph outside, new data path inside
Both sides show the path one write takes through an OSD. BlueStore sends every op through shared queues and locks; its data goes straight to the drive, its metadata through RocksDB and BlueFS. croit FlashStore OSD gives each CPU core its own lane, from the network to the drive, with no RocksDB in the metadata path.
1 · msgr2 · cephx
Ordinary on the outside
Monitors, managers, unmodified clients and classic OSDs talk to it as they always do, under the same OSDMap and CRUSH rules. One pool can hold both kinds of OSD. Inside, every core runs its own connections and hands each op to the core that owns its PG over lock-free rings.
2 · Threads
One core per placement group
Every PG belongs to exactly one pinned core, so it never waits for a lock. Core 0 also does the control work: monitors, managers, heartbeats and maps.
3 · Storage engine
A B+tree and log per core
Each core owns a shard of the drive with its own write-ahead log, copy-on-write B+tree, allocator and device queue. The B+tree holds only metadata; data goes to fresh 4K extents, never overwritten in place, written in parallel with its log record. By default there is no RocksDB, no BlueFS and no compaction.
4 · I/O path
No copies in the store
A write is copied once, on network receive, and with RDMA not even then. The store writes it by reference from the receive buffer through the core's own io_uring or SPDK ring. It is acknowledged only when it is on media everywhere.
“BlueStore runs a very large number of production clusters, and it has to stay stable and compatible for all of them. croit FlashStore OSD gave us the freedom to design a storage engine for NVMe from scratch, next to BlueStore rather than instead of it. What we learn, we will offer back to the Ceph project.”
Igor Fedotov, Ceph engineer at croit and member of the Ceph Steering Committee
What we measured
The same hardware, a faster storage engine
We ran the same benchmark tests on the same cluster of 30 NVMe OSDs, once with stock Ceph 21.2.0 BlueStore and once with croit FlashStore OSD, over kernel TCP with unmodified Ceph clients. Throughput rose by 50% on average on replicated pools and by 54% on erasure-coded pools, up to 187% for 4K random writes on erasure-coded pools. On replicated pools, p99 latency fell by 52% and OSD CPU per operation by 36% on average.
Predictable memory, no page cache
BlueStore reads its RocksDB metadata through the kernel page cache, which grows with the data set: up to 26 GiB per OSD in our erasure-coded 4K tests. croit FlashStore OSD uses direct I/O and leaves the page cache alone. Its OSDs stayed at a median of 2.2 to 4.8 GiB each in every test, and its process memory is about 13% below BlueStore's. For sizing a hyperconverged node, that predictability may matter more than the throughput figures.
| Change against BlueStore | Replicated, size 3 | Erasure coded 2+2 |
|---|---|---|
| Throughput | +50%, up to +123% | +54%, up to +187% |
| p99 latency | −52%, up to −94% | −57% |
| OSD CPU per operation | −36%, up to −59% | −37% |
48 clients at queue depth 16, 10 benchmark tests per pool type: 64K, 4K and mixed. The best single results are 4K sequential reads on the replicated pool and 4K random writes on the erasure-coded pool. croit FlashStore OSD reached or exceeded BlueStore throughput in 27 of 28 measured tests; the exception, 64K random reads on the replicated pool, is on par at 0.99x, and the former weak spot, 4K sequential writes on the erasure-coded pool, is now at 1.54x. p99 latency was lower in 25 of 28, and all 20 tests at queue depth 16 used less CPU per operation.
At queue depth 1, replicated IOPS rose by 9% on average over 8 tests, up to 18%. With 4 MiB objects, writes gained 14% on 5 OSDs and 28% on 10 OSDs, reads 1% and 4%; on 30 OSDs the 4 MiB tests were limited by the client network. Key/value metadata (OMAP), as used by the S3 bucket index and CephFS, was 22% faster on average on 30 OSDs, with 19 of 26 tests at or above BlueStore, and scans 39% faster; the lowest tests reached 0.70x. The same data takes the same space: 18.93 TiB raw used against 19.09 TiB with BlueStore.
Preview figures, subject to change: croit FlashStore OSD is still being optimized, so newer builds will measure differently, and we expect them to improve. We will publish updated results with their date and test conditions.
Benchmark report, fact sheet and press Q&A
The full benchmark report with every test, the fact sheet and the press Q&A for editors. The raw data comes by email, on request.
Is it right for your cluster?
croit FlashStore OSD is built for NVMe pools where the CPU is the bottleneck: virtual machines, databases, AI and HPC. It speeds up the OSD, so S3 and CephFS work that waits on the gateway or the metadata server stays as it is, large reads limited by the network stay level, and HDD pools stay on the classic OSD. As a preview, it belongs on a test cluster for now.
Try it next to your classic OSDs
- 1
Get it
Code and deployment guide are on GitHub. In the croit frontier release you enable it in the croit management software. croit customers get supervised testing: tell our engineers what to test, and they run it live in front of you.
- 2
Redeploy drives
The on-disk format differs, so there is no in-place conversion: wipe a drive and bring it back as a croit FlashStore OSD, one drive or one whole server at a time, depending on your failure domains. With croit you choose it per drive.
- 3
Let it backfill
The cluster refills it from the other copies, exactly as when a disk is replaced.
- 4
Compare
Classic and croit FlashStore OSDs serve the same pool throughout, so you can measure both side by side and stop after any step.
Part of the croit Rust stack for Ceph
croit is rebuilding the performance-critical parts of Ceph in Rust, and croit FlashStore OSD is the first part. Ceph's monitors and managers keep running unchanged. Read the announcement.
- Today
Public preview
croit FlashStore OSD; librustyrados, a drop-in librados with a native Rust API; librustyrbd, a drop-in librbd for QEMU/KVM; an RDMA transport (RoCEv2) between clients and OSDs.
- Q4 2026
Next preview
Deduplication for replicated and erasure-coded pools.
- Q1 2027
General availability and new previews
General availability of croit FlashStore OSD, the Rust client libraries, the RDMA transport and deduplication. Previews of a KVBM/NIXL plugin for GPUDirect RDMA into GPU memory, a new transaction-based file system compatible with libcephfs with its Rust client library, and a Rust RADOS Gateway (S3).
- Q2 2027
General availability
General availability of the KVBM/NIXL plugin, the new file system and the Rust RADOS Gateway.
All dates are indicative and subject to change based on croit customer demand.
Built by croit's Ceph engineers
croit FlashStore OSD is built by Igor Fedotov and Md Mahamudur Rahaman Sajib, supported by Martin Verges, CEO of croit. For croit customers it is included at no extra cost in the croit frontier release and enabled in the croit management software.
FAQs
Not yet. croit FlashStore OSD is available as a preview for testing and evaluation. It is not yet recommended for production data.
General availability is planned for the first quarter of 2027. croit will release it when its Ceph engineers are confident it is ready for production data, not by a fixed date.
Nothing. It is open source under LGPL-3.0, with no license fee and no capacity-based pricing. For croit customers it is included at no extra cost in the croit frontier release.
No. Code and deployment guide are on GitHub, and it runs in existing Ceph clusters with or without croit. In the croit frontier release you enable it in the croit management software and choose it per drive.
No. Every result on this page was measured with unmodified Ceph clients on stock librados. Monitors, managers and classic OSDs keep running unchanged as well.
No. The RDMA transport (RoCEv2) is part of the preview and optional. Every figure on this page was measured over kernel TCP.
No. NVMe drives with power-loss protection only. On spinning disks the drive is still the limit, which is where the classic OSD belongs.
Nothing changes. Performance stays as it is until you redeploy disks as croit FlashStore OSDs in the pool that serves the workload.
In our benchmark, croit FlashStore OSD reached or exceeded BlueStore throughput in 27 of 28 tests. The exception, 64K random reads on a replicated pool, is on par at 0.99x. p99 latency was lower in 25 of 28 tests, and some key/value metadata (OMAP) tests were slower, the lowest at 0.70x.
croit FlashStore OSD is licensed under LGPL-3.0, the Rust client libraries under MPL-2.0.
