Rewriting croit's backend from Kotlin to Go

croit's backend runs on the management node of a Ceph cluster: it boots the cluster's servers, configures Ceph and its gateways, and serves the UI and the API. Until recently it was a Kotlin application on the JVM. Since v2609.0 it is a single Go binary, and the Kotlin code is gone.

In this post we'd like to talk about why we wanted to make the switch, how we ran the old and the new code side by side while the port was underway, and what improvements we got out of it. In short:

• Shorter startup: croit is up and serving much sooner after a start or a restart.
• Lower CPU and memory usage: the management node has noticeably more room left over when croit is idle.
• Smaller footprint: the Java runtime and over two hundred libraries were replaced by one binary, so there is less to ship and less to patch.
• Better tests and design: we used the port to increase test coverage and to rework parts of the application, such as how croit talks to Ceph.

Why we switched

One language for the whole system: At croit we write more and more of our software in Go: croitd, the agent that runs on every server croit manages, was already Go, and so is a growing part of our internal infrastructure. The management backend was the one large Kotlin codebase left. Moving it to Go means one toolchain, one set of conventions and one language for everyone on the team. It also allows us to more easily share code between components in the croit app.

Simpler tooling: The Kotlin backend was built with Gradle, and updating Kotlin, the Java stack and the dependencies was much more cumbersome than in our Go projects. Gradle and the Java build ecosystem are complex on their own. You actually have to spend time learning them before you can work with them comfortably. Go's toolchain is much simpler to use and set up.

Faster iteration: Both the build time and the startup time of the Kotlin application were high. The time from a developer changing code to having that change running and testable was too long for us. Our goal was to reduce this dev iteration time by switching to Go.

Making code easier for AI: This is just a theory we have, not something we can prove with data. We expect AI tools to work better with our code when all of it is written in one language rather than split between Kotlin and Go. We also suspect Go code is easier for an AI to work with: Go tends to be explicit and verbose, while Kotlin relies more on implicit knowledge. The idea being that LLMs work better when things are spelled out rather than requiring latent knowledge to understand the code (same applies to new developers as well, fast onboarding is one of the selling points of Go after all). But as mentioned before, we have no data to back this up, it's just speculation.

How we did it

To make the transition easier and not disrupt the work of other developers on the croit application, during the port the Go backend ran in parallel next to the Kotlin backend, with the Kotlin backend forwarding requests to the Go backend for anything that has already been ported to Go. We ported one domain at a time from Kotlin to Go, without having to stop development of other features for the croit app. This allowed us to release croit version v2608 mid-port alongside a long list of new features, some already developed in the Go code, some still in Kotlin. Once everything was ported, we removed Kotlin and switched entirely to the Go backend in croit version v2609.

We also took this opportunity to extend our unit tests as we ported code to Go. Our test coverage increased considerably over the course of the port. We also redesigned parts of our backend, eliminating some tech debt that accumulated over the years, and got some performance improvements out of it.

Some numbers and graphs

Let's look at some numbers to get an idea of whether or not we achieved our initial goals with the rewrite. Please do note that a lot of the results of course depend on the hardware and cluster size, we just took some measurements in a development environment and they won't always reflect what you might see in the real world.

Performance

The Kotlin backend relied heavily on the ceph command-line tools to communicate with the ceph cluster. This did waste a bit of CPU power, as it ended up starting many processes for the many command-line tool calls. These tools also spend quite a bit of time comparing the command against all known commands to later display better help texts for unknown commands, something we don't actually need for our use case. With the Go backend we switched to a single connection to the cluster via librados for the most frequently called tools. In our dev setup we measured a tenth of the CPU usage for an empty and idle cluster (again, it's a contrived setup, but it shows a clear improvement). Unrelated to that, the backend process also holds about a quarter of the memory, which we attribute to Go binaries simply having a smaller footprint compared to the entire JVM.

Since the croit app basically acts as a webserver, we also wanted to look into HTTP and JSON performance. We ran a few tests by calling the croit API and measuring response times. Overall we found that the Go backend is roughly 1.5 to 2.5 times faster when it comes to sending a JSON response. But we did also notice that Go took longer when it came to decoding larger JSON request bodies compared to Kotlin. We think it's due to extra verification of the API schema that our Go backend does, which was simply missing in Kotlin. In practice this shouldn't matter much, POST endpoints are much less frequently called than GET endpoints.

We collected some test data for various endpoints. Some endpoints (like fetching the CRUSH map) are considerably faster with the Go backend, that's mostly thanks to the cluster connection we talked about before. Some endpoints ended up a little slower on Go, we think it's because of some changes in our authentication logic.

Dependencies and application size

The Kotlin backend shipped with 221 third-party libraries; the Go backend ships only 43 modules in comparison. Every dependency is code we did not write but are responsible for. Fewer dependencies mean fewer CVE advisories to assess, fewer forced updates, and a smaller surface to keep track of. The tooling for this is also simpler for Go, and since other components were already written in Go and were already using such tooling, we didn't have to relearn anything.

The application itself got smaller as well. The Kotlin backend with Java runtime and the libraries added up to about 300 MiB in all, the Go binary replacing it on the other hand is only about 90 MiB in size. Thanks to this the compressed croit docker image got 19% smaller.

Development iteration

Reducing development iteration is crucial to speeding up development. The development iteration time refers to the time it takes from a developer making a change to the code, to being able to test that change on a running system. We were unhappy with the startup times of the Kotlin backend, they made dev iteration slow and frustrating, especially when only changing code in other parts of the application.

The Go backend has a much faster startup compared to Kotlin. In our dev setup we measure about 80% less time from start until the backend is up and serving the API:

Compilation also became faster, but not always. A clean build (i.e. without build caches) compiles a lot faster in Go, but with caches it depends on where code changes have been made. We noticed an edit in code that gets imported and used in many other places can end up in longer compile times for Go than for Kotlin. We think it's because Kotlin only has to recompile the edited file to a Java class file (everything else happens at runtime in the JVM), whereas Go has to recompile entire packages.

But for dev iteration this didn't end up mattering. Because of the faster startup times, we still have faster iteration times with Go, regardless of compile times. We measured up to 30% less time from making code changes to the croit app being up and running for the Go backend, depending on where the code change was made.

Lines of code

One thing that got "worse" is the pure number of lines of code. We mentioned before already that Go is more verbose. Overall, we ended up with about 133% more lines of code (excluding comments and the like) in the backend after switching to Go. It's up to interpretation if that's really a bad thing, the case of AI might have it be an advantage, but more text also means more token usage, so who knows.

What this means for people running croit

For a croit operator, the rewrite is mostly noticeable by its absence: a management node that is quieter at rest and needs less memory, a web interface that is back within seconds after a restart, and a smaller set of dependencies for a more secure platform.

For us it means one language across our software, a codebase that is easier to understand and reason with AI with, and much faster and less frustrating dev iterations for new features being developed.