Ceph Mimic

This is our first release with full support for Ceph Mimic. Ceph Mimic was already released three months earlier in June, but we wanted to ensure that you get a stable release. Our goal is to guarantee an absolutely stable operation. Therefore, we generally advise against installing the latest or untested versions of Ceph.

We helped to fix open issues in Ceph we encountered in our testing of earlier Mimic releases, most notably problems creating OSDs in some settings and handling spurious read errors on some kernel versions. After months of testing and debugging, we can now recommend its use.

Upgrading to Ceph Mimic

Upgrading Ceph is easy with croit: there's a button to do that in the Maintenance view to start the upgrade task. That's all you need to do. The croit management software will take care of updating and restarting all daemons in the proper order and performs all necessary checks as described in the Mimic upgrade instructions.

Upgrading via the automated task requires the cluster to be in a healthy state and the task will wait until the cluster returns to full health after each step or daemon restart. This upgrade uses Ceph's high availability capabilities and requires no downtime.

If a failure occurs during upgrade such as an unrelated hardware failure (disk fails or server fails to reboot) that it cannot recover from within a step-specific timeout: do not panic. You can get the cluster back to health as you would usually by removing or replacing the failed hardware. Then run the upgrade task again: it's designed to be idempotent, so it will effectively continue from where it stopped last time. If in doubt: contact our support, we are happy to help.

This upgrade will be performed in-memory first and all servers will be set to boot our new Mimic image on the next boot. Run the Rolling reboot task in the image view to finalize the upgrade and boot all servers with the latest image (optional but recommended).

Redesigned UI and Night Mode

Trouble with Ceph late at night? Hopefully not, but our new night mode got your covered if you need to take a look at your cluster in the middle of the night.

We have also improved the usability of a lot of UI elements like our dropdown menus. Moreover, the setup UI has been completely revamped making it easier than ever to install Ceph with croit.

Cloud Backups for Your Configuration

"But isn't the management server running croit a single point of failure?" Is a fair question we get from time to time. Well, it's not because it is only needed to boot servers; your Ceph cluster continues running and is accessible if our docker container is not available. Losing the configuration stored on the management server would be annoying but recoverable. Unfortunately we cannot store some parts of the configuration within Ceph itself as we need to be able to boot it up after a full power outage. So we implemented the next best thing: cloud configuration backups.

To enable them, register your cluster with us in the Config Backups view under Maintenance. Once enabled, your configuration is backed up every night. All backups are fully encrypted, even we cannot access them without you telling us the encryption key. That means you have to keep your backup key in safe place -- even we cannot help you if you lose the key.

Restoring from a backup is also simple: just deploy our container as usual and select Restore from backup in the setup menu. All you need to provide is the previously created credentials and the encryption key.

New OSD Creation

All newly created OSDs will be based on ceph-volume with LVM, creating OSDs this way is future-proof and improves performance during creation and startup. Don't worry, your old OSDs are still supported and you do not need to change them.

We have also added new features during OSD creation:

Support for full disk encryption
Assigning custom device classes during creation instead
Re-using IDs of previously destroyed OSDs to avoid unnecessary data rebalancing

croit 1805 deprecated FileStore and this release removes the ability to create new FileStore OSDs. BlueStore OSDs offer a better performance, stability, and more features such as erasure coding. Existing FileStore OSDs will continue to be supported indefinitely, you do not have to change any existing OSD.

Notes for Filestore Users

It is recommended to upgrade to BlueStore eventually, destroy FileStore OSDs and re-create them using BlueStore to migrate. Contact our support if you need help with this process.

We have encountered performance problems in the form of high latency in some scenarios during testing with FileStore and the kernel 4.17 that we use for our latest images. If you are affected by this: we also ship an image with kernel 4.9 that you can use instead.

IPV6 Clusters

It was always possible to configure IPv6 networks with croit, but configuring Ceph and some other services required some interventions. Note that Ceph does not support dual stack at the moment, i.e., you cluster is either IPv4 only or IPv6 only and you must chose during setup. Chosing an IPv6 address for the first mon service automatically configures all Ceph services for IPv6. NFS will now always listen on all configured IPv6 addresses and supports IPv6 permissions for shares. HA groups also support IPv6 now.

We currently do not yet support PXE booting via IPv6, please keep the management network used for booting configured as IPv4 for now.

Technology Preview: ARM64 Support

We are now building our whole Debian image for ARM64. This is not yet ready for production, but feel free to deploy this on test clusters. The images are not shown by default, add the following to your /config/config.yml in the croit Docker container to enable them:

Software Release v1809