Reading the Dashboard

CDM reads many clusters, and at any moment one of them may be unreachable or may not support a feature. Rather than quietly folding that into a total, CDM keeps missing data visible. This page explains what you are looking at when something is incomplete.

A dash is not a zero

A dash means the value is unknown. Zero means CDM read the value and it was zero. A cluster that cannot be reached is left out of a total rather than counted as zero.

Totals say what they cover

Inventory totals, service counts and combined charts show how many clusters contributed when some data could not be read. Fleet capacity goes further: it includes only clusters reporting both used and total space, and shows over N of M clusters under the number, so an incomplete reading cannot produce a misleading fleet percentage. Hover it to see how many clusters were unreachable, lacked the data, or have not been checked yet.

The Servers, OSDs, Disks and Services tables list the rows with a problem first, for example a server that is not running or an OSD that is down or out, and mark each with a dot: point at it to read what is wrong. The boxes above each table count its rows, for example the OSDs that are up, down and out; click one to show only the rows it counts, and click it again to show them all. The first box counts every row. Pick a cluster in the page heading to narrow a table to it. The fleet-wide tables behave the same way as the totals. Disks, Pools, HA groups, OSDs, Servers and Services identify the clusters they could not read while keeping the available rows visible. An unreadable inventory is not an empty inventory, and unreadable disk health is not a count of zero failing disks.

Unsupported is not missing

A feature a cluster does not have is reported separately from data CDM failed to fetch. A plain Ceph cluster is named in the Disks view as not supported and left out of the disk counts, rather than adding zero disks to them. Charts work the same way: the Ceph API serves no time series, so there is nothing to draw.

Reachable and healthy are different measurements

A reachable cluster's row on the Clusters page says Connected even when its health is Unknown. Connectivity and health are measured separately, and one does not stand in for the other.

Findings and tasks

Advisor lists its findings under a heading per severity, the most severe first. Pick a cluster in the page heading to see only its findings; muted findings show in a section of their own with Show Muted. Advisor and Tasks distinguish an empty result from unavailable data. Partial results name the clusters they left out. When the finding count is incomplete, the Advisor entry in the navigation turns amber, and hovering it shows N+ open advisories. A recommendations request that failed does not mean the fleet is healthy.

Ceph health checks appear in Advisor, and on a cluster's own page behind its Ceph Health box, as croit shows them: click the box to open the checks, errors first, each opening the lines of ceph health detail where the cluster reports them. The croit-version comparison leaves out Ceph and demo clusters.

Select a task in Tasks and choose Output to read its log. A running task's log keeps growing while it is open. Plain Ceph clusters keep no task logs.

Audit

Audit identifies the clusters whose entries could not be read, including the case where the configured account lacks audit permissions on that cluster. CDM's own entries and the results from available clusters stay visible. Audit is visible to admins only. Being a CDM admin grants no extra permissions on a connected cluster.

Charts

Single-cluster and per-OSD charts distinguish a failed read from a chart that simply has no history yet. A chart that failed explains itself independently, so the others on the page stay usable. A server without samples does not hide the other servers' history in a combined chart. A history containing only missing measurements shows No data, while measured zeros stay visible. Chart colours follow your system theme without a reload.

With no cluster selected, latency, server load and per-OSD charts draw one line per cluster, server or OSD, named by its cluster. A chart with more than 12 lines draws 12: the 4 with the highest peak, the 4 around the median and the 4 with the lowest, each tier in its own colour, with the number of lines in the header, for example of 3000. Use the table button of a chart for the minimum, mean, median, maximum and last value of every line, and tick a line there to draw it as well. Select a cluster to see all of its lines.

The Latency chart on the overview cannot add up percentiles of different clusters, and their average would hide a cluster that spiked. It draws each cluster's 99th percentile (over its OSDs) and shows the slowest cluster and the median cluster at every moment. Below the chart, Slowest now names the cluster with the highest value and links to it; once that value is at least twice the cluster's own median over the range, it also says by how much, for example 2.4x its usual. Clusters differ by hardware, so a value counts as slow for a cluster only against that cluster's usual.

Capacity thresholds

The Capacity page answers where space runs out now, where it will soon, and how usage grew. Ceph fills each device class on its own, so CDM rates each cluster by its worst device class: Full and Nearfull count the clusters whose most filled OSD has reached the cluster's full or nearfull ratio, naming them. Next to fill names the cluster forecast to turn nearfull first and when; click it to open that cluster's capacity. The counts go by the most filled OSD, not the average usage, because a single full OSD is enough for Ceph to stop accepting writes. The table lists each cluster with its fill, its fullest OSD and when it turns nearfull and full. Click a count to list and chart only the clusters it counts; click it again to show all. Fill by cluster draws every cluster's used space as a share of its capacity today, so clusters of any size compare on one chart, and a capacity added later shows as a lower fill before it; Used by cluster draws the same in bytes. With a cluster picked in the page heading, the boxes show its used capacity, its fullest OSD with the ratios it is measured against, and when it turns nearfull and full; the table lists its device classes, and the charts draw each class beside its total, in bytes with its capacity as a dashed line in the same colour, and as a share of it. A forecast takes the class's growth over the last 30 days and projects it onto the class's fullest OSD, assuming the OSDs stay as unevenly filled as they are now; a rebalance or new OSDs change the forecast. A forecast needs at least 7 days of history and says Collecting data until then. For a croit cluster CDM reads the last month of its usage, and of every OSD's fill, from the cluster's metrics when you add it, so its forecasts appear at once. A threshold the fullest OSD has already reached is shown as now; a forecast further out reads in days, months past two months, years past two years, and > 10 years past ten. Choose a range above the charts, or drag across one to zoom in, and reset the zoom with the arrow beside the range. A cluster's own page charts its used space, each device class beside the total. If a threshold cannot be read, its forecast stays unknown, the table says Threshold unavailable, and the cluster is in none of the counts. CDM does not read the thresholds of plain Ceph clusters, so they always show Threshold unavailable.

S3 multisite replication

Open Replication, then S3 multisite, to see every multisite realm of your croit clusters. Each realm is a card titled with its realm and zone group (a cluster on an older croit version reports only the realm's id): its primary site on the left, the secondary sites it replicates to on the right, or below it on narrower screens. Every site names its zone. Click a realm's title to fold it away. Clusters that take part in no realm are not listed.

Every secondary reports Metadata and Data sync separately, and its Access: read-write, or read-only if it only pulls from the primary. A primary shows how many endpoints other sites reach it at, and its Incoming data sync only if another site writes to it.
The number of shards behind counts replication log shards with pending changes, not objects or bytes. The i button on a site opens its details: every sync state with its Oldest pending change while behind or its Last confirmed time once in sync, data sync per source zone as shards caught up out of the total, and the site's endpoints. All times are UTC.
A site CDM has not read for five minutes says No report for how long, and its last sync states turn grey, since nobody has confirmed them since.
A sync state the cluster did not report shows Unavailable, and a role it did not report shows Role unknown. S3 multisite is a licensed croit feature: a cluster without a valid licence, and a plain Ceph cluster, is named with the reason "S3 multisite management needs a licensed croit cluster", and a croit cluster appears on its own within minutes of getting a licence. When a cluster's status cannot be read for another reason, the page names it; check that cluster's connection and its credential.
The tiles on top show how many realms and sites there are, how many secondaries are in sync (and what the others are doing), the oldest change still waiting to replicate, and the site CDM read longest ago. Click In sync to list only the realms with a secondary that is not in sync, and click it again to list all. Click Oldest pending or Least recently checked to jump to that site, which flashes briefly.
The line to each secondary shows how its replication is doing: green while in sync, amber while catching up, grey while only one sync state is reported, and red and dashed when nothing replicates or a site has not been read for five minutes. Pulses travel along the line while data replicates.
Each realm says when CDM last read it. CDM reads every cluster once a minute; Refresh reads them all again at once. This view does not replace real-time failover monitoring.
A setup or a change of primary that CDM is running shows in its realm while it runs, with Show log.
Plain Ceph clusters and clusters CDM cannot reach are listed as not covered, never as disabled replication.

Setting up multisite

Set up multisite starts a new realm, and Add secondary on a realm adds a site to it. Both need a croit cluster registered with Write access (admin), an administrator credential, a valid licence and a croit version that offers the setup; the button and the wizard say which of these a cluster lacks. A plain Ceph cluster cannot take part and says so.

Sites: pick the primary, a cluster that serves S3, and one or more secondaries, clusters that do not run S3 yet.
Primary: only asked for a primary that is not a multisite primary yet. CDM converts it first and creates the sync user replication authenticates with.
One step per secondary: name its zone and check the servers CDM deploys its S3 gateways on, all of the secondary's servers to begin with. Tick Read-only zone if it should only receive data.

The Primary step and each secondary's step name the sync endpoints the other sites will use for that site: the primary's running S3 gateways on their plain HTTP ports, and a secondary's chosen gateway servers on port 80, as croit's own multisite dialogs fill them in. Where the other sites reach the site through NAT, a DNS name or a load balancer, untick Use the derived sync endpoints and edit the list, for example to https://s3.site-b.example.com; CDM checks them and keeps them as entered. While the list still holds only derived addresses, it follows a change of the gateway servers. 4. Review: CDM checks every cluster, for example that they run the same Ceph release and that the zone name is free, and tries the primary's endpoints from the secondary's gateway servers. That check says what those servers see ("answered in 12 ms", "did not answer"), which is what will pull the realm; it does not prove the site is reachable from anywhere else. It fails only when no gateway server reaches the primary, and warns when some do not or when a cluster's croit is too old to try. It lists only the checks that did not pass; Show all checks opens every check and where other sites will reach each site. Start setup is available once no check blocks.

Review starts with what the setup will change, and warns when a primary has to be converted: a cluster cannot leave multisite again once it is a primary.

The setup runs as one task per secondary, one after the other and each taking a few minutes. If one fails, the ones after it are not started. When a secondary's gateways are deployed, CDM tries their real endpoints from the primary's gateway servers, where a writable secondary's changes are pulled back to, and the task log warns if they do not answer. The first task's log opens right away, every running setup shows in its realm with Show log, and every step the clusters run shows in Tasks, where Output opens a task's log.

Changing the primary

Any secondary can become its realm's primary: open its details with i and choose Promote to primary. CDM says what the promotion does before anything runs:

Switch over: the primary still answers. The new primary first pulls the realm from it, and the old primary continues as a secondary. Use it for planned maintenance.
Fail over: the primary has stopped reporting. A realm whose primary has not reported for five minutes offers Fail over directly and lets you pick the secondary that takes over; pick one that is in sync. The old primary still names itself primary when it returns.
Fail back: a primary that returns after a failover shows as Claims primary in its realm, since it still has the realm from before. Fail back to it pulls the current realm and makes it the primary again; the site that took over continues as a secondary.

A site that is behind loses the writes it has not received yet when it takes over. CDM then asks first, and the confirm button reads Promote anyway. If the site fell behind after CDM last read it, the cluster itself refuses and the task says so; start the promotion again to confirm.

The promotion restarts the new primary's S3 gateways and runs as a task whose log opens right away. When it ends, CDM reads the realm again, and its cards move to their new places. Promoting needs the same administrator credential as the setup, and only one setup or promotion runs per realm at a time.

Refreshing

Live views and navigation counts update on their own each time CDM reads the clusters again, so you do not need to reload the page. A change on a cluster can take a little while to appear.

Logs

Select servers, OSDs, services or HA groups in their table and choose Logs. The logs page opens filtered to your selection, and the filter bar names each part of it: the cluster, the host and the service. Add or remove conditions there, for example a priority or a text in the message, the same way as in croit.

To read one cluster's log, pick the cluster in the page heading, as on the other pages under Servers. The viewer then shows only that cluster's lines and drops the Cluster column.

The viewer has two modes:

A time range. The chart above the lines shows how the matches are spread over the range, and the viewer shows the first 1,000 lines from its start. Narrow the range to see later lines. One range covers at most 24 hours.
Tail follows new lines from every selected cluster as they are written.

When a cluster cannot be read, the viewer says so once and the other clusters keep delivering lines. A followed log catches up on what that cluster wrote meanwhile as soon as it answers again.

Plain Ceph clusters keep one log for the whole cluster and only its latest entries. The viewer shows all of their lines: a selection, a priority or a text condition does not narrow them, and a past time range is not available for them; use Tail instead.

On narrow screens the navigation collapses to a column of icons, and it scrolls when it is taller than the window, so its lower entries and Logout stay reachable.

On this Page: