Task Advisor Warnings

The Task Advisor continuously checks your cluster and management node and raises a recommendation whenever it finds something worth your attention. Each recommendation shown in the croit UI (and available to hook scripts via the OnTaskAdvisorFindingAdded/OnTaskAdvisorFindingRemoved events) corresponds to one of the warning types below. For some warnings the UI also shows one or more buttons that apply a fix automatically, so it is worth checking a recommendation in the UI before fixing it by hand. This page lists every warning type with a general description and its usual causes; the croit UI additionally shows a short, per-cluster summary and, where applicable, links to the affected object. Clicking a recommendation's title in the croit UI opens this page at that warning.

Ceph

CephReleaseDoesNotMatch: Unexpected ceph version

The cluster is running a different Ceph release than the one configured in croit.

Usual causes: the Ceph packages/containers were upgraded (or downgraded) outside of croit, or the configured Ceph version in croit was never updated to match a completed upgrade.

MultipleCephReleasesRunning: Mismatched ceph versions

Some nodes in the cluster are running a different Ceph release than others.

Usual causes: a rolling upgrade is stuck partway through, or a node was reinstalled or manually patched without going through croit's upgrade flow.

ContainerMismatch: Switch to a different container

The cluster is configured to run one Ceph release's container image while a different default image is expected for that release.

Usual causes: croit's bundled container defaults changed after a croit update, and the cluster has not yet been switched over to the new container.

CephEOLWarning: Release nearing/at end of life

The currently configured Ceph release will no longer receive updates.

Usual causes: the cluster has not been upgraded in a while and is still running an older major Ceph release (Luminous through Quincy).

MON

MonCountBelowMin: Too few MONs

The cluster has only one or two MON services.

Usual causes: a small or test cluster was never scaled up, or MON services were removed without replacing them. Croit recommends 3 MONs for smaller and 5 MONs for bigger clusters, so that quorum survives the loss of a single node.

MonCountEven: Even number of MONs

The cluster has an even number of MON services (4 or 6).

Usual causes: a MON was added without considering that Ceph quorum requires an odd count. An even count adds a node without improving how many failures the cluster can tolerate, since quorum still needs more than half the MONs to agree.

MonCountAboveMax: Too many MONs

The cluster has more than 7 MON services.

Usual causes: MONs were added faster than they were removed during cluster growth or migrations. More than 7 MONs adds Paxos round-trip latency to every cluster map change without improving resilience.

OSD

OsdCountIsZero: No OSDs

The cluster has no OSDs, so no data can be stored.

Usual causes: a freshly bootstrapped cluster before disks were assigned, or all OSDs were removed during maintenance and not yet replaced.

OsdIsNearFull: OSD almost full

An OSD's usage ratio is above roughly 70%.

Usual causes: uneven data distribution (CRUSH weighting or PG imbalance), or overall cluster capacity approaching its limit. Left unaddressed this escalates to OsdIsTooFull.

OsdIsTooFull: OSD full

An OSD's usage ratio is above roughly 80%.

Usual causes: same as OsdIsNearFull, left unaddressed. At this level Ceph may start throttling or refusing writes to the pool, so free up space or rebalance promptly.

OsdIsSeverelyFragmented: OSD severely fragmented

A BlueStore OSD on spinning disk has very high (>98%) allocator fragmentation.

Usual causes: long-running OSDs with heavy small-object churn (e.g. RGW workloads) that have never been recreated. Fragmentation degrades read/write performance over time and usually only improves after the OSD is redeployed.

CrushmapHostOutsideRoot: Host outside CRUSH root

A host bucket exists in the CRUSH map but is not reachable from any root bucket.

Usual causes: a host was removed from a rack/root bucket during manual CRUSH map edits, or croit's expected topology and the actual CRUSH map have drifted apart. Data is not placed on OSDs on an unreachable host.

CheckMemoryConfig: OSD memory target too high

The sum of configured OSD memory targets on a server exceeds the server's total RAM.

Usual causes: OSDs were added to a server without re-tuning osd_memory_target downward, risking OOM kills under load.

CRUSH

MetadataNotOnSsd: Metadata pool on HDD

A metadata-heavy pool (e.g. an RGW metadata pool) is mapped by a CRUSH rule that does not target SSD/NVMe (or DB-on-SSD) OSDs, even though such a rule exists in the cluster.

Usual causes: the pool's CRUSH rule was never switched to the fast-media rule after SSDs were added, hurting metadata operation latency.

MinSizeTooLow: Insufficient pool redundancy

A pool's min_size is 1 (or equals k for erasure-coded pools), meaning it can serve writes with zero redundancy margin.

Usual causes: the pool was created with a permissive default, or min_size was lowered temporarily during a recovery event and never restored. A single further OSD failure can then put data at risk.

MinSizeTooHigh: Insufficient pool availability

A pool's min_size equals its size (or k+m for erasure-coded pools), so the pool becomes unavailable as soon as a single OSD goes down.

Usual causes: min_size was set equal to size by mistake, or a resilience setting was copied from a different pool profile.

MDS

CephFsAllowStandByReplay: CephFS standby replay disabled

A CephFS filesystem does not have standby-replay enabled.

Usual causes: the filesystem was created before standby-replay was recommended, or it was disabled to save resources without reviewing the resulting failover-speed trade-off.

CephFsNotEnoughStandby: Too few standby/standby-replay MDS

A filesystem using standby-replay has fewer replay MDS than active MDS, or the cluster has zero MDS in general standby.

Usual causes: MDS count was not scaled up alongside the number of active ranks, or standby MDS were removed during resource trimming. This delays failover if an active MDS goes down.

CephfsGuardDetection: CephFS Guard detected suspicious activity

CephFS Guard saw a pattern of file activity on a CephFS filesystem that resembles ransomware, such as a client rewriting or renaming a large number of files in a short time.

Usual causes: a compromised client encrypting files, or a legitimate bulk operation (a large migration, a backup restore, or a mass re-encryption) that happens to look the same. Check which client caused it before assuming an attack.

RBD Mirror

RbdMirrorPeerMissing: RBD Mirror peer missing

RBD mirroring is enabled on a pool, but no mirror peer is registered.

Usual causes: mirroring was turned on before the peer cluster/bootstrap token was configured, or a peer was removed without disabling mirroring.

RbdMirrorDaemonMissing: RBD Mirror daemon missing

Pools expect to receive mirrored data, but no rbd-mirror service is running.

Usual causes: mirroring was configured on the receiving side but the rbd-mirror daemon was never deployed, or it was stopped/removed.

iSCSI

IscsiLegacyCapacity: Data corruption issue affects your iSCSI disks

A kernel-backstore iSCSI image does not explicitly disable emulate_legacy_capacity, exposing it to a known last-512-byte-sector data-loss bug.

Usual causes: the image was created with an older croit version before this setting was defaulted to off. See the linked blog post in the UI for remediation steps.

IscsiInsufficientGateways: Only one iSCSI gateway configured

An iSCSI service has fewer than two gateway servers assigned.

Usual causes: the service was set up for testing with a single gateway and never extended for production HA. A single gateway is a single point of failure for that service's initiators.

NVMe-oF

NvmeOFConfigErrors: NVMe-oF configuration errors

One or more NVMe-oF gateway services failed to apply their configuration.

Usual causes: a gateway host is unreachable, a subsystem/namespace configuration conflicts with existing state, or the NVMe-oF service crashed mid-reconfiguration.

OS

HostOsUpdateAvailable: OS update available

A newer default OS image exists than the one currently marked default for the architecture (x86/ARM).

Usual causes: croit shipped a new OS image (e.g. with security fixes) since the cluster's default was last set.

RunningImageMismatch: Server reboot needed

One or more servers are running a different OS image than currently configured.

Usual causes: the configured image was changed after the server last booted. A reboot/PXE cycle is needed to apply it.

HostOsImageDeprecated: Outdated OS running

A server is running an OS image that has been marked deprecated.

Usual causes: the cluster has not been rebooted/updated since a newer image superseded the deprecated one.

DuplicateNicNames: Duplicate NIC names

croit has two or more entries for one OS interface name on a server, so it cannot tell which of them describes the interface that is really there.

Usual causes: a network card was replaced, so croit still has the old card's entry under the same name. "Remove duplicate NICs" resolves that: it asks the server for a fresh hardware report and then keeps a single entry, carrying over any bond, VLAN, or service configuration attached to it. A group is left for manual review when more than one of its entries carries such configuration, because croit cannot choose which one survives.

Software interfaces croit does not manage raise the same finding as a warning rather than a critical problem, because nothing is at risk: the entries are stale bookkeeping. The kernel gives an interface such as a dummy a new MAC address every time it is created, and croit identifies an interface by its MAC, so each recreate adds another entry. Pin the MAC in the hook script that creates the interface to stop that, as shown in Extra network interfaces. croit clears the leftover entries itself as soon as the server reports its hardware again.

Backup

BackupNotConfigured: No backup configured

No management-node backup (cloud or MON-disk) is configured for an installed cluster.

Usual causes: backup setup was skipped during initial installation.

MonBackupFailed: Database backup to a MON disk failed

A scheduled backup of the croit database to a MON disk failed.

Usual causes: the target MON disk ran out of space, became unreachable, or has permission/mount issues.

CloudBackupFailed: Cloud backup has failed

A scheduled cloud backup job failed.

Usual causes: invalid or expired cloud credentials, network connectivity issues to the backup endpoint, or the remote bucket/quota being unavailable.

S3BackupFailed: S3 backup has failed

A scheduled backup of the croit database to S3 failed.

Usual causes: invalid or expired S3 credentials, the endpoint being unreachable, or the target bucket being missing, full, or write-protected.

MGR

DuplicateMgrServicesOnServer: Duplicate MGR services found

A server has more than one MGR service database entry.

Usual causes: a MGR service was re-added without removing a stale entry, often after a server rename or reinstall.

Croit

AutoUpdateFailed: croit update failed

An automatic croit update task failed.

Usual causes: the update package could not be downloaded, disk space ran out during extraction, or a post-update step (e.g. container restart) failed.

MultipleDefaultGateways: Multiple default gateways configured

A server's network interfaces/VLANs resolve to more than one distinct default gateway address.

Usual causes: overlapping or duplicated network/VLAN configuration assigns conflicting gateways to the same server.

NoDefaultGateway: No default gateway configured

None of the cluster's configured networks have a default gateway set, so servers cannot reach other networks.

Usual causes: a network was set up without a gateway by mistake. This warning does not appear for clusters running in offline mode, since air-gapped clusters commonly have no gateway anywhere by design.

BootIpNotFound: Incorrect network configuration

A server's configured boot IP/interface no longer exists on the host.

Usual causes: the network interface was renamed, replaced, or reconfigured outside of croit. PXE boot fails until the configuration is corrected.

BootNetworkNetmaskMismatch: Boot network mismatches

A server's live network interface has a different prefix length than the netmask configured for its boot network.

Usual causes: the boot network's netmask was changed in croit without updating the interface configuration on the host, or vice versa.

NTPUnconfiguredInOfflineMode: NTP server unconfigured

The cluster is running in offline mode but no NTP server is configured.

Usual causes: offline mode was enabled without providing a local NTP source, leaving nodes without a time-sync target. Clock drift between nodes can then cause authentication and cluster-map timing issues.

LegacyNtpConfigFile: NTP config file is no longer used

A /config/ntp.conf is present on the management node, which keeps time with chrony now. chrony cannot read that file, so croit ignores it.

Usual causes: the file was written for the ntpd croit used to run on the management node. List your NTP servers in the croit ntp server setting instead, which configures the management node and every storage server; for anything that setting does not cover, edit the chrony-management template under Config. Delete /config/ntp.conf afterwards to clear the warning.

NotSynchronised: The management node's clock is not synchronized

The management node reached none of its NTP servers, so its clock is drifting.

Usual causes: the servers in the croit ntp server setting are not reachable from the management node, which is common on an offline cluster where the setting names hosts only the storage network can reach. The warning lists the servers the management node could not reach, so correct those in the setting. The storage servers read the time from the management node unless that setting gives them servers of their own, so Ceph starts reporting clock skew between the monitors once they disagree.

UnhealthyDisks: Disk health warnings

One or more disks report SMART health warnings.

Usual causes: disks approaching end-of-life, reallocated sectors, or other early-failure indicators reported by SMART. Plan to replace the affected disk before it fails outright.

BrokenEncryptedOsds: Attention: Imminent data loss

An encrypted DB/WAL partition fails its passphrase check, or an unused encrypted volume is unexpectedly still open.

Usual causes: an interrupted encryption operation or a partially completed disk-replacement/rekey step left the encryption state inconsistent. Do not reboot the affected host; contact support.

SshdMisconfigured: SSH misconfigured

A server's SSH daemon configuration was not applied successfully.

Usual causes: a manual edit to sshd_config conflicts with croit-managed settings, or the SSH service failed to reload after a configuration push.

SambaRecoveryLockUpgradeAvailable: Samba CTDB recovery lock can be upgraded

A Samba service still uses the legacy CephFS-file-based CTDB recovery lock instead of the RADOS-based helper lock.

Usual causes: the service was created before the RADOS-based lock was introduced and has not been migrated yet.

FailedMultipathPaths: Failed disk paths

A multipathed disk reports one or more failed (inactive) paths.

Usual causes: a cable, HBA port, or switch/fabric issue on one of the redundant paths. The disk is still reachable via its remaining path(s), but redundancy is reduced.

VulnerableSambaADAuditLogs: Active Directory login exposed

Audit logs from an older croit version recorded Active Directory admin credentials in plain text.

Usual causes: the cluster ran a croit version predating the audit-log redaction fix while Samba AD authentication was configured. Rotate the exposed credentials after upgrading.

VulnerableSambaSimpleAuditLogs: SMB passwords exposed

Audit logs from an older croit version recorded SMB root/croit-ro passwords in plain text.

Usual causes: same as VulnerableSambaADAuditLogs, for simple (non-AD) Samba authentication.

DeprecatedSyslogConfig: Syslog config has been deprecated

The deprecated syslogTarget configuration option is set.

Usual causes: the cluster was configured before VictoriaLogs became the recommended log destination and has not switched over.

LicenseExpirationWarning: License expires soon

The croit license will expire within 7 days.

Usual causes: license renewal has not been processed yet ahead of the expiry date.

LicenseAlreadyExpired: License has expired

The croit license has expired.

Usual causes: license renewal was not completed before the expiry date. Enterprise features stop working until a new license is applied.

CleanUpMetricsAndLogs: Clean up old logs

Old logs/metrics from before the migration to VictoriaLogs/VictoriaMetrics are still present.

Usual causes: the migration to VictoriaLogs/Metrics completed, but the one-time cleanup step was never run.

QuarantinedLogPartitions: Corrupted log data was moved aside

A day's worth of stored log data was corrupted, so croit moved it to /logs/quarantine on the management node to let log collection start again. The recommendation names the day it lost, or, when several partitions went, how many and the span they fall in, plus how much disk space they hold. It offers a button that deletes the data.

Deleting the quarantined data is what gets the disk space back. croit has no recovery for it and its log retention only covers the live log storage, so nothing else ever clears the quarantine.

Worth looking into is what corrupted the data in the first place, since the usual causes say something about the management node rather than about logging. Contact croit support if you cannot explain the corruption, if you would like a hand, and especially if it keeps coming back.

Usual causes: the management node lost power, or its disk is failing or ran out of space while the log storage was writing, leaving a partition without its index.

LogPartitionQuarantineFailed: Log collection has stopped

Corrupted log data was found but could not be moved out of the way, so log collection is not running. No new logs are being recorded until this is resolved.

Usual causes: /logs on the management node is full or not writable. croit retries every time the log service starts, but it cannot clear this on its own, so contact croit support if freeing up space or fixing the permissions does not bring logging back.

LegacySambaConfig: Legacy Samba configuration

Some SMB gateways still run the older Samba configuration layout.

Usual causes: the cluster's SMB gateways were set up before the current configuration format existed and were never migrated. The recommendation offers a button that upgrades them.

ManagementNodeUnderprovisioned: Management node underprovisioned

The management node's actual CPU/RAM/disk falls below what is recommended for the cluster's size.

Usual causes: the cluster grew (more OSDs/servers) without scaling up the management node's resources to match.

ManagementNodeInfoParsingFailed: Failed to determine a resource

croit could not determine one of the management node's CPU/RAM/disk resource values.

Usual causes: an unusual virtualization layer or restricted /proc//sys access prevents croit from reading the expected system information; contact support.

CroitVersionWarning: Version-specific warning (name/description vary)

The croit API published a warning for the exact croit version currently running.

Usual causes: a known issue was discovered in that specific release after it shipped. The warning text and severity come from croit's update-info service, so they vary by version.

CroitUpdateAvailable: croit update available

A newer croit version is available than the one currently installed.

Usual causes: none required; croit ships regular releases, so this is expected and not a sign of local misconfiguration.

DockerSocketNotAvailable: Docker not available

croit could not connect to a Docker socket, and no DOCKER_SOCKET override file exists.

Usual causes: the Docker socket was not mounted into the croit container, so auto-update and self-management features are unavailable.

DockerSocketNotWorking: Docker not reachable

A Docker socket is mapped but connecting to it still fails, for a reason other than the more specific cases below.

Usual causes: a stale or wrong socket path, or a Docker daemon that is not running on the host.

DockerSocketPermissionDenied: Docker permission denied

Connecting to the Docker socket fails with a permission error.

Usual causes: SELinux/AppArmor policies or file permissions on the host prevent the croit container from using the mounted Docker socket.

DockerCroitContainerNotFound: croit Docker container not found

croit can reach Docker but cannot find its own running container by the expected name.

Usual causes: the croit container was renamed away from croit, breaking self-management features that look it up by name.

DockerLegacyVolumes: Legacy Docker volume setup detected

croit's data is stored using the older bind-mount volume layout instead of named Docker volumes.

Usual causes: the installation predates the named-volume layout and has not been migrated.

DockerNotRunningInContainer: croit is not running in Docker

Docker is reachable, but the croit process itself does not appear to be running inside a container.

Usual causes: croit was started directly on the host instead of via the supported Docker deployment, which is unsupported.

DockerCapSysTimeNeeded: SYS_TIME needed

The croit container is missing the SYS_TIME Linux capability, and NTP management is not disabled.

Usual causes: the container was started with a capability set that dropped SYS_TIME, preventing croit from managing system time.

DockerCapNetAdminNeeded: NET_ADMIN needed

The croit container is missing the NET_ADMIN capability, and DHCP/dnsmasq management is enabled.

Usual causes: the container was started with a restricted capability set that prevents croit's DHCP stack from configuring network interfaces.

LowDiskSpaceManagementNode: Low disk space

One of the management node's monitored volumes (/logs, /stats, /var/lib/mysql, or /) has free space at or below its configured minimum threshold.

Usual causes: log/metric retention outgrew allocated disk space, or the database grew faster than expected. Free up space or extend the volume.