Task Advisor Warnings

The Task Advisor continuously checks your cluster and management node and raises a recommendation whenever it finds something worth your attention. Each recommendation shown in the croit UI (and available to hook scripts via the OnTaskAdvisorFindingAdded/OnTaskAdvisorFindingRemoved events) corresponds to one of the warning types below. For some warnings the UI also shows one or more buttons that apply a fix automatically, so it is worth checking a recommendation in the UI before fixing it by hand. This page lists every warning type with a general description and its usual causes; the croit UI additionally shows a short, per-cluster summary and, where applicable, links to the affected object.

Ceph

CephReleaseDoesNotMatch: Unexpected ceph version

The cluster is running a different Ceph release than the one configured in croit.

Usual causes: the Ceph packages/containers were upgraded (or downgraded) outside of croit, or the configured Ceph version in croit was never updated to match a completed upgrade.

MultipleCephReleasesRunning: Mismatched ceph versions

Some nodes in the cluster are running a different Ceph release than others.

Usual causes: a rolling upgrade is stuck partway through, or a node was reinstalled or manually patched without going through croit's upgrade flow.

ContainerMismatch: Switch to a different container

The cluster is configured to run one Ceph release's container image while a different default image is expected for that release.

Usual causes: croit's bundled container defaults changed after a croit update, and the cluster has not yet been switched over to the new container.

CephEOLWarning: Release nearing/at end of life

The currently configured Ceph release will no longer receive updates.

Usual causes: the cluster has not been upgraded in a while and is still running an older major Ceph release (Luminous through Quincy).

MON

MonCountBelowMin: Too few MONs

The cluster has only one or two MON services.

Usual causes: a small or test cluster was never scaled up, or MON services were removed without replacing them. Croit recommends 3 MONs for smaller and 5 MONs for bigger clusters, so that quorum survives the loss of a single node.

MonCountEven: Even number of MONs

The cluster has an even number of MON services (4 or 6).

Usual causes: a MON was added without considering that Ceph quorum requires an odd count. An even count adds a node without improving how many failures the cluster can tolerate, since quorum still needs more than half the MONs to agree.

MonCountAboveMax: Too many MONs

The cluster has more than 7 MON services.

Usual causes: MONs were added faster than they were removed during cluster growth or migrations. More than 7 MONs adds Paxos round-trip latency to every cluster map change without improving resilience.

OSD

OsdCountIsZero: No OSDs

The cluster has no OSDs, so no data can be stored.

Usual causes: a freshly bootstrapped cluster before disks were assigned, or all OSDs were removed during maintenance and not yet replaced.

OsdIsNearFull: OSD almost full

An OSD's usage ratio is above roughly 70%.

Usual causes: uneven data distribution (CRUSH weighting or PG imbalance), or overall cluster capacity approaching its limit. Left unaddressed this escalates to OsdIsTooFull.

OsdIsTooFull: OSD full

An OSD's usage ratio is above roughly 80%.

Usual causes: same as OsdIsNearFull, left unaddressed. At this level Ceph may start throttling or refusing writes to the pool, so free up space or rebalance promptly.

OsdIsSeverelyFragmented: OSD severely fragmented

A BlueStore OSD on spinning disk has very high (>98%) allocator fragmentation.

Usual causes: long-running OSDs with heavy small-object churn (e.g. RGW workloads) that have never been recreated. Fragmentation degrades read/write performance over time and usually only improves after the OSD is redeployed.

CrushmapHostOutsideRoot: Host outside CRUSH root

A host bucket exists in the CRUSH map but is not reachable from any root bucket.

Usual causes: a host was removed from a rack/root bucket during manual CRUSH map edits, or croit's expected topology and the actual CRUSH map have drifted apart. Data is not placed on OSDs on an unreachable host.

CheckMemoryConfig: OSD memory target too high

The sum of configured OSD memory targets on a server exceeds the server's total RAM.

Usual causes: OSDs were added to a server without re-tuning osd_memory_target downward, risking OOM kills under load.

CRUSH

MetadataNotOnSsd: Metadata pool on HDD

A metadata-heavy pool (e.g. an RGW metadata pool) is mapped by a CRUSH rule that does not target SSD/NVMe (or DB-on-SSD) OSDs, even though such a rule exists in the cluster.

Usual causes: the pool's CRUSH rule was never switched to the fast-media rule after SSDs were added, hurting metadata operation latency.

MinSizeTooLow: Insufficient pool redundancy

A pool's min_size is 1 (or equals k for erasure-coded pools), meaning it can serve writes with zero redundancy margin.

Usual causes: the pool was created with a permissive default, or min_size was lowered temporarily during a recovery event and never restored. A single further OSD failure can then put data at risk.

MinSizeTooHigh: Insufficient pool availability

A pool's min_size equals its size (or k+m for erasure-coded pools), so the pool becomes unavailable as soon as a single OSD goes down.

Usual causes: min_size was set equal to size by mistake, or a resilience setting was copied from a different pool profile.

MDS

CephFsAllowStandByReplay: CephFS standby replay disabled

A CephFS filesystem does not have standby-replay enabled.

Usual causes: the filesystem was created before standby-replay was recommended, or it was disabled to save resources without reviewing the resulting failover-speed trade-off.

CephFsNotEnoughStandby: Too few standby/standby-replay MDS

A filesystem using standby-replay has fewer replay MDS than active MDS, or the cluster has zero MDS in general standby.

Usual causes: MDS count was not scaled up alongside the number of active ranks, or standby MDS were removed during resource trimming. This delays failover if an active MDS goes down.

RBD Mirror

RbdMirrorPeerMissing: RBD Mirror peer missing

RBD mirroring is enabled on a pool, but no mirror peer is registered.

Usual causes: mirroring was turned on before the peer cluster/bootstrap token was configured, or a peer was removed without disabling mirroring.

RbdMirrorDaemonMissing: RBD Mirror daemon missing

Pools expect to receive mirrored data, but no rbd-mirror service is running.

Usual causes: mirroring was configured on the receiving side but the rbd-mirror daemon was never deployed, or it was stopped/removed.

iSCSI

IscsiLegacyCapacity: Data corruption issue affects your iSCSI disks

A kernel-backstore iSCSI image does not explicitly disable emulate_legacy_capacity, exposing it to a known last-512-byte-sector data-loss bug.

Usual causes: the image was created with an older croit version before this setting was defaulted to off. See the linked blog post in the UI for remediation steps.

IscsiInsufficientGateways: Only one iSCSI gateway configured

An iSCSI service has fewer than two gateway servers assigned.

Usual causes: the service was set up for testing with a single gateway and never extended for production HA. A single gateway is a single point of failure for that service's initiators.

NVMe-oF

NvmeOFConfigErrors: NVMe-oF configuration errors

One or more NVMe-oF gateway services failed to apply their configuration.

Usual causes: a gateway host is unreachable, a subsystem/namespace configuration conflicts with existing state, or the NVMe-oF service crashed mid-reconfiguration.

OS

HostOsUpdateAvailable: OS update available

A newer default OS image exists than the one currently marked default for the architecture (x86/ARM).

Usual causes: croit shipped a new OS image (e.g. with security fixes) since the cluster's default was last set.

RunningImageMismatch: Server reboot needed

One or more servers are running a different OS image than currently configured.

Usual causes: the configured image was changed after the server last booted. A reboot/PXE cycle is needed to apply it.

HostOsImageDeprecated: Outdated OS running

A server is running an OS image that has been marked deprecated.

Usual causes: the cluster has not been rebooted/updated since a newer image superseded the deprecated one.

Backup

BackupNotConfigured: No backup configured

No management-node backup (cloud or MON-disk) is configured for an installed cluster.

Usual causes: backup setup was skipped during initial installation.

MonBackupFailed: Database backup to a MON disk failed

A scheduled backup of the croit database to a MON disk failed.

Usual causes: the target MON disk ran out of space, became unreachable, or has permission/mount issues.

CloudBackupFailed: Cloud backup has failed

A scheduled cloud backup job failed.

Usual causes: invalid or expired cloud credentials, network connectivity issues to the backup endpoint, or the remote bucket/quota being unavailable.

MGR

DuplicateMgrServicesOnServer: Duplicate MGR services found

A server has more than one MGR service database entry.

Usual causes: a MGR service was re-added without removing a stale entry, often after a server rename or reinstall.

Croit

AutoUpdateFailed: croit update failed

An automatic croit update task failed.

Usual causes: the update package could not be downloaded, disk space ran out during extraction, or a post-update step (e.g. container restart) failed.

MultipleDefaultGateways: Multiple default gateways configured

A server's network interfaces/VLANs resolve to more than one distinct default gateway address.

Usual causes: overlapping or duplicated network/VLAN configuration assigns conflicting gateways to the same server.

BootIpNotFound: Incorrect network configuration

A server's configured boot IP/interface no longer exists on the host.

Usual causes: the network interface was renamed, replaced, or reconfigured outside of croit. PXE boot fails until the configuration is corrected.

BootNetworkNetmaskMismatch: Boot network mismatches

A server's live network interface has a different prefix length than the netmask configured for its boot network.

Usual causes: the boot network's netmask was changed in croit without updating the interface configuration on the host, or vice versa.

NTPUnconfiguredInOfflineMode: NTP server unconfigured

The cluster is running in offline mode but no NTP server is configured.

Usual causes: offline mode was enabled without providing a local NTP source, leaving nodes without a time-sync target. Clock drift between nodes can then cause authentication and cluster-map timing issues.

UnhealthyDisks: Disk health warnings

One or more disks report SMART health warnings.

Usual causes: disks approaching end-of-life, reallocated sectors, or other early-failure indicators reported by SMART. Plan to replace the affected disk before it fails outright.

BrokenEncryptedOsds: Attention: Imminent data loss

An encrypted DB/WAL partition fails its passphrase check, or an unused encrypted volume is unexpectedly still open.

Usual causes: an interrupted encryption operation or a partially completed disk-replacement/rekey step left the encryption state inconsistent. Do not reboot the affected host; contact support.

OldNetworkSetup: Old Network Configuration

The cluster is not using croit's network setup script.

Usual causes: the cluster was installed before the network setup script existed and was never migrated to it.

SshdMisconfigured: SSH misconfigured

A server's SSH daemon configuration was not applied successfully.

Usual causes: a manual edit to sshd_config conflicts with croit-managed settings, or the SSH service failed to reload after a configuration push.

SambaRecoveryLockUpgradeAvailable: Samba CTDB recovery lock can be upgraded

A Samba service still uses the legacy CephFS-file-based CTDB recovery lock instead of the RADOS-based helper lock.

Usual causes: the service was created before the RADOS-based lock was introduced and has not been migrated yet.

FailedMultipathPaths: Failed disk paths

A multipathed disk reports one or more failed (inactive) paths.

Usual causes: a cable, HBA port, or switch/fabric issue on one of the redundant paths. The disk is still reachable via its remaining path(s), but redundancy is reduced.

VulnerableSambaADAuditLogs: Active Directory login exposed

Audit logs from an older croit version recorded Active Directory admin credentials in plain text.

Usual causes: the cluster ran a croit version predating the audit-log redaction fix while Samba AD authentication was configured. Rotate the exposed credentials after upgrading.

VulnerableSambaSimpleAuditLogs: SMB passwords exposed

Audit logs from an older croit version recorded SMB root/croit-ro passwords in plain text.

Usual causes: same as VulnerableSambaADAuditLogs, for simple (non-AD) Samba authentication.

DeprecatedSyslogConfig: Syslog config has been deprecated

The deprecated syslogTarget configuration option is set.

Usual causes: the cluster was configured before VictoriaLogs became the recommended log destination and has not switched over.

LicenseExpirationWarning: License expires soon

The croit license will expire within 7 days.

Usual causes: license renewal has not been processed yet ahead of the expiry date.

LicenseAlreadyExpired: License has expired

The croit license has expired.

Usual causes: license renewal was not completed before the expiry date. Enterprise features stop working until a new license is applied.

CleanUpMetricsAndLogs: Clean up old logs

Old logs/metrics from before the migration to VictoriaLogs/VictoriaMetrics are still present.

Usual causes: the migration to VictoriaLogs/Metrics completed, but the one-time cleanup step was never run.

ManagementNodeUnderprovisioned: Management node underprovisioned

The management node's actual CPU/RAM/disk falls below what is recommended for the cluster's size.

Usual causes: the cluster grew (more OSDs/servers) without scaling up the management node's resources to match.

ManagementNodeInfoParsingFailed: Failed to determine a resource

croit could not determine one of the management node's CPU/RAM/disk resource values.

Usual causes: an unusual virtualization layer or restricted /proc//sys access prevents croit from reading the expected system information; contact support.

CroitVersionWarning: Version-specific warning (name/description vary)

The croit API published a warning for the exact croit version currently running.

Usual causes: a known issue was discovered in that specific release after it shipped. The warning text and severity come from croit's update-info service, so they vary by version.

CroitUpdateAvailable: croit update available

A newer croit version is available than the one currently installed.

Usual causes: none required; croit ships regular releases, so this is expected and not a sign of local misconfiguration.

DockerSocketNotAvailable: Docker not available

croit could not connect to a Docker socket, and no DOCKER_SOCKET override file exists.

Usual causes: the Docker socket was not mounted into the croit container, so auto-update and self-management features are unavailable.

DockerSocketNotWorking: Docker not reachable

A Docker socket is mapped but connecting to it still fails, for a reason other than the more specific cases below.

Usual causes: a stale or wrong socket path, or a Docker daemon that is not running on the host.

DockerSocketPermissionDenied: Docker permission denied

Connecting to the Docker socket fails with a permission error.

Usual causes: SELinux/AppArmor policies or file permissions on the host prevent the croit container from using the mounted Docker socket.

DockerCroitContainerNotFound: croit Docker container not found

croit can reach Docker but cannot find its own running container by the expected name.

Usual causes: the croit container was renamed away from croit, breaking self-management features that look it up by name.

DockerLegacyVolumes: Legacy Docker volume setup detected

croit's data is stored using the older bind-mount volume layout instead of named Docker volumes.

Usual causes: the installation predates the named-volume layout and has not been migrated.

DockerNotRunningInContainer: croit is not running in Docker

Docker is reachable, but the croit process itself does not appear to be running inside a container.

Usual causes: croit was started directly on the host instead of via the supported Docker deployment, which is unsupported.

DockerCapSysTimeNeeded: SYS_TIME needed

The croit container is missing the SYS_TIME Linux capability, and NTP management is not disabled.

Usual causes: the container was started with a capability set that dropped SYS_TIME, preventing croit from managing system time.

DockerCapNetAdminNeeded: NET_ADMIN needed

The croit container is missing the NET_ADMIN capability, and DHCP/dnsmasq management is enabled.

Usual causes: the container was started with a restricted capability set that prevents croit's DHCP stack from configuring network interfaces.

LowDiskSpaceManagementNode: Low disk space

One of the management node's monitored volumes (/logs, /stats, /var/lib/mysql, or /) has free space at or below its configured minimum threshold.

Usual causes: log/metric retention outgrew allocated disk space, or the database grew faster than expected. Free up space or extend the volume.