How to debug NTP Issues

From time to time the NTP may not have set the correct time on the system. This can be seen for example in mon server clock skew error messages. The procedure is always the same. First we check if the croit management node has a correct time and runs synchronized to other external NTP servers. These are the servers it uses:

root@mgmt-node ~ $ docker exec -it croit cat /etc/chrony.conf | grep -E '^(pool|server)'
pool 2.rhel.pool.ntp.org iburst

Make sure you can actually resolve those external NTP servers. If not, configure the croit ntp server setting in the croit settings page or fix the DNS setup on your docker host. Servers you configure there are used by the management node and by all storage servers, and they replace the pool above:

root@mgmt-node ~ $ docker exec -it croit cat /etc/chrony.conf | grep -E '^(pool|server)'
server ntp-1.example.com iburst
server ntp-2.example.com iburst

To see whether the management node is actually synchronized, and to which source:

root@mgmt-node ~ $ docker exec -it croit chronyc tracking
root@mgmt-node ~ $ docker exec -it croit chronyc -n sources

The management node keeps time with chrony. A storage server uses whichever time daemon its image enables, chrony or ntpd, so the commands differ depending on where you run them: chronyc inside the croit container, and chronyc or ntpq on a storage server.

For anything the croit ntp server setting does not cover, edit the chrony-management template under Config > Templates (chrony-server and ntp-server are the storage servers' equivalents). The management node serves the time to the networks configured in croit and to nobody else, so a storage server that cannot reach it is worth checking there first.

To fix the current time, connect to the management server via SSH, but not to the docker container. Make sure that no ntp is running at the moment:

root@mgmt-node ~ $ timedatectl status
...
System clock synchronized: no
...

If the system time is not synchronized, the update of the Ceph Storage Server fails. In this case you can initiate a manual update of the NTP time. First stop all running NTP services on the server, no matter if inside or outside the container.

root@mgmt-node ~ $ timedatectl set-ntp off
root@mgmt-node ~ $ docker exec -it croit pkill -9 chronyd

Please check that really no other NTP service is running. Depending on the operating system used, a different NTP implementation may be used, which may result in other commands being correct.

To manually update the system time over the network, execute the following command:

root@mgmt-node ~ $ docker exec -it croit chronyd -q 'server 0.pool.ntp.org iburst'
2026-09-14T11:25:38Z chronyd version 4.5 starting
2026-09-14T11:34:48Z System clock wrong by 542.994624 seconds (step)

Restart the docker container to restore the NTP and container to a clean state.

root@mgmt-node ~ $ docker restart croit

Now the MON servers should update the time after a few minutes and the clock skew warning should disappear. To speed this up you can force the NTP update. Connect to the appropriate server and do the following:

root@mgmt-node ~ $ docker exec -it croit ssh ${mon-host} "systemctl stop ntp.service; sleep .1; ntpd -gq; systemctl start ntp"
14 Sep 09:46:04 ntpd[3839354]: ntpd 4.2.8p12@1.3728-o (1): Starting
...
14 Sep 09:46:11 ntpd[3839354]: ntpd: time slew -0.001482 s
ntpd: time slew -0.001482s