Achieving Accurate Clock Management in YugabyteDB

Introduction

In our recent blog, Achieving Precise Clock Synchronization on AWS, we discussed why clock management is vitally important to distributed systems. An accurate clock ensures correctness, and a more accurate clock ensures enhanced performance in a YugabyteDB Universe.

YugabyteDB leverages a hybrid logical clock to provide distributed synchronization, but requires that no two nodes are more than 500 ms apart. Accurate clocks and synchronization allow YugabyteDB to deliver this.

AWS offers market-leading clock synchronization technology with Clockbound and Precision Time Protocol, but there are many scenarios where using this functionality is not possible. Enter self-managed database-as-a-service YugabyteDB Anywhere, which employs several methods to help ensure the clock on all nodes in a universe is accurate and remains accurate.

In this blog, we discuss how YugabyteDB Anywhere achieves accurate clock management. First, by ensuring Network Time Protocol services are set up and running correctly from the initial creation of the universe, and throughout its entire lifecycle. Second, by constantly monitoring every node to ensure clock skew stays low.

What Are NTP Services?

To begin, let’s review what Network Time Protocol (NTP) is and how it works. NTP is at the heart of keeping internal node clocks in sync. It is a protocol used to synchronize the clocks of computers over a network, to ensure consistent time across multiple devices.

NTP uses a hierarchical system of time sources, called “strata,” where Stratum 0 represents the most accurate time sources (like atomic clocks), and Stratum 1 servers synchronize with Stratum 0, and so on. NTP helps maintain precise time synchronization with millisecond accuracy, even over long distances, by compensating for network delays and time drift.

There are several implementations of this protocol, and YugabyteDB Anywhere supports:

• Chrony (recommended)
• Ntpd
• Systemd-timesyncd

In addition to requiring an NTP service running on each node, it is also possible to run YugabyteDB with AWS Precision Hardware Clock (PHC) and Precision Time Protocol (PTP) – both to be used alongside Chrony.

PHC and PTP can decrease the clock skew of a node to under 40 microseconds. We can also use AWS Clockbound to give YugabyteDB the exact error bounds for clock drift, which can be used to further improve read performance. We have steps for PHC and PTP documented HERE.

Clock Management During Initial Provisioning

Provisioning is the set of steps YugabyteDB Anywhere takes to set up virtual machines for YugabyteDB installation. It ensures that all dependencies are met by installing necessary packages and sets operating system-level configurations. It is during provisioning that YugabyteDB Anywhere will install and configure NTP services.

When provisioning, YugabyteDB Anywhere can directly integrate with AWS, GCP, or Azure as “cloud providers” to fully manage the lifecycle of nodes in a universe, including the creation of the virtual machines. There is also a notion of on-premises providers, where the nodes must be provided to YugabyteDB Anywhere.

Cloud Providers

When provisioning a universe in a public cloud, YugabyteDB Anywhere can set up and configure Chrony on each node.

We start by making sure Chrony is installed. From here, it must be configured to use the recommended NTP server for the cloud being used. For example, metadata.google.internal for GCP and 169.254.169.123 for AWS. These NTP Servers use atomic clocks to deliver extremely accurate timings.

A user can also specify custom NTP servers if the options above do not meet their requirements. When creating a cloud provider, use the option “Specify Custom NTP Servers.” YugabyteDB Anywhere will still install Chrony, and then configure Chrony to use the selected NTP servers instead of the cloud defaults. As a final option, YugabyteDB Anywhere can ignore any setup of Chrony and assume the servers already have an NTP service configured. This is useful if using a different NTP service like Ntpd or systemd-timesyncd, or if installing Chrony with custom, advanced configuration values. For example, to configure Chrony with PHC or PTP.

On-Premises

When creating a universe using the on-premises provider, YugabyteDB Anywhere will still give you the option to configure custom NTP servers with Chrony or to assume NTP is already configured.

The difference is that it will be unable to automatically configure the nodes to use AWS or GCP Timesync servers – though these can be manually specified by the user if desired. YugabyteDB Anywhere would not know where the given nodes are running to correctly use a cloud’s time servers. If using on-premises with fully manual provisioning, the NTP service needs to be fully configured by the user.

AWS Clockbound

To leverage AWS Clockbound, it should be installed on all nodes in the universe. Steps are provided in our documentation HERE.

With both Masters and Tservers having “time_source” set to “clockbound”, the YugabyteDB Universe will use AWS Clockbound with the internal physical clock to get an upper limit on the clock skew. This can improve latency in the system as we lower the maximum clock skew – especially if PTP is in use. Find out more HERE.

Monitoring

In addition to the initial deployment checks YugabyteDB Anywhere performs during provisioning, there is also active monitoring of each node in the universe. This will alert users if any node starts to fall out of sync with the remote NTP server or with other nodes in the universe.

First, we collect the metric “yb_node_clock_drift_check_ms” for each node. This metric is used in both alerts and health checks to show when a node has drifted too far from NTP server. There is a 200ms warning threshold and a 400ms error threshold. There is also a metric “yb_node_ntp_service_status” to track the NTP service status on a node. This is binary 0 or 1 depending on the status. 1 for success and 0 for failure.

An example of a failed status is when timedatectl status indicates that the NTP service is inactive or that the System clock is not synchronized. The root cause is not found here, only an indication that the node’s clock is in a bad state

[centos@dev-server-dshubin ~]$ timedatectl status
               Local time: Thu 2024-12-05 00:08:00 UTC
           Universal time: Thu 2024-12-05 00:08:00 UTC
                 RTC time: Thu 2024-12-05 00:08:01
                Time zone: UTC (UTC, +0000)
System clock synchronized: yes
              NTP service: inactive  <--- NTP service not running
          RTC in local TZ: no

[centos@dev-server-dshubin ~]$ timedatectl status
               Local time: Thu 2024-12-05 00:08:10 UTC
           Universal time: Thu 2024-12-05 00:08:10 UTC
                 RTC time: Thu 2024-12-05 00:08:11
                Time zone: UTC (UTC, +0000)
System clock synchronized: no  <----- System clock is not synced
              NTP service: active
          RTC in local TZ: no

While the NTP service being offline may not cause immediate failures in the universe, it will allow the node to slowly get out of sync with both the NTP server and the rest of the universe. This will lead to worse performance and eventually may crash the universe if the difference becomes too great; remember earlier, each node must be within 500ms of every other node.

There is also a metric directly collected from the database called “hybrid_clock_skew” with a related Clock Skew alert. This metric reports the greatest clock difference between any two nodes in the universe. It is useful as we can have a node that is 150ms slow and a second that is 150ms fast. Neither node would trigger the previous alerts, but the 300ms difference between the nodes will trigger this clock skew alert and warn us that nodes in the universe may be getting out of sync with each other.

How YugabyteDB Maintains Clock Health

The final actions YugabyteDB Anywhere takes to maintain the universe clock’s health are with the Systemd service files for the Tserver and Master.

YugabyteDB Anywhere uses the “ExecStartPre” directive to run a custom clock sync check.

ExecStartPre will block the main service if the clock sync check does not pass. The clock sync check will prevent YugabyteDB services on a node from starting when a node’s time drift is too large or its NTP service is not running, as this could further degrade the universe. The service files will continue to try starting the service, so a single clock sync failure will not keep the node offline if this is a temporary issue and will automatically start the services when the issue is resolved.

Conclusion

Managing time in a cluster is a core requirement for maintaining a healthy distributed system. YugabyteDB accomplishes this by leveraging industry-standard tools like Chrony and Systemd to both keep node clocks in sync and to prevent bad nodes from joining an otherwise healthy Universe. It also actively monitors each node and alerts users if nodes start to get out of sync. This allows you to take action before the time difference becomes too great.

To find out more about how YugabyteDB uses clock sync and its importance to the database, check out ‘A Matter of Time: Evolving Clock Sync for Distributed Databases.’