An introduction to yb_stats

Frits Hoogland

To fully understand your database, it’s important to have runtime information available in a single location. For monolithic databases, this information is available on the database server; however, the classic divide remains between the filesystem information (typically the logfile) and the in-database tables and views.

With distributed databases, like YugabyteDB, the database typically runs on multiple machines. If containers are used, the different processes run within their own containers/pods. Therefore, needed information is scattered across different machines/containers/pods.

A lot of the information about a single YugabyteDB daemon, like a tablet server or a master server, is provided through its web UI. For more cluster-wide management we provide YugabyteDB Anywhere, which is typically used for deployment, creating backups, etc. However, to get the “raw facts” from a YugabyteDB cluster into a single location, you can use the yb_stats tool.

What is yb_stats?

yb_stats is used to obtain YugabyteDB cluster status which can be used for troubleshooting, ad hoc analysis, and support. It can gather all needed facts from every YugabyteDB cluster component and store them in a “snapshot”. A snapshot here represents what has been gathered from a YugabyteDB cluster at a single point in time. The facts only contain metadata—not any actual user data.

A yb_stats snapshot contains a great deal of information, including:

  • All masters and tablet servers’ performance counters and gauges.
  • All masters and tablet servers’ performance “histograms”, including different percentiles and a counter and number of occasions of the histogram’s topic.
  • YSQL level performance counters.
  • YSQL level statement performance data, which is a full overview of pg_stat_statements and their statistics.
  • YCQL level performance counters and gauges.
  • All masters and tablet servers gflags.
  • All masters and tablet servers mem-trackers page information.
  • All masters and tablet servers log output (last 1M).
  • All masters and tablet servers versions.
  • All masters and tablet servers threads backtrace and runtime statistics.
  • All masters and tablet servers, YSQL and YCQL RPC (network connections).
  • All masters dump of entities (DocDB database, object, tablet and replicas).
  • All masters detailed status.
  • All masters and tablet servers dump of /memz endpoint.
  • All masters and tablet servers dump of /pprof/growth endpoint.
  • All node_exporter counters and gauges.

yb_stats has two goals. First, it fetches all the available information that can be obtained via HTTP in one go. Second, it eliminates the endless cycle of investigation, more data collection, investigation, more data collection, etc.

yb_stats stores the cluster’s data—in its entirety—in a single place as CSV files.This allows you to investigate the files manually and load the data into a database. It also allows security officers to investigate the collected data. The best way of using the data is using the yb_stats utility, which can read the CSV data and process it to show the specific data needed for the investigation using filters.

How does that work?

The normal cycle of use is:

  1. Choose a node to run yb_stats snapshots. The node must be able to see all cluster http endpoints. This is typically a management node or the first server in a YugabyteDB cluster. In addition, you need to be able to log on to that node via ssh.
  2. Install yb_stats via the RPM package for CentOS/Alma/any other RH compatible clone version 7 or 8 via yum. There also is a homebrew tap for Mac OSX available, or you can install via the source.
  3. Perform the first yb_stats execution, specifying the hosts or endpoint IP addresses and optionally the port numbers (if these have been changed from the YugabyteDB defaults). By doing so, yb_stats will write a .env file in the current working directory that stores hostnames or IP addresses, ports, and parallelism.
  4. Invoke yb_stats without any argument for ad-hoc performance query mode (which doesn’t store anything). Or you can invoke yb_stats with the –snapshot (and optionally the –snapshot-comment switch) to perform a full snapshot that stores all data in CSV files.

And what do I get?

When yb_stats is used with the –snapshot switch, it will store the data, but it will not output a lot of the data. It’s sole focus is to store the data:

% yb_stats --snapshot
snapshot number 6

yb_stats ad-hoc mode

However, if you do not specify –snapshot, you get:

% yb_stats
Begin metrics snapshot created, press enter to create end snapshot for difference calculation.

It tells you it created a begin snapshot (which is in-memory). Press enter to create an end snapshot to calculate and display the difference between the beginning and ending snapshot, per server. If you do so, you will get:

Time between snapshots:  121.299 seconds  server   cpu_stime                                                                          730 ms               6.019 /s  server   cpu_utime                                                                          442 ms               3.644 /s  server   involuntary_context_switches                                                         2 csws             0.016 /s  server   server_uptime_ms                                                                121288 ms             999.992 /s  server   threads_started                                                                      4 threads           0.033 /s  server   threads_started_thread_pool                                                          4 threads           0.033 /s  server   voluntary_context_switches                                                       38618 csws           318.397 /s   server   cpu_stime                                                                          115 ms               0.948 /s
...much data...   counter  node_vmstat_pgpgout                                                                138.000000           1.140 /s   counter  node_xfs_block_mapping_extent_list_insertions_total_sdb1                             1.000000           0.008 /s   counter  node_xfs_block_mapping_extent_list_lookups_total_sdb1                                5.000000           0.041 /s   counter  node_xfs_block_mapping_reads_total_sdb1                                              2.000000           0.017 /s   counter  node_xfs_block_mapping_writes_total_sdb1                                             1.000000           0.008 /s   counter  node_xfs_read_calls_total_sdb1                                                       3.000000           0.025 /s

These are all the yb_stats statistics, which are the counter-based and counter- and sum-based statistics per server, and the node_exporter counter based statistics for all the specified nodes.

yb_stats snapshot-diff

The above ad-hoc mode doesn’t store anything. It’s typically used when a testing cycle is performed in a controlled environment, which means that storing every begin and end situation of testing would mean too much data. In most of the other situations, and especially in client situations, it’s important to understand the entire picture. You also don’t want to continuously fetch data, especially if that requires asking someone else to do that. In such situations, storing a snapshot will get you all the available information, which you can use and saves you from needing the cluster to be available to look it up. You might also want other people to look at the data who may not be available or don’t have access to the cluster you are looking at. All the data in the snapshot is persistent. It does not change.

Once two or more snapshots are taken, a difference overview can be obtained using the yb_stats –snapshot-diff switch.

% ./target/release/yb_stats --snapshot-diff
  0 2022-11-06 14:57:19.801329 +01:00
  1 2022-11-06 15:00:08.100975 +01:00
  2 2022-11-06 15:02:09.157553 +01:00
  3 2022-11-06 17:00:14.007897 +01:00
  4 2022-11-07 22:17:35.932471 +01:00
  5 2022-11-08 14:51:56.669687 +01:00
  6 2022-11-08 14:54:42.636357 +01:00
  7 2022-11-08 15:05:27.319409 +01:00
Enter begin snapshot: 6
Enter end snapshot: 7  server   cpu_stime                                                                         4236 ms               6.573 /s  server   cpu_utime                                                                         2277 ms               3.533 /s  server   glog_info_messages                                                                  95 msgs             0.147 /s  server   involuntary_context_switches                                                        38 csws             0.059 /s  server   server_uptime_ms                                                                644483 ms             999.983 /s  server   threads_started                                                                     32 threads           0.050 /s  server   threads_started_thread_pool                                                         32 threads           0.050 /s  server   voluntary_context_switches                                                      207444 csws           321.871 /s   server   cpu_stime                                                                          698 ms               1.083 /s
...   counter  node_xfs_vnode_reclaim_total_sdb1                                                   12.000000           0.019 /s   counter  node_xfs_vnode_release_total_sdb1                                                   12.000000           0.019 /s   counter  node_xfs_vnode_remove_total_sdb1                                                    12.000000           0.019 /s   counter  node_xfs_write_calls_total_sdb1                                                    244.000000           0.379 /s

NOTE: The difference between using the ad-hoc mode and snapshot-diff mode is that the ad-hoc mode does not require stored files.

Further benefits of yb_stats snapshots

When using snapshots, more data is stored, for which it doesn’t make sense to provide it in a difference overview—shown above—such as logs or RPC network connections. The complete list of additional snapshot data is listed above.

For example, after upgrading to a YugabyteDB cluster, you can validate that all components were successfully upgraded by using a snapshot to print the version information to validate that all cluster servers show the version that is expected:

% yb_stats --print-version 7
hostname_port        version_number  build_nr   build_type build_timestamp          git_hash        231        RELEASE    22 Oct 2022 19:21:11 UTC 981ed35d6a47730ee663d8b14d541ba264dd3bc8        231        RELEASE    22 Oct 2022 19:21:11 UTC 981ed35d6a47730ee663d8b14d541ba264dd3bc8        231        RELEASE    22 Oct 2022 19:21:11 UTC 981ed35d6a47730ee663d8b14d541ba264dd3bc8        231        RELEASE    22 Oct 2022 19:21:11 UTC 981ed35d6a47730ee663d8b14d541ba264dd3bc8        231        RELEASE    22 Oct 2022 19:21:11 UTC 981ed35d6a47730ee663d8b14d541ba264dd3bc8        231        RELEASE    22 Oct 2022 19:21:11 UTC 981ed35d6a47730ee663d8b14d541ba264dd3bc8

Many more print options exist, such as logs, masters, RPC ports, gflags, etc.

Learn more about yb_stats

There are many more options, such as reducing the output by filtering on hostname, statistic name, or table name; adding gauge type statistics; enabling details to split the per table and tablet statistics to their table and tablets; and many more –print options.

If you want to learn more about yb_stats:

Frits Hoogland

Related Posts

Explore Distributed SQL and YugabyteDB in Depth

Discover the future of data management.
Learn at Yugabyte University
Get Started Business
Browse Yugabyte Docs
Explore docs Business
PostgreSQL For Cloud Native World
Read for Free Business