YugabyteDB Community Update, Tricks and Tips – Nov 22, 2019
Welcome to this week’s community update where we recap a few interesting questions that have popped up in the last week or so on the YugabyteDB Slack channel, the Forum, GitHub or Stackoverflow. We’ll also review upcoming events, new blogs and documentation that has been published since the last update. Ok, let’s dive right in:
YugabyteDB uses a tiered compaction strategy that has the advantage of lower disk write (IO) amplification. Normally, compactions require 50% free space, but with YugabyteDB this is fixed by splitting the table into multiple smaller tablets. Compactions are throttled and prioritized so they don’t exhaust system resources. For more information check out the Docs on “Throttled Compactions” plus “Small and Large Compaction Queues.”
Our tests are split into several categories:
- Module/class-level unit tests
- End-to-end functional tests
- Mini-cluster tests which simulate a multi-node setup on a single node
- Integration tests that run on real multi-node setup.
The range of things we test includes testing for correctness of functionality, correctness under failure modes like the crash of node/process, network partition etc. (e.g., Jepsen tests) and performance tests to catch unexpected regressions.
Every diff triggers an automated build, which uses our in-house, distributed build toolchain spreading work across an autoscaling GCP instance group and an in-office fleet of servers, for parallelizing our Darwin efforts. For each build, once complete, a Spark job acts as our distributed task coordination framework and runs our entire in-repo test suite, somewhere around 4,000 tests, across C++ and Java. The tests are run against binaries from multiple compilers and build targets/systems including release and debug builds. Additionally, the tests are run on two special targets: CentOS Clang AddressSanitizer (ASAN) and CentOS Clang ThreadSanitizer (TSAN). The former runs the tests under an address sanitizer which checks for memory leaks/errors, and the latter instruments memory accesses to detect potential data races.
Finally, we use Jenkins pipelines to stitch everything together and another in-house webapp – aptly named Detective – to link diffs to their respective Jenkins jobs and logs and perform some automated test failure analysis and regression detection and let engineers “play Sherlock” on code failures. All of this happens automatically. On every version. For every diff in review. And once again, after landing… All the time!
We are continuously improving our testing and delivery pipeline and view it as a key strength that allows us to move as fast as we do with development, while ensuring that every release of YugabyteDB is as reliable, robust and performant as our users deserve.
Currently, we recommend a load balancer for users who make use of the YSQL API, but it is not required when using the YCQL API. Cluster aware clients for YSQL are getting ready to enter “release candidate” stage. You can learn more about our work in this area by checking the following:
- Designing a Change Data Capture and Two Data Center Architecture for a Distributed SQL Database
- Distributed Databases Deconstructed: CockroachDB, TiDB and YugabyteDB
- We have uploaded over a dozen videos from this year’s Distributed SQL Summit. You can links to the presentations, slides and a recap of the highlights by checking out, “2019 Distributed SQL Summit Recap and Highlights.”
- YugabyteDB configuration reference documentation for TServer, Master nodes and port assignments
- Two data center (2DC) deployment instructions
- Jan 16: Silicon Valley PostgreSQL Meetup
- Dec 11: Developing Microservices with Istio Service Mesh and Distributed SQL
- Dec 18: Distributed SQL Database Administration 101
- Jan 15: Developing Cloud-Native Spring Microservices with a Distributed SQL Backend
- Dec 2-6: Las Vegas
YugaByte is growing fast and we’d like you to help us keep the momentum going! Check out our currently open positions:
- Software Engineer – Cloud Infrastructure – Sunnyvale, CA
- Software Engineer – Core Database – Sunnyvale, CA
- Software Engineer – Full Stack – Sunnyvale, CA
- Developer Advocate – Sunnyvale, CA
- Senior Technical Writer – Remote
Our team consists of domain experts from leading software companies such as Facebook, Oracle, Nutanix, Google and LinkedIn. We have come a long way in a short time but we cannot rest on our past accomplishments. We need your ideas and skills to make us better at every function that is necessary to create the next great software company. All while having tons of fun and blazing new trails!
Ready to start exploring YugabyteDB features? Getting up and running locally on your laptop is fast. Visit our quickstart page to get started.