YugabyteDB Community Update, Tricks and Tips – Nov 22, 2019
November 22, 2019
Welcome to this week’s community update where we recap a few interesting questions that have popped up in the last week or so on the YugabyteDB Slack channel, the Forum, GitHub or Stackoverflow. We’ll also review upcoming events, new blogs and documentation that has been published since the last update. Ok, let’s dive right in:
How do data compactions work in YugabyteDB?
YugabyteDB uses a tiered compaction strategy that has the advantage of lower disk write (IO) amplification. Normally, compactions require 50% free space, but with YugabyteDB this is fixed by splitting the table into multiple smaller tablets. Compactions are throttled and prioritized so they don’t exhaust system resources. For more information check out the Docs on “Throttled Compactions” plus “Small and Large Compaction Queues.”
How does YugabyteDB test its code base?
Our tests are split into several categories:
- Module/class-level unit tests
- End-to-end functional tests
- Mini-cluster tests which simulate a multi-node setup on a single node
- Integration tests that run on real multi-node setup.
The range of things we test includes testing for correctness of functionality, correctness under failure modes like the crash of node/process, network partition etc. (e.g., Jepsen tests) and performance tests to catch unexpected regressions.
Every diff triggers an automated build, which uses our in-house, distributed build toolchain spreading work across an autoscaling GCP instance group and an in-office fleet of servers, for parallelizing our Darwin efforts. For each build, once complete, a Spark job acts as our distributed task coordination framework and runs our entire in-repo test suite, somewhere around 4,000 tests, across C++ and Java. The tests are run against binaries from multiple compilers and build targets/systems including release and debug builds. Additionally, the tests are run on two special targets: CentOS Clang AddressSanitizer (ASAN) and CentOS Clang ThreadSanitizer (TSAN). The former runs the tests under an address sanitizer which checks for memory leaks/errors, and the latter instruments memory accesses to detect potential data races.
Finally, we use Jenkins pipelines to stitch everything together and another in-house webapp – aptly named Detective – to link diffs to their respective Jenkins jobs and logs and perform some automated test failure analysis and regression detection and let engineers “play Sherlock” on code failures. All of this happens automatically. On every version. For every diff in review. And once again, after landing… All the time!
We are continuously improving our testing and delivery pipeline and view it as a key strength that allows us to move as fast as we do with development, while ensuring that every release of YugabyteDB is as reliable, robust and performant as our users deserve.
Do I need an external load balancer with YugabyteDB?
Currently, we recommend a load balancer for users who make use of the YSQL API, but it is not required when using the YCQL API. Cluster aware clients for YSQL are getting ready to enter “release candidate” stage. You can learn more about our work in this area by checking the following:
- YugabyteDB JDBC driver
- YugabyteDB Spring Data Module
- See it in action with the Spring Data Example
New Documentation, Blogs, Tutorials, and Videos
New Blogs
New Videos
- Designing a Change Data Capture and Two Data Center Architecture for a Distributed SQL Database
- Distributed Databases Deconstructed: CockroachDB, TiDB and YugabyteDB
- We have uploaded over a dozen videos from this year’s Distributed SQL Summit. You can links to the presentations, slides and a recap of the highlights by checking out, “2019 Distributed SQL Summit Recap and Highlights.”
New Docs
- YugabyteDB configuration reference documentation for TServer, Master nodes and port assignments
- Two data center (2DC) deployment instructions
Upcoming Meetups and Conferences
PostgreSQL Meetups
- Jan 16: Silicon Valley PostgreSQL Meetup
Distributed SQL Webinars
- Dec 11: Developing Microservices with Istio Service Mesh and Distributed SQL
- Dec 18: Distributed SQL Database Administration 101
- Jan 15: Developing Cloud-Native Spring Microservices with a Distributed SQL Backend
AWS re:Invent
- Dec 2-6: Las Vegas
Get Started
Ready to start exploring YugabyteDB features? Getting up and running locally on your laptop is fast. Visit our quickstart page to get started.
What’s Next?
- Compare YugabyteDB in depth to databases like CockroachDB, Google Cloud Spanner and MongoDB.
- Get started with YugabyteDB on macOS, Linux, Docker, and Kubernetes.
- Contact us to learn more about licensing, pricing or to schedule a technical overview.
