Announcing YugabyteDB 2.8

Yugabyte is on a mission to build the default database for cloud native applications and simplify operational data infrastructure. Our latest stable release — YugabyteDB 2.8 — is a big step towards that goal. YugabyteDB 2.8 includes significant enhancements to the open source database and our self-managed DBaaS offering, Yugabyte Platform. While we add more and more features on the Distributed SQL side, we are committed to making Yugabyte Platform easy for our users to use in their production cloud environments.

In the following sections, we go over these enhancements in YugabyteDB 2.8.

Yugabyte Platform API: Securely deploying YugabyteDB clusters using Infrastructure-as-Code

The Yugabyte Platform console orchestrates and manages YugabyteDB clusters in one or more regions across public cloud and private, on-prem data centers. Yugabyte Platform console is a powerful interface with an immense amount of automation to set up your databases across all major cloud providers around the globe. Through the UI, you can easily deploy a fully scalable global cluster across several regions and even across different cloud providers in a matter of minutes. The UI is super intuitive and great, but you need an API to automate these operations in your way.

With the YugabyteDB 2.8 release, the Yugabyte Platform API opens new doors for automating operations in your environment using your favorite tools and programming language. It allows you to programmatically interact with the Yugabyte Platform console to perform a variety of administrative functions using a secure RESTful API.

The new Yugabyte Platform API enables users to integrate Yugabyte Platform with all other aspects of their software development life cycle. This allows DevOps teams to create automation on their current processes across all their environments like dev, testing/QA, UAT, and production. For example, DevOps teams can leverage APIs to create ephemeral databases to run their CI/CD processes in lower environments for test purposes. Once it is done, they can just terminate the database deployment.

The Yugabyte Platform API reference guide provides details on all of the available endpoints, including parameters and responses. This guide is an essential resource for learning the ins and outs of the API. Please refer to this blog post, where I highlight some of the key features of the Yugabyte Platform API and provide some next steps for getting started.

Alerts and notifications in Yugabyte Platform

Yugabyte Platform offers a dedicated observability and monitoring view, which serves as a single pane of glass for the customer’s performance metrics across all clouds. It also comes with various default alerts and built-in features to protect against failure, such as health checks and a metrics dashboard.

In addition to the above features, it is critical to actively monitor the overall health and performance of clusters running in production by creating alerting rules that promptly send notifications when there are events that require either investigation.

We also understand that every customer is unique, so we offer our customers the choice of how they want to use those pre-defined rules. They can be tuned to customer-specific use cases and requirements.

In YugabyteDB 2.8, we outsourced the tedious task of watching observability screens to intelligent, reliable automated assistants. These assistants can transform all anomalies into notifications when something abnormal happens to the database deployments. This new feature helps our customers organize and curate all events into categories. It also removes repetitive notifications and allows focus on the important notifications for each functional team.

To get notified in real time about database alerts, users can set alert policies based on their universe metrics. Alert policies notify you when a performance metric rises above or falls below a threshold you set. Please refer to this blog post, where I walk through how this new feature works and give you a taste of what to expect in future releases.

Support for root CA certificate rotation

Yugabyte Platform allows you to protect data in transit by using both node-to-node and client-to-node encryption. Yugabyte Platform allows you to create the following certificates:

1. Self-signed certificates generated by Yugabyte Platform

2. Self-signed self-provided certificates

3. Custom CA-signed self-provided certificates

Your organization’s compliance policy requires periodical certificate rotation. You may need to rotate the node, client, or root CA certificates because they may be expiring soon, you need to modify the contents of a certificate, or the key for the certificate is compromised.

Yugabyte Platform already allows rotation for node certificates. In the 2.8 release we expanded on this to add rotation support for all types of certificates along with root certificates.

Seamlessly upgrade to major releases without stopping the database

Creating a new Yugabyte cluster initializes YSQL with a snapshot of pre-run initdb, which creates and populates system tables. Major YugabyteDB releases regularly add new features that often change the layout of the system tables, but the internal data storage format rarely changes. Postgres pg_upgrade uses this to perform upgrades by creating new system tables, but it needs the database to be stopped.

In the 2.8 release, with the new upgrade_ysql, existing YSQL deployments can upgrade to major versions without the data dump/reload typically required for major version upgrades.

The upgrade is seamless, online and does not require stopping a running cluster. This simple to use tool establishes one connection per database and it is idempotent, so rerunning in case of failures/time-outs does not cause any side effects.

An upgrade is performed manually through yb-admin after the usual Yugabyte upgrade steps are performed:

$ ./bin/yb-admin upgrade_ysql

After completion of the upgrade, you’ll see the following output on the console.

YSQL successfully upgraded to the latest version

The upgrade_ysql advances the cluster to the latest YSQL version by applying a sequence of migrations through a special pre-engineered YSQL upgrade supplied as a part of a Yugabyte installation. It is designed to advance a YSQL cluster forward one version each. It applies all of them in order and YSQL is brought to the latest version.

New YSQL built-in function: yb_hash_code

The new yb_hash_code is a YSQL built-in function that can take the primary key columns of a row and return a 16-bit hash value used by DocDB to partition a table. Tables are partitioned into tablets, with each tablet being responsible for a range of partition hash values. In effect, it provides direct access to the hash value of any given row of a YSQL table, allowing users to infer a row’s physical location.

This function is useful in various scenarios as it allows callers to run parallel queries (or jobs) that work on different slices of YSQL table data. For example, writing a parallel table export, transform, or copy utility.

The yb_hash_code function can be used in any YSQL expression and you can input a variable number of parameters whose types must be allowed in primary keys.

yb_hash_code(a1 : t1, a2 : t2, a3 : t3, a4 : t4…..) → int4 (32 bit integer)

Where a1, a2, a3… are expressions of type t1, t2, t3…, respectively. t1, t2, t3… must be types that are currently allowed in a primary key.

This function can be either evaluated at the YSQL layer after resolving each of its argument expressions, or it will be pushed down to be evaluated at the DocDB layer if possible. This pushdown will happen if the set of argument expressions matches the full hash key of the requested index.This pushdown functionality is supported for secondary indexes as well.

Given a set of values, we can compute the hash value of them directly as follows:

SELECT yb_hash_code(1::int, 2::int, ‘my string’::text);
   yb_hash_code
---------------------
      1047

PITR support in YSQL

Point-in-Time Recovery (PITR) enables granular data protection with a low recovery point objective (RPO) and recovery time objective (RTO), and has minimal impact on the cluster. PITR allows you to restore the state of your cluster’s data metadata about database objects from a specific point in time.  This can be relative, such as “three hours ago”, or an absolute timestamp. We already supported this feature on the CQL side and with the YugabyteDB 2.8 release we added support for this feature in SQL. Refer to the Features, Use cases, and Limitations doc pages for details on this feature.

Also in this release

In addition to the above key capabilities, YugabyteDB 2.8 includes the GA version of several features and improvements announced in the YugabyteDB 2.9 release:

1. Transaction savepoints that allow all statements executed after it was established to be rolled back, restoring the transaction state to what it was at the time of the savepoint.

2. Faster compaction with TTL (Time To Live) file expiration: Instead of iterating through every K/V pair in the file to identify expired entries, we have done optimization to improve disk usage and compaction performance by directly removing files that have completely expired based on their TTL.

3. RPC compression to significantly reduce costs associated with cross-AZ or cross-region network traffic by compressing internode network traffic with minimal CPU overhead.

4. Support for chunking/throttling options for secondary index backfill operations in order to control performance.

5. The smart client driver for SQL that allows applications to get better performance and fault tolerance by connecting to any node in a distributed SQL database cluster without the need for an external load balancer.

What’s coming – roadmap teaser

At Yugabyte, we strive to be fully transparent with our customers and user community, and to that end we publish our roadmap on GitHub. Here are some notable features you can expect in

upcoming releases. Note that the current roadmap is subject to change as we finalize our planning for the next releases.

Database

Several database features are on the roadmap, continued work to unlock even greater database performance is always a work in progress.

• YSQL Follower Reads – Support for read-only transactions being run against follower nodes.
• Support for Internationalization and Collation in YSQL – Support a COLLATE option for text-based columns. In particular, support for primary key and indexed columns requires specific DocDB handling to ensure correct equality and sorting semantics.
• Automatic tablet splitting GA – Support automatic splitting of tablets, based on size thresholds, while not affecting user operations.
• Support for GIN Index 

Platform

• xCluster replication management UI – An easy and user-friendly interface for setting up xCluster replication, ensuring setup correctness, monitor, and tracking xCluster replication through the Yugabyte Platform console rather than using CLI commands
• Integration with Hashicorp Vault
• Support for Ubuntu 18.04, Oracle Linux 8, and ARM
• GCP marketplace integration

Get started

We’re very happy to be able to release all of these enterprise-grade features in the newest version of our flagship product – YugabyteDB 2.8. We invite you to learn more and try it out:

• YugabyteDB 2.8 is now available in General Availability (GA). You can install the release in just a few minutes.
• Join us in Slack for interactions with the broader YugabyteDB community and real-time discussions with our engineering teams.