The Principle of the Elasticity Technology of the Cloud-native Data Warehouse AnalyticDB 🌐🔍.

Article hero image

Overview 📄

In the dynamic landscape of digital transformation, businesses are seeking innovative solutions to maximize resource utilization and cost efficiency. Alibaba Cloud’s cloud-native data warehouse, AnalyticDB for MySQL Data Lakehouse Edition (hereinafter referred to as AnalyticDB for MySQL), has introduced a revolutionary multi-cluster elastic resource model 🔀 to address this challenge. This model adapts to user demand, automatically configures resources, and incrementally improves performance, further aiding users in reducing expenses and enhancing computational efficiency. 🚀

Introduction to the Elastic Models 🔢

AnalyticDB for MySQL offers two distinct elastic models to cater to diverse business needs:

1. Min-Max Elastic Model ⬆️⬇️

  • The resources available to a single SQL statement can be scaled between minimum and maximum values.
  • Suitable for ETL scenarios, enhancing the performance of individual SQL statements.
  • Example: If the minimum is set to 16 cores and the maximum is set to 32 cores, a single SQL statement will utilize resources within the range of 16 to 32 cores.

2. Multi-Cluster Elastic Model 🧑‍🔧

  • Resources are scaled at the cluster level, with each SQL statement confined to a single cluster.
  • Suitable for online analysis and interactive analytic scenarios, improving SQL concurrency.
  • Example: With a single cluster size of 16 cores, and the minimum and maximum number of clusters set to 1 and 2, respectively, each SQL statement can utilize 16 cores (one cluster). As concurrency increases or decreases, the number of clusters automatically adjusts between 1 and 2, ensuring isolated execution.

Advantages of the Multi-Cluster Elastic Model ✨

The multi-cluster elastic model addresses the limitations of the min-max elastic model, offering enhanced usability, performance, and cost-effectiveness.

1. Usability 📝

  • Automatic cluster adjustment based on real-time business load, eliminating the need for manual scaling.
  • Users only need to define the upper and lower limits of the number of clusters and the size of each cluster.

2. Performance 🚀

  • Cluster isolation ensures that a single SQL statement only affects its own cluster, preventing interference with other SQL statements.
  • Query concurrency improves linearly as the number of clusters increases, with up to a 28% increase compared to the min-max elastic model.

3. Cost Savings 💰

  • Dynamic cluster scaling based on user load, handling business peaks and troughs efficiently.
  • Example: By using the multi-cluster elastic model, a user can achieve approximately 38.7% cost savings compared to scheduled elasticity.

Multi-Cluster Technology Architecture 🏗️

The AnalyticDB MySQL multi-cluster model is designed to achieve accurate, fast, and efficient scaling, with three core layers:

  1. Access Layer 🌐: Delivers user queries to specific resource groups and distributes them to clusters based on load.
  2. Execution Layer ⚙️: Within each resource group, multiple clusters of the same size execute user queries.
  3. Decision Layer 🧠: Continuously monitors resource load to make informed scaling decisions for multi-cluster resource groups.

Fast Scaling: Real-time Data Link ⏱️

  • AnalyticDB for MySQL implements a region-level metric collection system to ensure timely scaling.
  • Internal business metrics (queued queries, CPU usage) are updated in real-time and collected by the metric collection process, with a delay of approximately 10 seconds.

Accurate Scaling: Stable Scaling Policies 🎯

To address the challenges of bottleneck identification, metric selection, and scaling decision validation, AnalyticDB for MySQL employs a three-stage approach:

1. Decision-making 🧠

  • Bottleneck Identification: Positive metrics provide load status feedback for scaling decisions, while negative metrics identify external bottlenecks.
  • Estimation of the Cluster Number: Candidate clusters are calculated based on user CPU utilization, memory usage, and queued queries.
  • Stability Window: A stability window algorithm is used to prevent metric jitter and ensure stable scaling decisions.

2. Execution ⚙️

  • The in-house operator manages clusters and Kubernetes custom resources, implementing the Kubernetes scale subresource.
  • After the decision-making system determines the target cluster number, it sends the request to the custom operator through the Kubernetes scale API for scaling.

3. Feedback: Effectiveness Evaluation 📊

  • After scaling, the decision-making system evaluates the effectiveness by observing changes in query performance metrics.
  • If the scaling is deemed invalid, the system restores the original cluster number and sends an alert.

Good Scaling: Routing Policy Based on Load Balancing ⚖️

  • AnalyticDB for MySQL automatically routes queries to the cluster with the minimum load based on a load balancing algorithm, ensuring efficient resource utilization.

Summary 🏁

The AnalyticDB MySQL Multi-Cluster elastic model offers the following benefits:

  1. Cost Savings 💰: Automatic scaling in and out to fit business loads, reducing costs compared to fixed resources in a single cluster.
  2. Query Performance 🚀: Linear query increase and superior query isolation compared to the min-max resource group model.
  3. Automatic Elasticity ⚙️: No manual operation required to adjust resource group size.

Future enhancements include proactive elasticity, load decoupling, elasticity efficiency improvements, and performance visualization. 🔍

Embrace the power of intelligent scaling with AnalyticDB for MySQL, and unlock new levels of performance and cost efficiency for your data-driven operations. 🌟