This article unveils the journey of crafting a multi-tenant big data platform on Alibaba Cloud’s MaxCompute, focusing on seamless cross-tenant data access and governance. We’ll explore how a leading Southeast Asian tech group, GoTerra, transitioned from Google BigQuery to MaxCompute, overcoming challenges with innovative solutions. 🚀
🎯 Why Choose a Multi-Tenant Architecture?
GoTerra, a trailblazer in Southeast Asia’s tech landscape, initially relied on Google BigQuery for its data warehousing needs. However, their need for scalability and strict compliance requirements demanded a multi-tenant approach, where each business entity operated under separate cloud accounts for resource management, usage tracking, and billing.
This setup introduced complexities, including:
- Cross-tenant resource access: Enabling secure data sharing across accounts.
- Unified governance: Centralizing oversight without compromising compliance.
- Cost management: Streamlining expenses across multiple tenants.
- Avoiding infrastructure redundancy: Centralizing deployment while maintaining isolation.
To tackle these, GoTerra leveraged Google Cloud Platform’s (GCP) multi-tenant features and BigQuery’s cross-tenant data access capabilities. Their architecture featured:
- Shared Control Plane 🛠️: Centralized services like task scheduling, metadata management, data quality monitoring, and access control, managed by GoTerra’s data engineering team with costs distributed across entities.
- Isolated Data Planes with Cross-Tenant Access 🔒: Each business entity operated its own data plane, powered by BigQuery’s ingestion, computing, and storage capabilities, with controlled cross-tenant data access facilitated by the control plane.
🛠️ MaxCompute’s Multi-Tenant Capabilities
When GoTerra decided to migrate to Alibaba Cloud’s MaxCompute, the goal was to replicate this robust architecture while addressing unique challenges. Let’s break down MaxCompute’s multi-tenant features:
| Feature | Description |
|---|---|
| Multi-Tenant Support | MaxCompute supports multiple tenants within a single cluster or region, each with distinct projects, quotas, and network connections. |
| Project Management | Projects handle data objects, job instances, and user roles, ensuring organized resource allocation. |
| Resource Quotas | Quotas manage computing resources, balancing performance across tenants. |
| Cross-Tenant Access | Controlled data sharing via cross-project access and proxy authorization. |
While MaxCompute excels at isolation and security, its traditional focus hasn’t fully addressed complex cross-account scenarios for large enterprises like GoTerra. Unlike BigQuery’s Data Mesh model, MaxCompute required enhancements to support flexible cross-tenant data access and governance.
🚀 Crafting a Multi-Tenant Big Data Platform on MaxCompute
To mirror GoTerra’s GCP-based architecture on Alibaba Cloud, we focused on three critical capabilities:
1. Cross-Tenant Data Access 🔄
MaxCompute offers a unified namespace similar to BigQuery’s [project_name].[dataset_name].[object_name], using [project_name].[schema_name].[object_name]. This structure simplifies data access across tenants. Existing data access object management capabilities in MaxCompute made this transition seamless.
2. Cross-Tenant Data Access Control 🔐
MaxCompute’s original cross-tenant access control relied on:
- Package-based cross-project access: Lacked fine-grained control.
- Alibaba Cloud account proxy authorization: Limited flexibility.
To address these gaps, we enhanced MaxCompute with:
- Granular ACL and policy-based authorization: Providing precise control over data access.
- Flexible proxy authorization: Enabling dynamic access management for cross-tenant scenarios.
These upgrades, developed for GoTerra, are now available to all MaxCompute users on Alibaba Cloud.
3. Cross-Tenant Metadata Discovery 🔍
Effective data utilization requires metadata access across tenants. While MaxCompute technically supports shared metadata, its console and DataWorks previously limited discovery to single-tenant views. New enhancements now enable cross-tenant metadata discovery, empowering users to explore and process data efficiently.
Implementation with Alibaba Cloud RAM
To enable cross-account resource access, we utilized Alibaba Cloud Resource Access Management (RAM):
- Control Plane Account: Created a central account with a RAM role (
dataplatform_controlplane) that cloud services like Elastic Compute Service (ECS) and Container Service for Kubernetes (ACK) can assume. - Data Plane Accounts: Each tenant account has a RAM role allowing the control plane to access its MaxCompute resources.
- Cross-Account Authorization: ECS and ACK services assume the control plane’s RAM role to access data plane resources securely.
This architecture mirrors GoTerra’s GCP setup, ensuring compliance, scalability, and efficiency.
Control Plane Account
└── RAM Role: dataplatform_controlplane
├── Assumed by: ECS, ACK
└── Accesses: Data Plane RAM Roles
Data Plane Account(s)
└── RAM Role: Grants access to MaxCompute resources
└── Assumed by: Control Plane Account
🌟 Future Horizons for Multi-Tenancy
GoTerra currently uses a self-managed suite of big data tools for its control plane, replicating DataWorks capabilities. However, many enterprises prefer a fully managed control plane. To meet this demand, DataWorks is evolving with:
- Flexible multi-tenant data source management 📊
- Cross-tenant metadata discovery for DataAnalysis
- Enhanced DataStudio for multi-tenant data source management
Some features are already live, with others planned for future releases, promising to elevate DataWorks’ enterprise-grade multi-tenant capabilities.
🎉 Conclusion
By leveraging MaxCompute’s enhanced multi-tenant features and Alibaba Cloud’s RAM, GoTerra successfully migrated its big data platform from GCP to Alibaba Cloud. This journey showcases how strategic architecture and innovative enhancements can address complex cross-tenant challenges, delivering a scalable, secure, and compliant big data solution. 🌍
For more details on MaxCompute and Alibaba Cloud’s capabilities, visit Alibaba Cloud MaxCompute and Resource Access Management.
