Business Use Case:

A company needs a centralized platform to enable end-to-end data discovery, observability, and governance. Deploying DataHub on Amazon EKS facilitates comprehensive metadata management, improving data accessibility and compliance.

Overview:

This guide outlines deploying DataHub on an Amazon EKS cluster, leveraging services like Amazon OpenSearch, Amazon MSK, and Amazon RDS for MySQL. It uses Kubernetes for scalable deployment and management of DataHub components.

Detailed Steps:

1. Infrastructure Setup:

• VPC and Subnets: Create a new VPC with public and private subnets or use an existing VPC.

• EKS Cluster: Set up an EKS cluster with managed node groups for different workloads.

2. Storage and Messaging Services:

• Amazon MSK: Use Amazon Managed Streaming for Apache Kafka for metadata ingestion.

• Amazon RDS: Deploy Amazon RDS for MySQL for relational database storage.

• Amazon OpenSearch: Utilize Amazon OpenSearch for scalable search capabilities.

3. DataHub Deployment:

• Helm Charts: Deploy DataHub using Helm charts, setting up pods and services on the EKS cluster.

• Ingress Configuration: Enable Ingress for the DataHub frontend UI, provisioned by AWS LoadBalancer Controller.

4. Monitoring and Scaling:

Metrics and Autoscaling: Deploy Metrics Server, Cluster Autoscaler, and Prometheus for monitoring and scaling the cluster.

Business Value:

• Scalability: Kubernetes ensures efficient scaling and management of DataHub components.

• Efficiency: Managed AWS services simplify deployment and operation, reducing administrative overhead.

• Data Governance: Centralized metadata management enhances data governance and compliance.