A Guide to Create AWS MSK Cluster with IAM-Based Authentication

Introduction

Amazon MSK (Managed Streaming for Kafka) is a fully managed AWS service. It helps build and run the applications based on Apache Kafka for processing and streaming the data. AWS MSK manages the configuration, maintenance, and provisioning of the Apache Zookeeper nodes and clusters.

AWS MSK provides a fully-managed Kafka cluster, eliminating the need to manage and maintain your infrastructure. With AWS MSK, you can scale your Kafka clusters to meet your needs, with the ability to add or remove brokers as needed.

Features and Benefits

Fully Managed: AWS MSK is a fully managed service, which means that AWS manages the infrastructure, security, and availability of your Kafka cluster. This lets you focus on building your applications and processing your data without worrying about the underlying infrastructure.

High Availability and Durability: AWS MSK is designed to be highly available and durable. It uses Apache Zookeeper to ensure the cluster is always available, even if individual brokers fail. Additionally, data is replicated across multiple brokers to ensure durability.

Elastic Scalability: AWS MSK allows you to quickly scale your Kafka cluster up or down as needed. You can add or remove brokers as needed, and the cluster will automatically rebalance data to ensure that each broker has an even share of the workload.

Secure: AWS MSK provides several security features to help you secure your Kafka cluster. This includes support for rest and transit encryption and integration with AWS Identity and Access Management (IAM) to control access to your resources.

Prerequisite

• AWS account with permissions to create resources in Amazon MSK
• Amazon VPC
• Amazon EC2

Getting Started with AWS MSK

To get started with AWS MSK, you can use the AWS Management Console, AWS CLI, or AWS SDKs. You can create a new cluster or migrate an existing Kafka cluster to AWS MSK. Once your cluster runs, you can use the Kafka protocol to produce and consume data from your applications.

Step-by-Step Guide

Creating MSK Cluster

Go to MSK Dashboard and click on create the cluster.

Step 1: Configuring cluster setting

Select custom create, give your cluster a name, select provisioned and select the Apache Kafka version you want to use.

Select the broker type, number of nodes, and broker per zone according to your requirement.

Give the storage per broker and click next.

Step 2: Configuring Networking

Select the VPC Subnet (here, private subnets have been selected), create a new security group for the cluster, choose that security group, and click next.

Step 3: Configuring Security settings

In this, select the authentication methods which are

1. Unauthenticated access
2. IAM role-based authentication
3. SASL/SCRAM authentication
4. TLS client authentication through AWS Certificate Manager (ACM

In this demo, we are showing it with “IAM role-based authentication”. By default, AWS managed key will be used for encryption, but we can also use the customer managed key. Click next.

Step 4: Configure Monitoring

Select the CloudWatch metric accordingly for the monitoring of the broker, and we can deliver the logs in CloudWatch Log groups or Amazon S3 bucket.

Add tags if required and click next.

Step 5: Review the setup

Review the whole configuration and click on “create cluster”.

Cluster configuration will take approximately 20-25 min.

Once cluster status is in Active state, click “view client information” and copy the Bootstrap server’s private endpoint.

Bastion Setup

We will need a running EC2 instance in the same VPC to test our MSK cluster. To give EC2 instance the connectivity of MSK, a role must be attached with the permissions of the MSK cluster. Add an inbound rule on the MSK cluster’s security group allowing the traffic from the EC2 instance security group. Once the EC2 instance is ready, ssh into it and follow the below step to configure it.

• Install java – “sudo yum install java-1.8.0”
• Download the Kafka and unzip it –

• Download aws-msk-iam-auth file to use for IAM authentication –

• Create “client.properties” file and add the below data to it

• Before running the Kafka command, we need to export the aws-msk-iam-auth file –

Note: Download the same version of Kafka selected in the MSK cluster.

Once ec2 is ready and configured, let’s start testing.

Testing the IAM-based Authentication

Firstly, we will create a topic, below is the command to create a topic:

After running the above command, it will show “Created topic <your topic name>”. Now we will produce and consume messages from the Kafka cluster. Use the below command to produce message

Use the below command to consume message

Replace <your broker endpoint> with the list of brokers for your Amazon MSK cluster, and replace <your topic name> with the name of your Kafka topic. You can find the list of brokers in the Amazon MSK console.

Conclusion

AWS MSK is a fully managed service that makes it easy to build and run Apache Kafka clusters in the cloud. With its elastic scalability, high availability, and integration with other AWS services, AWS MSK is a powerful tool for building real-time streaming data pipelines. Whether building a new application or migrating an existing Kafka cluster, AWS MSK can help you simplify your infrastructure and focus on building your applications.

About CloudThat