AWS, Cloud Computing

5 Mins Read

A Guide to Create AWS MSK Cluster with IAM-Based Authentication

Introduction

Amazon MSK (Managed Streaming for Kafka) is a fully managed AWS service. It helps build and run the applications based on Apache Kafka for processing and streaming the data. AWS MSK manages the configuration, maintenance, and provisioning of the Apache Zookeeper nodes and clusters.

AWS MSK provides a fully-managed Kafka cluster, eliminating the need to manage and maintain your infrastructure. With AWS MSK, you can scale your Kafka clusters to meet your needs, with the ability to add or remove brokers as needed.

Features and Benefits

Fully Managed: AWS MSK is a fully managed service, which means that AWS manages the infrastructure, security, and availability of your Kafka cluster. This lets you focus on building your applications and processing your data without worrying about the underlying infrastructure.

High Availability and Durability: AWS MSK is designed to be highly available and durable. It uses Apache Zookeeper to ensure the cluster is always available, even if individual brokers fail. Additionally, data is replicated across multiple brokers to ensure durability.

Elastic Scalability: AWS MSK allows you to quickly scale your Kafka cluster up or down as needed. You can add or remove brokers as needed, and the cluster will automatically rebalance data to ensure that each broker has an even share of the workload.

Secure: AWS MSK provides several security features to help you secure your Kafka cluster. This includes support for rest and transit encryption and integration with AWS Identity and Access Management (IAM) to control access to your resources.

  • Cloud Migration
  • Devops
  • AIML & IoT
Know More

Prerequisite

  • AWS account with permissions to create resources in Amazon MSK
  • Amazon VPC
  • Amazon EC2

Getting Started with AWS MSK

To get started with AWS MSK, you can use the AWS Management Console, AWS CLI, or AWS SDKs. You can create a new cluster or migrate an existing Kafka cluster to AWS MSK. Once your cluster runs, you can use the Kafka protocol to produce and consume data from your applications.

Step-by-Step Guide

Creating MSK Cluster

Go to MSK Dashboard and click on create the cluster.

Step 1: Configuring cluster setting

Select custom create, give your cluster a name, select provisioned and select the Apache Kafka version you want to use.

step1

Select the broker type, number of nodes, and broker per zone according to your requirement.

step1b

Give the storage per broker and click next.

step1c

Step 2: Configuring Networking

Select the VPC Subnet (here, private subnets have been selected), create a new security group for the cluster, choose that security group, and click next.

step2

Step 3: Configuring Security settings

In this, select the authentication methods which are

  1. Unauthenticated access
  2. IAM role-based authentication
  3. SASL/SCRAM authentication
  4. TLS client authentication through AWS Certificate Manager (ACM

In this demo, we are showing it with “IAM role-based authentication”. By default, AWS managed key will be used for encryption, but we can also use the customer managed key. Click next.

step3

Step 4: Configure Monitoring

Select the CloudWatch metric accordingly for the monitoring of the broker, and we can deliver the logs in CloudWatch Log groups or Amazon S3 bucket.

Add tags if required and click next.

step4

Step 5: Review the setup

Review the whole configuration and click on “create cluster”.

Cluster configuration will take approximately 20-25 min.

Once cluster status is in Active state, click “view client information” and copy the Bootstrap server’s private endpoint.

step5

step5b

Bastion Setup

We will need a running EC2 instance in the same VPC to test our MSK cluster. To give EC2 instance the connectivity of MSK, a role must be attached with the permissions of the MSK cluster. Add an inbound rule on the MSK cluster’s security group allowing the traffic from the EC2 instance security group. Once the EC2 instance is ready, ssh into it and follow the below step to configure it.

  • Install java – “sudo yum install java-1.8.0”
  • Download the Kafka and unzip it –
  • Download aws-msk-iam-auth file to use for IAM authentication –
  • Create “client.properties” file and add the below data to it

bastion

  • Before running the Kafka command, we need to export the aws-msk-iam-auth file –

Note: Download the same version of Kafka selected in the MSK cluster.

Once ec2 is ready and configured, let’s start testing.

Testing the IAM-based Authentication

Firstly, we will create a topic, below is the command to create a topic:

After running the above command, it will show “Created topic <your topic name>”. Now we will produce and consume messages from the Kafka cluster. Use the below command to produce message

Use the below command to consume message

Replace <your broker endpoint> with the list of brokers for your Amazon MSK cluster, and replace <your topic name> with the name of your Kafka topic. You can find the list of brokers in the Amazon MSK console.

Conclusion

AWS MSK is a fully managed service that makes it easy to build and run Apache Kafka clusters in the cloud. With its elastic scalability, high availability, and integration with other AWS services, AWS MSK is a powerful tool for building real-time streaming data pipelines. Whether building a new application or migrating an existing Kafka cluster, AWS MSK can help you simplify your infrastructure and focus on building your applications.

Get your new hires billable within 1-60 days. Experience our Capability Development Framework today.

  • Cloud Training
  • Customized Training
  • Experiential Learning
Read More

About CloudThat

CloudThat is also the official AWS (Amazon Web Services) Advanced Consulting Partner and Training partner and Microsoft gold partner, helping people develop knowledge of the cloud and help their businesses aim for higher goals using best in industry cloud computing practices and expertise. We are on a mission to build a robust cloud computing ecosystem by disseminating knowledge on technological intricacies within the cloud space. Our blogs, webinars, case studies, and white papers enable all the stakeholders in the cloud computing sphere.

Drop a query if you have any questions regarding Amazon MSK and I will get back to you quickly.

To get started, go through our Consultancy page and Managed Services Package that is CloudThat’s offerings.

FAQs

1. What is the pricing model for Amazon MSK?

ANS: – Amazon MSK uses a pay-as-you-go pricing model, where you are charged based on the number of Apache Kafka brokers, the amount of storage used, and the amount of data transferred in and out of the cluster.

2. Can I scale up storage in my cluster?

ANS: – You can scale up storage in your provisioned clusters using the AWS Management Console or the AWS CLI. You can also create an auto-scaling storage policy to expand cluster storage automatically.

3. What does the Amazon MSK SLA guarantee?

ANS: – Amazon MSK SLA guarantees a Monthly Uptime Percentage of at least 99.9% for Amazon MSK (not applicable to MSK Serverless).

WRITTEN BY Aniket Kumar Ambasta

Aniket Kumar Ambasta works as a Research associate- TC - Infra, Security, and Migration at CloudThat. He is AWS Solutions Architect- Associate certified and has completed his Bachelor's in Computer Applications. He has good experience in Cloud technologies. Apart from professional interests, he loves exploring and learning new technologies.

Share

Comments

    Click to Comment

Get The Most Out Of Us

Our support doesn't end here. We have monthly newsletters, study guides, practice questions, and more to assist you in upgrading your cloud career. Subscribe to get them all!