Login
March 6, 2023|
Voiced by Amazon Polly |
Introduction
Amazon MSK (Managed Streaming for Kafka) is a fully managed AWS service. It helps build and run the applications based on Apache Kafka for processing and streaming the data. AWS MSK manages the configuration, maintenance, and provisioning of the Apache Zookeeper nodes and clusters.
AWS MSK provides a fully-managed Kafka cluster, eliminating the need to manage and maintain your infrastructure. With AWS MSK, you can scale your Kafka clusters to meet your needs, with the ability to add or remove brokers as needed.
Features and Benefits
Fully Managed: AWS MSK is a fully managed service, which means that AWS manages the infrastructure, security, and availability of your Kafka cluster. This lets you focus on building your applications and processing your data without worrying about the underlying infrastructure.
High Availability and Durability: AWS MSK is designed to be highly available and durable. It uses Apache Zookeeper to ensure the cluster is always available, even if individual brokers fail. Additionally, data is replicated across multiple brokers to ensure durability.
Elastic Scalability: AWS MSK allows you to quickly scale your Kafka cluster up or down as needed. You can add or remove brokers as needed, and the cluster will automatically rebalance data to ensure that each broker has an even share of the workload.
Secure: AWS MSK provides several security features to help you secure your Kafka cluster. This includes support for rest and transit encryption and integration with AWS Identity and Access Management (IAM) to control access to your resources.
Helping organizations transform their IT infrastructure with top-notch Cloud Computing services
- Cloud Migration
- Devops
- AIML & IoT
Prerequisite
- AWS account with permissions to create resources in Amazon MSK
- Amazon VPC
- Amazon EC2
Getting Started with AWS MSK
To get started with AWS MSK, you can use the AWS Management Console, AWS CLI, or AWS SDKs. You can create a new cluster or migrate an existing Kafka cluster to AWS MSK. Once your cluster runs, you can use the Kafka protocol to produce and consume data from your applications.
Step-by-Step Guide
Creating MSK Cluster
Go to MSK Dashboard and click on create the cluster.
Step 1: Configuring cluster setting
Select custom create, give your cluster a name, select provisioned and select the Apache Kafka version you want to use.
Select the broker type, number of nodes, and broker per zone according to your requirement.
Give the storage per broker and click next.
Step 2: Configuring Networking
Select the VPC Subnet (here, private subnets have been selected), create a new security group for the cluster, choose that security group, and click next.
Step 3: Configuring Security settings
In this, select the authentication methods which are
- Unauthenticated access
- IAM role-based authentication
- SASL/SCRAM authentication
- TLS client authentication through AWS Certificate Manager (ACM
In this demo, we are showing it with “IAM role-based authentication”. By default, AWS managed key will be used for encryption, but we can also use the customer managed key. Click next.
Step 4: Configure Monitoring
Select the CloudWatch metric accordingly for the monitoring of the broker, and we can deliver the logs in CloudWatch Log groups or Amazon S3 bucket.
Add tags if required and click next.
Step 5: Review the setup
Review the whole configuration and click on “create cluster”.
Cluster configuration will take approximately 20-25 min.
Once cluster status is in Active state, click “view client information” and copy the Bootstrap server’s private endpoint.
Bastion Setup
We will need a running EC2 instance in the same VPC to test our MSK cluster. To give EC2 instance the connectivity of MSK, a role must be attached with the permissions of the MSK cluster. Add an inbound rule on the MSK cluster’s security group allowing the traffic from the EC2 instance security group. Once the EC2 instance is ready, ssh into it and follow the below step to configure it.
- Install java – “sudo yum install java-1.8.0”
- Download the Kafka and unzip it –
|
1 |
wget https://archive.apache.org/dist/kafka/2.8.1/kafka_2.12-2.8.1.tgz |
- Download aws-msk-iam-auth file to use for IAM authentication –
|
1 |
sudo wget https://github.com/aws/aws-msk-iam-auth/releases/download/v1.1.5/aws-msk-iam-auth-1.1.5-all.jar |
- Create “client.properties” file and add the below data to it
- Before running the Kafka command, we need to export the aws-msk-iam-auth file –
|
1 |
export CLASSPATH=<directory where file is>/aws-msk-iam-auth-1.1.5-all.jar |
Note: Download the same version of Kafka selected in the MSK cluster.
Once ec2 is ready and configured, let’s start testing.
Testing the IAM-based Authentication
Firstly, we will create a topic, below is the command to create a topic:
|
1 |
<directory of kafka>/kafka_2.12-2.8.1/bin/kafka-topics.sh --bootstrap-server <your broker endpoint> --create --replication-factor 3 --partitions 3 --topic <your topic name> --command-config client.properties |
After running the above command, it will show “Created topic <your topic name>”. Now we will produce and consume messages from the Kafka cluster. Use the below command to produce message
|
1 |
<directory of kafka>/kafka_2.12-2.8.1/bin/kafka-console-producer.sh --broker-list <your broker endpoint> --topic <your topic name> --producer.config client.properties |
Use the below command to consume message
|
1 |
<directory of kafka>/kafka_2.12-2.8.1/bin/kafka-console-consumer.sh --bootstrap-server <your broker endpoint> --topic <your topic name> --from-beginning --consumer.config client.properties |
Replace <your broker endpoint> with the list of brokers for your Amazon MSK cluster, and replace <your topic name> with the name of your Kafka topic. You can find the list of brokers in the Amazon MSK console.
Conclusion
AWS MSK is a fully managed service that makes it easy to build and run Apache Kafka clusters in the cloud. With its elastic scalability, high availability, and integration with other AWS services, AWS MSK is a powerful tool for building real-time streaming data pipelines. Whether building a new application or migrating an existing Kafka cluster, AWS MSK can help you simplify your infrastructure and focus on building your applications.
Get your new hires billable within 1-60 days. Experience our Capability Development Framework today.
- Cloud Training
- Customized Training
- Experiential Learning
About CloudThat
CloudThat is also the official AWS (Amazon Web Services) Advanced Consulting Partner and Training partner and Microsoft gold partner, helping people develop knowledge of the cloud and help their businesses aim for higher goals using best in industry cloud computing practices and expertise. We are on a mission to build a robust cloud computing ecosystem by disseminating knowledge on technological intricacies within the cloud space. Our blogs, webinars, case studies, and white papers enable all the stakeholders in the cloud computing sphere.
Drop a query if you have any questions regarding Amazon MSK and I will get back to you quickly.
To get started, go through our Consultancy page and Managed Services Package that is CloudThat’s offerings.
FAQs
1. What is the pricing model for Amazon MSK?
ANS: – Amazon MSK uses a pay-as-you-go pricing model, where you are charged based on the number of Apache Kafka brokers, the amount of storage used, and the amount of data transferred in and out of the cluster.
2. Can I scale up storage in my cluster?
ANS: – You can scale up storage in your provisioned clusters using the AWS Management Console or the AWS CLI. You can also create an auto-scaling storage policy to expand cluster storage automatically.
3. What does the Amazon MSK SLA guarantee?
ANS: – Amazon MSK SLA guarantees a Monthly Uptime Percentage of at least 99.9% for Amazon MSK (not applicable to MSK Serverless).
WRITTEN BY Aniket Kumar Ambasta
Aniket Kumar Ambasta works as a Research associate- TC - Infra, Security, and Migration at CloudThat. He is AWS Solutions Architect- Associate certified and has completed his Bachelor's in Computer Applications. He has good experience in Cloud technologies. Apart from professional interests, he loves exploring and learning new technologies.
PREV
Comments