AWS, Cloud Computing

6 Mins Read

Generating Logs and Metrics of Kubernetes Components (EKS Cluster) using EFK Stack

Overview

This blog is about getting logs and metrics from EKS Cluster and displaying them in the Kibana Dashboard by setting up the EFK (ElasticSearch, Fluentbit, and Kibana). The setup mainly concentrates on the Kubernetes cluster, which is created with the help of AWS EKS and monitoring and generating logs and metrics of the Kubernetes cluster components. EFK Stack and Metric Beat are used to generate logs and metrics, respectively.

Introduction

EFK stack is most popular for log aggregation and management and getting metrics from Kubernetes. The ‘F’ in the EFK stack usually refers to Fluentd, the former version of Fluent bit. We have chosen Fluent bit because it is a lightweight service suitable for basic log management and metric generation use case.

Usually, we can get logs in the Kubernetes Dashboard/terminal itself, but it will get generated with the command kubectl logs. But it is likely got only one pod logs at a time. It will be most difficult to get the logs of all pods. EFK Stack will get all the metrics and logs integration at a time.

Intro

Pioneers in Cloud Consulting & Migration Services

  • Reduced infrastructural costs
  • Accelerated application deployment
Get Started

Steps for Creating of EKS Cluster

  1. Create IAM Role for EKS Cluster with AmazonEKSClusterPolicy and name it as
  1. Create IAM Role for Worker node (EKSNG_Role) with AmazonEKSWorkerNodePolicy, AmazonEC2ContainerRegistryReadOnly, AmazonEKS_CNI_Policy.
  2. After that, go to the search bar and search for EKS, then you will get the landing page of EKS Cluster.
  3. Then click on Add cluster button and choose to create option.
  4. Give the cluster name (EKS_POC) and attach the role created earlier, i.e., EKSC_Role and choose Next.

step5

6. You need to Select VPC and Subnet selections in this Section and Choose the Cluster End Point Access, where you need to access your cluster and choose Next.

step6

7. No need to enable logging, as we will be using other tools for logging. Click on Next.

step7

8. Then Review and create.

step8

9. The cluster will get created.

step9

10. Then need to create a Node Group. For that, navigate to Compute tab, as shown in the above screenshot.

step10

11. Choose Add Node Group.

12. Enter the Node Group name and the role (EKSNG_Role) for the node group created earlier. And select Next.

step12

13. Select the AMI type, Instance Type of your choice, and capacity type as OnDemand. Select the no. of nodes you need in the Node Group Scaling configuration.

step13

14. Specify the Network configuration in which subnets the Nodes are to be launched, and if needed, you can go with enable the SSH section. For that, you need to create a KeyPair in EC2.

step14

15. Then Review and Create.

step15

step15b

16. We need to update kubeconfig file on the host machine

17. We need to set up Metric Server and cAdvisor for monitoring clusters.

cAdvisor Setup:

EFK Stack Setup

18. Install elastic search on the cluster using helm repository (Need to install helm)

19. Install Kibana on the cluster using helm

20. Make both Kibana and ElasticSearch service as Loadbalancer

step20

MetricBeat Setup

21. Install MetricBeat Server (Need to install MetricBeat after Kubestate metrics server and Kibana got installed)

step21

22. After the Successful installation of MetricBeat, check with the cluster once.

step22

23. We need to install dashboards in Metricbeat pods.

step23

24. Copy the Service External-IP of both Kibana and Elastic Search and paste it in the browser as:

<External-IP>:Port

25. Elastic-Search dashboard

<External-IP>:Port/_cat/indic

step25

26. Kibana Dashboard with elastic search logs:

To get logs in Kibana, go to Stack Management, navigate to Kibana, click on Index Patterns, and Select Index Pattern from the list where you want to get logs and place * at the end. Choose @timestamp for getting logs based on time.

Afterward, go to Discover, and the Index logs will be displayed below.

step26

27. Kibana Dashboard with MetricBeat

  • To display the Metrics of Pods and Nodes, we need to customize the Kibana dashboard.
  • In the Navigation pane in the Observability section, select the Metrics tab, and you will navigate to the Kibana dashboard configured with Metricbeat Metrics there, you can see the logs of the Kubernetes cluster.

step27

Metrics of Elastic_Search_POD

step27b

Metrics of Own_POD

Conclusion

So, finally, to get logs and metrics from the Kubernetes cluster, we have successfully set up the EFK stack. As mentioned earlier, the EFK stack refers to Elasticsearch, Fluent Bit, and Kibana. The process is completely done based on Elasticsearch, FluentBit, and Kibana. In addition, Metricbeat is used to get the metrics and make them observable in the observability section of the Kibana dashboard.

Get your new hires billable within 1-60 days. Experience our Capability Development Framework today.

  • Cloud Training
  • Customized Training
  • Experiential Learning
Read More

About CloudThat

CloudThat is also the official AWS (Amazon Web Services) Advanced Consulting Partner and Training partner and Microsoft gold partner, helping people develop knowledge of the cloud and help their businesses aim for higher goals using best in industry cloud computing practices and expertise. We are on a mission to build a robust cloud computing ecosystem by disseminating knowledge on technological intricacies within the cloud space. Our blogs, webinars, case studies, and white papers enable all the stakeholders in the cloud computing sphere.

Drop a query if you have any questions regarding Kubernetes, EKF Stack and I will get back to you quickly.

To get started, go through our Consultancy page and Managed Services Package that is CloudThat’s offerings.

FAQs

1. What is an Elasticsearch index?

ANS: – An Elasticsearch index is a group of documents mostly related among them. Elasticsearch usually stores the data in the format of JSON documents. The document is associated with a set of keys usually having the properties or  names of fields, probably having matching values like Booleans, numbers, arrays of values, geolocations, dates, strings, etc.

2. What is Kibana used for?

ANS: – Kibana is a data management and visualization tool for Elasticsearch that offers real-time maps, line graphs, pie charts, and histograms. Kibana also provides advanced tools like Elastic Maps for visualizing geographical data and Canvas, which lets users build unique dynamic infographics based on their data.

3. Is Elasticsearch free?

ANS: – Elasticsearch is free, and open features are freely used under the SSPL or the Elastic License. The Elastic License offers additional free services, and paying subscriptions grant access to support and more sophisticated capabilities like alerts and machine learning.

4. What is Metricbeat used for?

ANS: – Installing Metricbeat, a lightweight shipper will let you regularly gather metrics from the servers operating system and active services.

WRITTEN BY Bhanu Prakash K

K Bhanu Prakash is working as a Subject Matter Expert in CloudThat. He is proficient in Managing and configuring AWS Infrastructure as well as on Kubernetes and DevOps tools like Terraform, ansible, Jenkins, and Git. He is very keen on learning new technologies and publishing blogs for the tech community.

Share

Comments

    Click to Comment

Get The Most Out Of Us

Our support doesn't end here. We have monthly newsletters, study guides, practice questions, and more to assist you in upgrading your cloud career. Subscribe to get them all!