Achieving End-to-End Observability with Jaeger for Distributed Tracing

Overview

Understanding the flow of requests across microservices in distributed systems is essential for identifying bottlenecks and optimizing application performance. Jaeger, an open-source tracing system, enables developers to monitor and troubleshoot distributed applications effectively. When deployed on Amazon Elastic Kubernetes Service (EKS), Jaeger provides a scalable and robust tracing solution integrated into a cloud-native ecosystem.

Pioneers in Cloud Consulting & Migration Services

Reduced infrastructural costs
Accelerated application deployment

Get Started

Introduction

Jaeger is an open-source, end-to-end distributed tracing system. It is a Cloud Native Computing Foundation (CNCF) project designed to help with:

Distributed context propagation: Track requests across multiple services.
Root cause analysis: Identify bottlenecks and performance issues.
Service dependency analysis: Visualize dependencies between services.
Performance optimization: Understand and improve request latencies.

By providing visibility into service interactions and latencies, Jaeger empowers teams to troubleshoot complex distributed systems and enhance reliability and performance.

Prerequisites

Amazon EKS Cluster: Ensure you have an Amazon EKS cluster up and running.
kubectl: Installed and configured to access the Amazon EKS cluster. Test the connection by running kubectl get nodes to ensure kubectl can communicate with the cluster.
Helm: Installed for deploying Jaeger components. Confirm the Helm version with helm version and update if necessary to avoid compatibility issues.
Application Instrumentation: Ensure your application is instrumented with a tracing library compatible with Jaeger, such as OpenTelemetry SDKs. Confirm that the instrumentation setup is sending spans and traces.
AWS IAM Roles and Permissions: If you use an AWS IAM role for your Kubernetes service account, ensure it has permissions to create and manage resources, including PersistentVolumes, in your Amazon EKS cluster.
Storage Class: Confirm that a storage class is available for persistent data storage, especially when using production-grade setups like Elasticsearch or Badger.
Network Configuration: Validate that no network policies or firewalls block communication between Jaeger components or between your application and Jaeger.

Steps to Deploy Jaeger on Amazon EKS

Step 1: Set Up Namespace

Create a dedicated namespace for Jaeger to organize its resources.
kubectl create namespace monitoring

Step 2: Add Helm Repository

Add the official Jaeger Helm repository.

helm repo add jaeger-all-in-one https://raw.githubusercontent.com/hansehe/jaeger-all-in-one/master/helm/charts

1	helm repo add jaeger-all-in-one https://raw.githubusercontent.com/hansehe/jaeger-all-in-one/master/helm/charts

Step 3: Install Jaeger

Deploy Jaeger using Helm. You can choose the all-in-one setup for testing or a production setup with distributed components.

helm install jaeger-all-in-one jaegertracing/jaeger --namespace monitoring \
  --set collector.persistence.enabled=true \
  --set storage.type=badger \
  --set storage.badger.directoryValue=/data/jaeger/badger \
  --set persistence.storageClass=standard \
  --set persistence.size=10Gi


helm upgrade jaeger-all-in-one jaegertracing/jaeger --namespace monitoring \
  --set collector.grpcEnabled=false \
  --set collector.httpEnabled=true \
  --set agent.reporter.type=http \
  --set agent.reporter.endpoint=http://jaeger-all-in-one-collector.monitoring.svc:14268/api/traces

helm install jaeger-all-in-one jaegertracing/jaeger --namespace monitoring \

--set collector.persistence.enabled=true \

--set storage.type=badger \

--set storage.badger.directoryValue=/data/jaeger/badger \

--set persistence.storageClass=standard \

--set persistence.size=10Gi

helm upgrade jaeger-all-in-one jaegertracing/jaeger --namespace monitoring \

--set collector.grpcEnabled=false \

--set collector.httpEnabled=true \

--set agent.reporter.type=http \

--set agent.reporter.endpoint=http://jaeger-all-in-one-collector.monitoring.svc:14268/api/traces

Step 4: Verify Deployment

Check that Jaeger components are running:

Kubectl get po –n monitoring

1	Kubectl get po –n monitoring

Step 5: Expose Jaeger Query Service Using Ingress

Access the Jaeger UI via an Ingress resource to make it accessible externally:
Create the Ingress YAML file

apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
   annotations:
     alb.ingress.kubernetes.io/certificate-arn: 
    alb.ingress.kubernetes.io/group.name: monitoring
     alb.ingress.kubernetes.io/group.order: "10"
     alb.ingress.kubernetes.io/scheme: internal
     alb.ingress.kubernetes.io/target-type: ip
   finalizers:
   - group.ingress.k8s.aws/monitoring
   generation: 1
   name: jaeger-ingress
   namespace: monitoring
 spec:
   ingressClassName: alb
   rules:
   - host: jaeger.domain.com 
    http:
       paths:
       - backend:
           service:
             name: jaeger-all-in-one-query
             port:
               number: 80
         path: /
         pathType: Prefix

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

annotations:

alb.ingress.kubernetes.io/certificate-arn:

alb.ingress.kubernetes.io/group.name: monitoring

alb.ingress.kubernetes.io/group.order: "10"

alb.ingress.kubernetes.io/scheme: internal

alb.ingress.kubernetes.io/target-type: ip

finalizers:

- group.ingress.k8s.aws/monitoring

generation: 1

name: jaeger-ingress

namespace: monitoring

spec:

ingressClassName: alb

rules:

- host: jaeger.domain.com

http:

paths:

- backend:

service:

name: jaeger-all-in-one-query

port:

number: 80

path: /

pathType: Prefix

Apply the Ingress:

kubectl apply -f jaeger-ingress.yaml

1	kubectl apply -f jaeger-ingress.yaml

Verify the Ingress:

kubectl get ingress -n monitoring

1	kubectl get ingress -n monitoring

Step 6: Instrument Your Applications

To send traces from your application to Jaeger, update your deployment file to configure the tracing settings.

Update the Application Deployment: Add environment variables or sidecars to route traces to the Jaeger agent or collector.
Verify Instrumentation: Ensure your application emits traces of the specified Jaeger endpoint. You can test this by generating traffic and observing Jaeger UI traces.

step6

Step 7: Key Guidelines

Use Distributed Setup for Production: A distributed setup of Jaeger components is preferred for production environments to handle higher traffic and ensure fault tolerance.
Leverage OpenTelemetry: Use OpenTelemetry SDKs to instrument applications for broader compatibility and easier integration with other observability tools.
Monitor Jaeger Performance: Continuously monitor the resource utilization of Jaeger components to optimize storage and processing capacity.
Secure Jaeger Endpoints: Restrict access to Jaeger’s Query and Collector services using network policies and authentication mechanisms.
Backup Tracing Data: Regularly backup the underlying storage (e.g., Badger or Elasticsearch) to prevent data loss in case of failures.
Optimize Storage Configuration: Configure your storage backend (e.g., Elasticsearch or Cassandra) with sufficient capacity and scalability for production.
Establish Retention Policies: Define retention policies for tracing data to balance between storage costs and troubleshooting needs.

Conclusion

By deploying Jaeger, we gain a scalable tracing solution to monitor and optimize your distributed applications.

With easy Helm-based installation and Kubernetes-native scaling, Jaeger seamlessly integrates into your cloud infrastructure, empowering your team to analyze and improve application performance effectively.

Drop a query if you have any questions regarding Jaeger and we will get back to you quickly.

Empowering organizations to become ‘data driven’ enterprises with our Cloud experts.

Reduced infrastructure costs
Timely data-driven decisions

Get Started

About CloudThat

CloudThat is an award-winning company and the first in India to offer cloud training and consulting services worldwide. As a Microsoft Solutions Partner, AWS Advanced Tier Training Partner, and Google Cloud Platform Partner, CloudThat has empowered over 850,000 professionals through 600+ cloud certifications winning global recognition for its training excellence including 20 MCT Trainers in Microsoft’s Global Top 100 and an impressive 12 awards in the last 8 years. CloudThat specializes in Cloud Migration, Data Platforms, DevOps, IoT, and cutting-edge technologies like Gen AI & AI/ML. It has delivered over 500 consulting projects for 250+ organizations in 30+ countries as it continues to empower professionals and enterprises to thrive in the digital-first world.

FAQs

1. What storage backends are supported by Jaeger?

ANS: – Jaeger supports multiple storage backends, including Elasticsearch, Cassandra, Kafka, and Badger. For production environments, Elasticsearch and Cassandra are recommended for their scalability.

2. Can Jaeger be used with other tracing tools?

ANS: – Yes, Jaeger is compatible with OpenTelemetry, which allows integration with other tracing tools and observability platforms.

WRITTEN BY Bhupesh .

Bhupesh is working as a Research Associate at CloudThat. He is passionate about learning and gaining industrial experience in cloud computing technologies like AWS and Azure. Bhupesh is also an excellent communicator and collaborator. He also proactively seeks new challenges and opportunities to learn and grow in his role. His passion for learning and exploring new technologies and his technical expertise make him a valuable member of any team working in the field.