Login- Consulting
- Training
- Partners
- About Us
x
Non-Profit Organization
Amazon EKS, Amazon S3, ClickHouse, OpenTelemetry, Amazon CloudWatch
Copilot said: SigNoz storage optimization, ClickHouse remediation, S3 tiered storage, OTel retry management, and CloudWatch monitoring on Amazon EKS
The client is a leading Indian ed-tech non-profit organization dedicated to improving early childhood education and foundational learning outcomes for children across the country. Through partnerships with state governments and education systems, the organization focuses on empowering parents, teachers, and caregivers with localized, play-based digital learning solutions. Its mission is to leverage technology and community engagement to enhance learning experiences and strengthen foundational education at scale.
Reduction in ClickHouse Free-Space Threshold
Restoration of ClickHouse Write Operations
OTel Retry Storm Eliminated
The incident was caused by a misconfigured storage threshold that reduced ClickHouse’s usable disk capacity. Failed S3 tiering prevented data offloading, causing ClickHouse write operations to stop when storage limits were reached. OTel collectors continuously retried failed writes, generating excessive cross-AZ network traffic. Limited monitoring and retry controls delayed detection, resulting in observability data loss.
• Adjusted ClickHouse storage threshold by reducing defaultKeepFreeSpaceBytes from 100 GB to 10 GB to restore usable disk capacity.
• Configured S3 tiered storage correctly to enable data offloading from local ClickHouse storage.
• Restored ClickHouse write operations after applying the updated storage configuration.
• Controlled OTel collector retries to prevent continuous retry traffic when ClickHouse becomes unavailable.
• Introduced independent monitoring for ClickHouse disk utilization and write failures.
• Added CloudWatch-based monitoring for infrastructure storage and network activity.
• Reviewed ClickHouse replica placement to evaluate the impact of cross-AZ traffic.
• Established proactive storage alerts to identify disk capacity issues before ClickHouse reaches its effective storage limit.
Optimized ClickHouse storage and observability infrastructure by restoring write operations, enabling S3 tiered storage, eliminating excess network traffic, and strengthening monitoring and governance controls.
SHARE
Our support doesn't end here. We have monthly newsletters, study guides, practice questions, and more to assist you in upgrading your cloud career. Subscribe to get them all!
