Empower Network hosts one of the largest blogging communities online and provides educational training products and services to online marketers, entrepreneurs, start-ups, home-based and small businesses.
Problem Statement
Clients MongoDB sharded cluster configuration needs to be upgraded with best practices. As the CPU of main shard was reaching 100% at times leading to interruption in services, degrading the end-user experience, effecting the whole business landscape.
Business Objectives
To achieve a highly available and faster results time for the blogging platform
A highly scalable solution for high-customer traffic for the website
Provide continuous service to customers without any interruptions and lag
Technical Objectives
To be able to move an on-premises MongoDB cluster to AWS cloud.
Setup the environment with high availability and low latency
Secure the MongoDB cluster
Backward compatibility with on-premises application
Design Factors
The client wanted high availability, but also strong security.
We have created separate private subnets for all the MongoDB Nodes.
We have followed all the best practices provided by the AWS and MongoDB team, like using private static IP addresses; so that a down host would not cause long downtime and can be quickly brought up with the right IP Address.
We have integrated the nodes with new relic for monitoring.
Amazon Services Used
Amazon EC2
VPN
Elastic Load Balancer
Auto Scaling
Amazon Route 53
Elastic Container Service
IAM
CloudWatch
CloudTrail
Elastic search service
Elastic Kubernetes service
Architecture Diagram and Designs
Outcomes
All the best practices specified by MongoDB have been implemented.
Detailed log analysis was performed on logs during the 100% CPU incident and the issue was narrowed down to a slow query.
The query was modified, and the incident has not repeated since.
Automated backups and alarms configured on CloudWatch, Mongo Monitoring Service.
Lessons Learned
Hourly backed up data improves point in-time recovery.
Service interruptions have been mitigated through implementation of AWS best practices.