DevOps Services for Healthcare Platform

About Client​

The Client is a healthcare network platform. The Client connects users to a network of 19,000 licensed doctors, 1,000 certified partner pharmacies through medical delivery service ApotikAntar, online pharmaceutical providing licensed medical laboratory services. Users can communicate with these medical professionals via chat, video, and voice call and avail quick services and medicines delivered at their address.

Problem Statement

A healthcare application was required for a robust and world-class infrastructure deployed on the AWS cloud. The application is an asset to improve the patient, provider, and hassle-free experience across the care continuum and accelerate the delivery of medication to their customers. To achieve these facilities, they are needing to focus on fault-tolerant HA (Highly Available) application hosted on AWS Cloud.

Business Objective

  • Aims to simplify access throughout the healthcare continuum
  • Users should be able to connect to doctors easily through a highly secure gateway
  • Connects customers to pharmacies and laboratory services. The connection is made via the ApoticAntar delivery service (integrated with Go-Jek’s Go-MED)
  • Cost effective healthcare insurance to the customers

Technical Objectives

1. Application to be deployed on a highly available and fault-tolerant infrastructure
2. Implement a simple and easily deployable Jenkins jobs to make the development independent from the DevOps
3. Configuration of all the internal applications to be accessible only within the secure VPN network
4. Configuration of Highly Available (HA) VPN infra with an active-active type that enables users to send and receive data across shared or public networks
5. Implement orchestration tools like Kubernetes for a cost-effectiveness and resource optimization
6. Implement Password-based authentication for AWS SFTP, used by Client partners to drop data for further processing
7. Implement in-house Git Repository for an extra layer of security
8. Implement Database pipeline for MySQL and PostgreSQL
9. Implement third-party tools to expose our internal APIs to the public
10. Implement a One-Click solution to deploy infrastructure for Dev and Production environment
11. Implement static content for the S3
12. Implement UI for a better view of the Kubernetes Pods
13. Implement centralized logging system for the application logs of the pods and service running on application.

Design Factors

1. To ensure high availability and fault tolerance Production environment, placed ASG with different scaling policies on different availability zones
2. Configured Jenkins jobs for Continuous integration, and continuous deployment with generic ansible playbook for multi-service deployment
3. Both WAF rules and an internal load balancer is being configured for accessing internal applications only via VPN
4. Use elastic file system for sharing the configuration between two active-active VPN servers and introduced Route53 to distribute the traffic equally
5. Configured AWS EKS Control plane cluster with self-managed workers. For high Availability of the pods, used HPA based on CPU, memory, JVM heap, GC, etc.
6. Used AWS Cloud Formation for launching API Gateway and lambda. Using Lambda, API gateway, and secret manager serviceable to achieve the password-based authentication for the client.
7. Installed GIT on Private EC2 instance and used RDS for the Data storage for in-house GIT repository. For DR configured hourly backup of the instance
8. Use Common Jenkins job for any data modification or database update on more than 25 MySQL RDS and common job for Postgres RDS as well. This is achieved via a third-party tool called Liquibase
9. Use third party software called Repose which is exposed via CloudFront, so that any public request for internal APIs comes only via repose
10. Configured AWX and Jenkins jobs for infrastructure deployment on Production and Development
11. Used s3 for all the static content of the website which helped to decrease 70 percent load on the server. Routing is done through CloudFront
12. Configured Rancher for the EKS Cluster which helps Developers to see the logs and pods behaviors.
13. Configured self-managed elastic search cluster node in EC2 instance and use fluent bit for pushing application logs of stage and prod.

Amazon Services Used

  • Amazon EC2 
  • VPC 
  • VPN 
  • Elastic Load Balancer 
  • Auto Scaling 
  • Amazon Route 53 
  • Amazon RDS (MySQL, PostGres)
  • Amazon CloudFormation
  • Elastic file system
  • CloudFront
  • Elastic search service
  • Elastic Kubernetes service
  • DynamoDb
  • Lambda
  • SFTP
  • Secret manager

Outcome

We have built a highly secure and robust infrastructure to serve 10000+ doctor consultation per day. We have migrated around 30+ micro services out of 70+ microservices from monolithic EC2 to Kubernetes which helped in improving the performance with the reduction in overall cost by 15%.

Lessons Learned

  1. Based on few Security breach incidents we have taken-up following Actions:
    We have segregated Dev and Prod network to get a better visibility on our infrastructure side.
  2. We have applied WAF over public endpoints to rate limit for all the APIs calls which can cause high load on servers, can hamper application performance.