Infrastructure Migration of Tercept from GCP to AWS Cloud Platform

About Client

Tercept is a performance data platform that helps marquee publishers like Futbol Sites/BolaVIP, Carousell, TSM Games/BlitZ, LaOpinion, ElDiario, 1Weather, Times Internet – Gaana, InMobi, FrontStory, SonyLIV, Zee5, Digit & eBay with use cases around analytics & optimization to help them manage & boost their ad-revenues.

Problem Statement

  • The Client requirements were to migrate their Infrastructure along with data from GCP to AWS
  • All the applications and services which belongs to GCP should be migrated to AWS

Business Objectives

  • Use of AWS Cloud services for their Analytics and Application requirements
  • Reduce data storage cost
  • Optimize the performance of data analytics

Technical Objectives

  • Setup of highly available and scalable application for serving the massive traffic with the help of EC2, ALB, CloudFront
  • Setup of WordPress application on AWS Lightsail
  • Setup of RDS DB
  • Configure SNS and SQS for all the topics and subscription that belong to Google Pub-Sub
  • Migrated GCS data to AWS S3
  • Configure Glue Crawlers on S3 to create the databases and tables for Athena

Design Factors

  • Configure Lightsail instance for web hosting
  • Configure SNS with SQS to make internal message communication with applications running on EC2
  • Configure EC2 servers of multiple applications
  • Configure Glue Crawlers for data reside in S3
  • For BigQuery feature of deleting data older than 30 days, Configured Lambda automation scripts
  • CloudWatch Triggers to automate the Lambda execution
  • Lambda Automation for archiving data which is older than 30 days from main-logs S3 bucket to another S3 bucket

Cloud Services Used

  • Amazon Athena
  • AWS Glue
  • Amazon S3
  • Amazon SNS
  • Amazon SQS
  • Amazon EC2
  • AWS Elastic Load balancing
  • Amazon CloudFront
  • AWS CloudWatch
  • Amazon RDS
  • Amazon QuickSight

Architecture Diagram and Designs

  1. AWS Architecture Diagram
  2. AWS Architecture Diagram

  3. Automation of serving Athena tables with data of last 30 days from S3
    • Deleting partition from the tables which are older than 30 days
    • Moving the Datafiles that are deleted from table to different S3 bucket
  4. Automation of updating Athena tables data every hour, which is written in the S3 bucket

Outcomes

  • The Core Application is Highly Scalable and Available running on an EC2 server with Load balancing and CDN in place
  • We have migrated GCS buckets to Amazon S3 and used AWS Glue and Athena to works seamlessly
  • The client has started using AWS services for their analytics, such as Amazon Athena, AWS Glue

Lessons Learned

  • The tables in the database are updating on an hourly basis
  • The Client application is querying on S3 using Amazon Athena, which helped in the optimization of cost and performance
  • All the data is archiving and stored as Back-up