Infrastructure Migration of Tercept from GCP to AWS Cloud Platform

About Client Problem Statement Business Objectives Technical Objectives Design Factors Services Used Diagram and Designs Outcome Lessons Learned

About Client

Tercept is a performance data platform that helps marquee publishers like Futbol Sites/BolaVIP, Carousell, TSM Games/BlitZ, LaOpinion, ElDiario, 1Weather, Times Internet – Gaana, InMobi, FrontStory, SonyLIV, Zee5, Digit & eBay with use cases around analytics & optimization to help them manage & boost their ad-revenues.

Problem Statement

  • The Client requirements were to migrate their Infrastructure along with data from GCP to AWS
  • All the applications and services which belongs to GCP should be migrated to AWS

Business Objectives

  • Use of AWS Cloud services for their Analytics and Application requirements
  • Reduce data storage cost
  • Optimize the performance of data analytics

Technical Objectives

  • Setup of highly available and scalable application for serving the massive traffic with the help of EC2, ALB, CloudFront
  • Setup of WordPress application on AWS Lightsail
  • Setup of RDS DB
  • Configure SNS and SQS for all the topics and subscription that belong to Google Pub-Sub
  • Migrated GCS data to AWS S3
  • Configure Glue Crawlers on S3 to create the databases and tables for Athena

Design Factors

  • Configure Lightsail instance for web hosting
  • Configure SNS with SQS to make internal message communication with applications running on EC2
  • Configure EC2 servers of multiple applications
  • Configure Glue Crawlers for data reside in S3
  • For BigQuery feature of deleting data older than 30 days, Configured Lambda automation scripts
  • CloudWatch Triggers to automate the Lambda execution
  • Lambda Automation for archiving data which is older than 30 days from main-logs S3 bucket to another S3 bucket

Cloud Services Used

  • Amazon Athena
  • AWS Glue
  • Amazon S3
  • Amazon SNS
  • Amazon SQS
  • Amazon EC2
  • AWS Elastic Load balancing
  • Amazon CloudFront
  • AWS CloudWatch
  • Amazon RDS
  • Amazon QuickSight

Architecture Diagram and Designs

  1. AWS Architecture Diagram
  2. AWS Architecture Diagram

  3. Automation of serving Athena tables with data of last 30 days from S3
    • Deleting partition from the tables which are older than 30 days
    • Moving the Datafiles that are deleted from table to different S3 bucket
  4. Automation of updating Athena tables data every hour, which is written in the S3 bucket

Outcomes

  • The Core Application is Highly Scalable and Available running on an EC2 server with Load balancing and CDN in place
  • We have migrated GCS buckets to Amazon S3 and used AWS Glue and Athena to works seamlessly
  • The client has started using AWS services for their analytics, such as Amazon Athena, AWS Glue

Lessons Learned

  • The tables in the database are updating on an hourly basis
  • The Client application is querying on S3 using Amazon Athena, which helped in the optimization of cost and performance
  • All the data is archiving and stored as Back-up