Achieving a 60% Efficiency Boost with AWS Glue Jobs for a Software Company

Solutions

• The solution is deployed in the Oregon region, with Amazon S3 buckets replicated to the Singapore region. 
• Data is sourced from Amazon MSK via an MSK sink connector for Amazon S3, and it is stored as a single file in a day-wise partition in a staging bucket. 
• An AWS glue crawler runs on this bucket creating a raw table in the database. 
• There are 5 glue jobs in total which run on top of the raw table every day, extracting and storing files in 5 separate partitions in a processed data bucket. The extraction is done based on the keys extracted from the payload.  
• The sub-partitions are created in the 5 main partitions which are further partitioned into day-wise basis. 
• Everyday crawlers run on the main partitions populating the Glue Catalog. 
• The dynamic creation of the crawler is achieved using an AWS Lambda.  
• The processed data bucket is used as an event notification trigger for the AWS Lambda to create the crawlers if required. 
• Amazon Athena is used to create the dataset based on the tables created in the glue catalog for the main data partition key. i.e. n tables for n partitions.