LineClear Achieved Streamlined Data Management, Real-Time Analytics & Visualization, and the Generation of Business Centric Reports

Solutions

• Utilized Amazon DMS to import operational data from Amazon Aurora RDS and synchronized continuous data changes to Amazon S3.
• Created a cross-account AWS IAM role for secure access between Amazon DMS and Amazon S3 bucket in the BI AWS account.
• Leveraged Synology Cloud Sync to connect the client’s file store to Amazon S3 for seamless CSV data upload.
• Established a Lakehouse architecture with Databricks on top of Amazon S3 Data Lake for advanced analytics.
• Mounted Amazon S3 Data Lake onto Databricks clusters for efficient data access and processing.
• Real-time CDC jobs scheduled on the compute cluster for continuous data updates.
• Employed Apache Spark SQL for data querying, transformation, and storage in Delta Tables.
• Utilized Amazon S3 bucket as the data source for Amazon QuickSight dashboards.
• Programmatic refreshes triggered from Databricks Job every 15, 30, and 45 minutes for timely updates.
• Implemented row-level security based on the Branch column for region-specific access control