Conversion of CSV Files to Parquet Format using AWS Glue Job

Overview

AWS Glue is a serverless AWS data integration service that facilitates the discovery, preparation, movement, and integration of data from many sources for analytics users. It can be used for application development, machine learning, and analytics. Additional productivity and data operations functionality is also included for authoring, running jobs, and implementing business workflows.

You can find and connect to more than 70 data sources using AWS Glue and manage your data in a centralized data catalog. You can graphically build, run, and monitor extract, transform, and load (ETL) pipelines to load data into your data lakes. You may also rapidly search and query cataloged data using Amazon Athena, Amazon EMR, and Amazon Redshift Spectrum.

Now we will convert some CSV files to parquet format without writing any program code.

Step-by-Step Guide

Step 1: CSV files in Amazon S3 buckets must be converted to Parquet format.

Go to Amazon S3 Bucket from the AWS Console,

Step 2: Go to the AWS Glue Console and select ETL Jobs from the left menu pane,

Select the “Visual with a source and target” option, then click on “Create”

Select the Data source, edit the “Data source properties – S3,” and provide your S3 bucket and the path in which CSV files residing,

Select the Data format as CSV,

Now select the “Data target – S3 bucket”, then select the Format as parquet, compression type as Uncompressed,

Now select the Output file location,

Then please save the Job which we created,

Now we need to click on the “Run” Job,

Step 3:

You can view that your Job is Running,

Once the Job is complete, you can view the parquet formatted file in your target path in the Amazon S3 bucket,

Conclusion

Now we have easily converted CSV files to parquet format without using any programming code. Nodes for your job are configured using the visual job editor. Every node represents a certain action, such as reading data from the source location or transforming the data. Each node you include in your job includes characteristics that reveal details about the transform or the location of the data.

About CloudThat