Kinesis firehose captures data from\u00a0web\u00a0applications, sensors, mobile applications, and various different sources and\u00a0streams into Amazon S3 or Redshift. Kinesis Firehose takes care of monitoring, scaling and management of data.
\n[showhide type=”diagram”\u00a0more_text=”Show Diagram” less_text=”Hide Diagram” hidden=”yes”]<\/p>\n
Fluentd is a unified open source data collector which unify data collection and consumption for data use and understanding data.<\/p>\n td-agent is the stable lightweight server agent distribution of fluentd which resides on data generating application.td-agent is a data collection daemon. It collects data from various data sources and uploads them to treasure datastore. Here, we are going to stream Nginx logs to S3 using td-agent (logging tool) through Kinesis Firehose which is a managed service for streaming data to S3 or Red Shift.<\/p>\n [showhide type=”fluent_iam_role”\u00a0more_text=”Click here for the role” less_text=”Hide Details” hidden=”yes”]<\/p>\n Role “<\/span>fluentd” is as follows:<\/span><\/p>\n [\/showhide]<\/p>\n Initially, create Kinesis Firehose Delivery stream using AWS Management console For more information, go through \u00a0https:\/\/aws.amazon.com\/kinesis\/firehose\/<\/a><\/p>\n [showhide type=”firehose”\u00a0more_text=”For Detailed Instructions..” less_text=”Hide Details” hidden=”yes”]<\/p>\n <\/p>\n After creating Kinesis Firehose delivery stream, select the destination where to send the streamed data. The destination might be either S3 bucket or Redshift.<\/p>\n Here we are going to configure buffer size, buffer interval and compression options for a stream<\/p>\n Enter the Buffer size as 5 and Buffer interval as 300<\/p>\n Kinesis Firehose buffers data up to\u00a05 MB or 300 seconds whichever condition is satisfied first.\u00a0 \u00a0<\/a> [\/showhide]<\/p>\n Fluentd is available as a Ruby gem ( [showhide type=”td-agent”\u00a0more_text=”For Detailed Instructions..” less_text=”Hide Details” hidden=”yes”]<\/p>\n Here, we proceed with td-agent.<\/p>\n![\"kinesis\"](\"https:\/\/content.cloudthat.com\/resources\/wp-content\/uploads\/2022\/11\/kinesis1.jpg\")
<\/a>[\/showhide]<\/p>\nFluentd:<\/strong><\/h4>\n
Features<\/strong><\/h5>\n
\n
td-agent<\/strong><\/h5>\n
\n[showhide type=”diagram2″\u00a0more_text=”Graphical Representation…” less_text=”Hide Image” hidden=”yes”]
\n<\/a>
\n[\/showhide]<\/p>\nPrerequisites<\/strong><\/h4>\n
\n
\n
{ \"Effect\":\"Allow\",\r\n \"Action\":[ \"s3:Get*\",\r\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0\"s3:List*\",\r\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0\"s3:Put*\",\r\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0\"s3:Post*\" \r\n ],\r\n \"Resource\":[ \"arn:aws:s3:::YOUR_BUCKET_NAME\/logs\/*\",\r\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0\"arn:aws:s3::: YOUR_BUCKET_NAME\" ]\r\n}<\/pre>\n\n
Tasks to stream logs to S3 bucket.<\/strong><\/h4>\n
\n
Task 1: Creating Kinesis Firehose Delivery stream<\/strong><\/span><\/h3>\n
Step 1: Select the Destination where you want to stream the logs<\/strong><\/span><\/h4>\n
\n
\n
<\/p>\nStep 2: Configuring Firehose<\/strong><\/span><\/h4>\n
<\/p>\nStep 3: Select IAM role \u201cfirehose_delivery_role\u201d<\/strong><\/span><\/h4>\n
<\/p>\nTask 2: Install Fluentd<\/strong><\/span><\/h3>\n
gem install fluentd<\/code>). Also,\u00a0Treasure Data<\/a>\u00a0packages it with all the dependencies as\u00a0td-agent<\/code>.<\/p>\nInstall td-agent using the following command<\/span><\/strong><\/h4>\n