Data is the most superior aspect in this existing world. Configuring and managing data is the basic requirement in every aspect of life. Data is the facts and statistics collected during every operations of business, They can be used to measure or record business activities may be internal or external. Here we are streaming logs to S3 bucket so that there won’t be any data loss.
Kinesis firehose captures data from web applications, sensors, mobile applications, and various different sources and streams into Amazon S3 or Redshift. Kinesis Firehose takes care of monitoring, scaling and management of data.
[showhide type=”diagram” more_text=”Show Diagram” less_text=”Hide Diagram” hidden=”yes”]
Fluentd is a unified open source data collector which unify data collection and consumption for data use and understanding data.
Unified logging with JSON (structured logs)
Supports memory and file-based buffering to prevent data loss
td-agent is the stable lightweight server agent distribution of fluentd which resides on data generating application.td-agent is a data collection daemon. It collects data from various data sources and uploads them to treasure datastore.
[showhide type=”diagram2″ more_text=”Graphical Representation…” less_text=”Hide Image” hidden=”yes”]
Here, we are going to stream Nginx logs to S3 using td-agent (logging tool) through Kinesis Firehose which is a managed service for streaming data to S3 or Red Shift.
An EC2 instance with IAM Role “fluentd” and attach to an instance
[showhide type=”fluent_iam_role” more_text=”Click here for the role” less_text=”Hide Details” hidden=”yes”]
Configuration file is located in “/etc/td-agent/td-agent.conf”.
[showhide type=”agent_conf” more_text=”For Detailed Instructions..” less_text=”Hide Details” hidden=”yes”]
Copy and paste the following contents into the file and provide your access key and secret key.
Task 4: Start service td-agent
We now have to start the agent service.
[showhide type=”start_agent” more_text=”For Detailed Instructions..” less_text=”Hide Details” hidden=”yes”]
Start service using the following command:
Use the below commands to create logs This command is used for simple load testing. It will create 1000 requests with 10 requests running concurrently.
Task 5: Operation Check
Make sure that the logs are getting streamed to S3 bucket.
NOTE: It might take 10 minutes for data to appear in your bucket due to buffering. Make sure that role should be attached to an instance, so that Fluentd had access to write data into bucket.
We have configured td-agent.conf file to collect access logs from Nginx server from the path /var/log/nginx/access.log and send logs to Kinesis Firehose, which in turn stream logs to S3 bucket which can be used for other purposes.