The collections and analyzing of the log becomes extremely difficult because of the diversity generated. For example we have access logs, error logs, application logs etc. which are associated with an application or a server.<\/p>\n
In this blog, I will be demonstrating how to install and configure ELK Stack. Before we begin, let\u2019s have a quick overview of the overall architecture with their components, followed by the implementation procedure.<\/strong> <\/p>\n Working:<\/strong> Prerequisites:<\/strong><\/p>\n Step 1: Launching EC2 Instance and all Installations<\/strong><\/p>\n \u00a0Install Java 8<\/strong><\/p>\n Install ElasticSearch\u00a0<\/strong><\/p>\n \u00a0Install Logstash\u00a0<\/strong><\/p>\n Install Kibana\u00a0<\/strong><\/p>\n Step 2: Configurations<\/strong> NOTE<\/strong>: This will make Kibana accessible to instance_ip only. If we want to allow external access, then need to use Nginx as reverse proxy.<\/p>\n To allow external access following are the steps to configure with Nginx<\/strong><\/p>\n This will prompt for a password that you will need to access Kibana dashboard along with kibadmin user<\/p>\n This configuration will make nginx to direct the server\u2019s HTTP traffic to kibana which is listening on localhost:5601. This will enable to access kibana dashboard with elasticsearch server\u2019s public ip. Step 3: Access Kibana Dashboard<\/strong><\/p>\n This is the dashboard that we will get.<\/p>\n In this way, we get the logs.But, there are many options to view logs in different formats and to filter them.<\/p>\n
\nELK stands for: E<\/strong>lasticsearch, L<\/strong>ogstash and K<\/strong>ibana.<\/p>\n
\n\u00a0<\/strong><\/p>\nArchitecture of ELK Stack:<\/strong><\/h3>\n
![\"blog-archi\"](\"https:\/\/content.cloudthat.com\/resources\/wp-content\/uploads\/2022\/11\/blog-archi1.png\")
<\/a><\/p>\n\n
\n
\n
\n
\nThe ELK stack architecture is very simple and clearly specifies the flow of the process.Various logs from different locations will be pulled by the Logstash (If you install Nginx for allowing external access then the logs will go to Nginx first), it will process the logs.
\nLogstash is the center where all the logs are processed and differentiated. Logs are then pushed to ElasticSearch, which is a Retrieval engine, it will index all the logs as per index pattern and will store it to be further accessed by Kibana.
\nKibana is a Web UI through which we will do all the activities such as visualizing and analyzing, creating index patterns, etc.<\/p>\n\n
\n
Making ELK Stack Up and Running:<\/strong><\/h3>\n
\n
\n
sudo add-apt-repository -y ppa:webupd8team\/java\r\nsudo apt-get update\r\nsudo apt-get -y install oracle-java8-installer\r\n<\/pre>\n
\n
sudo wget -qO - https:\/\/packages.elastic.co\/GPG-KEY elasticsearch | sudo apt-key add \u2013\r\necho \"deb https:\/\/packages.elastic.co\/elasticsearch\/2.x\/debian stable main\" | sudo tee -a \/etc\/apt\/sources.list.d\/elasticsearch-2.x.list\r\nsudo apt-get update\r\nsudo apt-get -y install elasticsearch\r\nsudo service elasticsearch restart\r\ncurl localhost:9200\r\nsudo update-rc.d elasticsearch defaults 95 10\r\n<\/pre>\n
echo \"deb https:\/\/packages.elasticsearch.org\/logstash\/1.5\/debian stable main\" | sudo tee -a \/etc\/apt\/sources.list\r\nsudo apt-get update\r\nsudo apt-get install logstash\r\nsudo update-rc.d logstash defaults 97 8\r\nsudo service logstash start\r\nsudo service logstash status\r\n<\/pre>\n
wget https:\/\/download.elastic.co\/kibana\/kibana\/kibana-4.1.1-linux-x64.tar.gz\r\ntar -xzf kibana-4.1.1-linux-x64.tar.gz\r\ncd kibana-4.1.1-linux-x64\/\r\nsudo mkdir -p \/opt\/kibana\r\nsudo mv kibana-4.1.1-linux-x64\/* \/opt\/kibana\r\ncd \/etc\/init.d && sudo wget https:\/\/raw.githubusercontent.com\/akabdog\/scripts\/master\/kibana4_init -O kibana4\r\nsudo chmod +x \/etc\/init.d\/kibana4\r\nsudo update-rc.d kibana4 defaults 96 9\r\nsudo service kibana4 start\r\n<\/pre>\n
\n\u00a0Configure Logstash<\/strong>:<\/p>\n\n
\n
input {\r\nfile {\r\ntype => \"syslog\"\r\npath => [ \"\/var\/log\/messages\", \"\/var\/log\/*.log\",\"\/var\/log\/httpd\/access_log\",\"\/var\/log\/httpd\/error_log\" ]\r\n}\r\n}\r\noutput {\r\nstdout {\r\ncodec => rubydebug\r\n}\r\nif ([program] == \"logstash\" or [program] == \"elasticsearch\" or [program] == \"nginx\") and [environment] == \"production\" \r\n{\r\n elasticsearch {\r\n host => \"localhost\"\r\n index => \"httpd-%{*}\"\r\n }\r\n }\r\nelse {\r\nelasticsearch {\r\nhost => \"localhost\" # Use the internal IP of your Elasticsearch server for production\r\n}}}\r\nfilter {\r\n if [type] == \"syslog\" {\r\n grok {\r\n match => { \"message\" => \"%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{COMBINEDAPACHELOG} %{DATA:syslog_program}(?:\\[%{POSINT:sy$\r\n add_field => [ \"received_at\", \"%{@timestamp}\" ]\r\n add_field => [ \"received_from\", \"%{host}\" ]\r\n }\r\n syslog_pri { }\r\n date {\r\n match => [ \"syslog_timestamp\", \"MMM d HH:mm:ss\", \"MMM dd HH:mm:ss\" ]\r\n }}}\r\n<\/pre>\n\n
sudo service elasticsearch restart<\/pre>\n
sudo service kibana4 restart<\/pre>\n
sudo service logstash restart\r\n<\/pre>\n
\n
sudo service kibana4 restart<\/pre>\n
\n
sudo htpasswd -c \/etc\/nginx\/htpasswd.users kibadmin<\/pre>\n
\n
sudo vi \/etc\/nginx\/sites-available\/default<\/pre>\n
server {\r\nlisten 80;\r\nserver_name example.com;\r\n auth_basic \"Restricted Access\";\r\nauth_basic_user_file \/etc\/nginx\/htpasswd.users;\r\nlocation \/ {\r\nproxy_pass https:\/\/localhost:5601;\r\n proxy_http_version 1.1;\r\nproxy_set_header Upgrade $http_upgrade;\r\n proxy_set_header Connection 'upgrade';\r\n proxy_set_header Host $host;\r\n proxy_cache_bypass $http_upgrade; \r\n}\r\n}\r\n<\/pre>\n
\nRestart nginx to apply changes that we made<\/p>\nsudo service nginx restart<\/pre>\n
\n
\n
<\/a><\/p>\n<\/a><\/p>\n