INSTALLING ELK STACK VIA UBUNTU OS THROUGH AMAZON MACHINE IMAGE (AMI) ON NGINX/SYSTEM WEBSERVER
Prerequisite- sudo apt update (update the ubuntu in AMI)
- sudo apt install openjdk-8-jdk (install latest java on ubuntu in AMI)
- sudo apt-get install -y nginx (install nginx webserver)
- sudo systemctl enable nginx (enabling nginx to run)
1) Install Elasticsearch
- wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.2.0-amd64.deb
- sudo dpkg -i elasticsearch-7.2.0-amd64.deb
2) Install kibana
- sudo wget https://artifacts.elastic.co/downloads/kibana/kibana-7.2.0-amd64.deb
- sudo dpkg -i kibana-7.2.0-amd64.deb
3) Install Logstash
- sudo wget https://artifacts.elastic.co/downloads/logstash/logstash-7.2.0.deb
- sudo dpkg -i logstash-7.2.0.deb
5) Install FileBeat
- wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.2.0-amd64.deb
- sudo dpkg -i filebeat-7.2.0-amd64.deb
6) Modify Elasticsearch yaml file (modify some configurations)
- sudo vi /etc/elasticsearch/elasticsearch.yml
- Make below changes in this file
- cluster.name: my-application
- node.name: node-1
- network.host: localhost (change this line)
- http.port: 9200
- sudo systemctl start elasticsearch
Make below changes in the file
server.port: 5601
server.host: "localhost"
sudo systemctl start kibana
sudo apt-get install -y apache2-utils
sudo htpasswd -c /etc/nginx/htpasswd.users kibadmin
sudo systemctl restart nginx
vi /etc/logstash/conf.d/apache.conf (upload the file through logstash and then search using elastic search then kibana will find it and analyze it)
9) Getting data from filebeat (So what is a filebeat? it basically is a tool to ship from a specific server, so right now we are taking from an nginx webserver)
server.port: 5601
server.host: "localhost"
sudo systemctl start kibana
sudo apt-get install -y apache2-utils
sudo htpasswd -c /etc/nginx/htpasswd.users kibadmin
sudo vi /etc/nginx/sites-available/default
type in this below:
server {listen 80;server_name 3.108.42.168;(THIS CHANGE TO YOUR UNIQUE PUBLIC IP)auth_basic "Restricted Access";auth_basic_user_file /etc/nginx/htpasswd.users;location / {proxy_pass http://localhost:5601;proxy_http_version 1.1;proxy_set_header Upgrade $http_upgrade;proxy_set_header Connection 'upgrade';proxy_set_header Host $host;proxy_cache_bypass $http_upgrade;}}
sudo systemctl restart nginx
now check by putting 3.110.88.42 in the web
browser
put in username and pw (u/n:
kibadmin, p/w: kibadmin)
8) Download some sample data
sudo wget https://logz.io/sample-data
type in ls and you will find a list of files and folders:
sudo wget https://logz.io/sample-data
type in ls and you will find a list of files and folders:
sudo mv sample-data apache.log (to rename sample-data to apache.log)
Below is an example of a .log file (in this case the apache.log file):
vi /etc/logstash/conf.d/apache.conf (upload the file through logstash and then search using elastic search then kibana will find it and analyze it)
type in below:
input {file {path => "/home/ubuntu/apache.log"start_position => "beginning"sincedb_path => "/dev/null"}}filter {grok {match => { "message" => "%{COMBINEDAPACHELOG}" }}date {match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]}geoip {source => "clientip"}}output {elasticsearch {hosts => ["localhost:9200"]index => "petclinic-prd-1" (ANY INDEX YOU WANT SO THAT IT IS SEARCHABLE IN KIBANA)}
sudo systemctl restart logstash
go to kibana à Management à Index Patterns à Create Index Pattern
Then now you can utilise the Kibana Query Language (KQL)
sudo filebeat modules list (a lists of enabled and disabled modules will appear)
sudo filebeat modules enable nginx (enable nginx)
sudo filebeat modules enable system (enable system)
cd /etc/filebeat/modules.d/
sudo filebeat modules enable nginx (enable nginx)
sudo filebeat modules enable system (enable system)
cd /etc/filebeat/modules.d/
sudo vi nginx.yml (edit the configuration for nginx)
access:
enabled: true
var.paths: ["/var/log/nginx/access.log*"]
# Error logs
error:
enabled: true
var.paths: ["/var/log/nginx/error.log*"]
access:
enabled: true
var.paths: ["/var/log/nginx/access.log*"]
# Error logs
error:
enabled: true
var.paths: ["/var/log/nginx/error.log*"]
sudo vi system.yml (edit configuration for system)
- module: system
# Syslog
syslog:
enabled: true
var.paths: ["/var/log/syslog*"]
auth:
enabled: true
var.paths: ["/var/log/auth.log*"]
- module: system
# Syslog
syslog:
enabled: true
var.paths: ["/var/log/syslog*"]
auth:
enabled: true
var.paths: ["/var/log/auth.log*"]
sudo systemctl restart filebeat
sudo systemctl restart logstash
sudo systemctl restart logstash
go to kibana à Management à Index Patterns à Create Index Pattern
