Elasticsearch elk stack

5 Easy Steps To Deploy the ElasticSearch Logstash Kibana and Beats ELK Monitoring Stack For Free

Elastic Search

Elasticsearch or simply ELK stack, according to the official website definition, is a distributed, free and open search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured. Elasticsearch is built on Apache Lucene and was first released in 2010 by Elasticsearch N.V. (now known as Elastic).

The Kibana, Logstash, soon became a part of the tools that make the Elasticsearch an industry standard for monitoring, and logging infrastructure and application data. The 3 (and more recently includes Beats to make it 4) service is christianed the ELK Stack or Elastic Stack, or simply ELK (that is Elasticsearch, Logstash, and Kibana).

The Elasticsearch service has evolve now to include some more varied collection of lightweight shipping agents called Beats which sends data to Elasticsearch.

That been said, let us start by installing Elasticsearch

1. Add Elastic’s signing key so that the downloaded package can be verified (skip this step if you’ve already installed packages from Elastic)
$wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Elasticsearch ELK signing key

2. We need to then install the apt-transport-https package:

$sudo apt-get update && sudo apt-get install apt-transport-https

Update apt repository and install apt-transport-https

3. Add the repository definition to your system:
$echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

Add the repository artifact for elasticsearch

4. Install a version of Elasticsearch ELK that contains only features licensed under Apache 2.0 (aka OSS Elasticsearch)
$echo "deb https://artifacts.elastic.co/packages/oss-7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

Install the Elasticsearch version under Apache 2.0 license

5. update your repositories and install Elasticsearch:
$sudo apt-get update && sudo apt-get install elasticsearch

Update the apt repository and install elasticsearch elk

6. Open the yml configuration file to bind Elasticsearch to either a private IP or localhost.
$sudo vim /etc/elasticsearch/elasticsearch.yml

Edit the elasticsearch.yml file

In the file, find and uncomment the below block. Remove # to uncomment.
network.host: "localhost"
cluster.initial_master_nodes: ["<PrivateIP"]

Elasticsearch ELK web interface dashboard at port 9200
Install and Configure Logstash

1. First make sure java is installed in the machine.
$java -version
Result: openjdk version "1.8.0_191" OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-2ubuntu0.16.04.1-b12) OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)

Else install with: $sudo apt-get install default-jre

2. Install logstash
$sudo apt-get install logstash

Install logstash with from the apt repository

3. Configure logstash with data. Create a logstash configuration file in the logstash directory
$sudo nano /etc/logstash/conf.d/apache-01.conf

Then download this sample data file and import the log data to logstash. Add the path where it is downloaded in the command below:
$input { file { path => "/home/ubuntu/apache-daily-access.log" start_position => "beginning" sincedb_path => "/dev/null" } } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] } geoip { source => "clientip" } } output { elasticsearch { hosts => ["localhost:9200"] } }

4. Create a configuration file called 02-beats-input.conf where you will set up your Filebeat input:
$sudo nano /etc/logstash/conf.d/02-beats-input.conf

input {
  beats {
    port => 5044

5. Next, create a configuration file called 30-elasticsearch-output.conf:
$sudo nano /etc/logstash/conf.d/30-elasticsearch-output.conf

Insert the following output configuration. Essentially, this output configures Logstash to store the Beats data in Elasticsearch

output {
if [@metadata][pipeline] {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
pipeline => "%{[@metadata][pipeline]}"
} else {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

6. Test to be sure there is no syntax error in your logstash configuration.
$sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t

Testing for syntax error in logstash settings

7. Then start the logstash service
$sudo service logstash start
In case the above command returns Logstash: Unrecognized service

Start the logstash service for elasticsearch elk

In a situation where you get an error like “logstash not installed” after installing and running the logstash service, do not panic, simply install the system/service package for logstash

$sudo /usr/share/logstash/bin/system-install /etc/logstash/startup.options sysv

Install the system/service package for logstash

Then run the start command again.


1. Install the kibana from the apt repository
$sudo apt-get install kibana
$sudo service kibana start

Install Kibana and start kibana system service for elasticsearch

2. Configure username and password for kibana login
$echo "kibanaadmin:openssl passwd -apr1" | sudo tee -a /etc/nginx/htpasswd.users

Setup password and username for kibana UI dashboard for elasticsearch elk

Visit http://localhost:5601/status to see information about your server/host

Kibana web UI displayed

1. Beats are several lightweight data shippers that collects data from various sources and transport them to Logstash or Elasticsearch.
Examples includes:
Filebeat for files, Metricbeat for metric data, Packetbeat that analyze network data, Winlogbeat for windows events, Auditbeat audit framework and file integrity, Heartbeat for service availability and active probing.

We begin by installing and configuring the filebeat
$ sudo apt install filebeat

Install filebeat elasticsearch elk

2. Open the filebeat.yml file in etc directory to comment out the elasticsearch output, this is because file beat will not send direct data to elasticsearch. It rather will send it to logstash.

$sudo nano /etc/filebeat/filebeat.yml

Scroll to Elasticsearch Output section and add the # sign to comment out the output.elasticsearch and hosts line.


# Array of hosts to connect to.
#hosts: ["localhost:9200"]

Just below, in the Logstash Output section, uncomment the logstash host by removing the # sign from the hosts: [localhost:5044]

# The Logstash hosts
hosts: ["localhost:5044"]

Once that is done, you can then save, and close your file editor.

Crtl X, Y Enter.

configure elasticsearch elk logstash in filebeat

More filebeat modules can be found in https://www.elastic.co/guide/en/beats/filebeat/7.6/filebeat-modules.html

Then start filebeat
$sudo service filebeat start

start filebeat for elasticsearch elk

Next, enable the filebeat system
$sudo filebeat modules enable system

Enable the filebeat module system

Check the list of modules in the filebeat
$sudo filebeat modules list

Show filebeat modules list

You can see the parameters of the module in the /etc/filebeat/modules.d/system.yml configuration file.

Now, we should Setup filebeat ingestion pipeline which will parse log data before sending it to elasticsearch through logstash

$sudo filebeat setup --pipelines --modules system

Install and configure the Metricbeat
$sudo apt-get install metricbeat
$sudo service metricbeat start

Install and start metricbeats






Leave a Reply

Your email address will not be published. Required fields are marked *