Skip to content

Instantly share code, notes, and snippets.

@Dev-Dipesh

Dev-Dipesh/ELK with Nginx.md

Last active June 6, 2025 19:31
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save Dev-Dipesh/2ac30a8a01afb7f65b2192928a875aa1 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save Dev-Dipesh/2ac30a8a01afb7f65b2192928a875aa1 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. Dev-Dipesh revised this gist Aug 8, 2016. 1 changed file with 54 additions and 1 deletion.
    55 changes: 54 additions & 1 deletion ELK with Nginx.md
    View file
    Original file line number Diff line number Diff line change
    @@ -150,10 +150,63 @@ Delete the deb-src entry from the /etc/apt/sources.list.d/kibana.list file to cl
    - Update apt and install Kibana
    - `$ sudo apt-get update && sudo apt-get install kibana`

    - Open Kibana configuration file for editing
    - `$ sudo vi /opt/kibana/config/kibana.yml`

    - In the Kibana configuration file, find the line that specifies *server.host*, uncomment and replace the IP address ("0.0.0.0" by default) with "localhost":
    - `server.host: "localhost"`
    - Save and exit. This setting makes it so Kibana will only be accessible to the localhost. This is fine because we will use an Nginx reverse proxy to allow external access.

    - Enable start on boot
    - `$ sudo update-rc.d kibana defaults 95 10`

    - Start service Kibana
    - `$ sudo service kibana start`

    - Test: At this stage you can point your browser to **`http://YOUR_ELASTIC_IP:5601`** after Kibana is started.
    - Before we can use the Kibana web interface, we have to set up a reverse proxy. Let's do that now, with Nginx.

    ---

    ####3 - Nginx Installation
    **Note:** If you already have an Nginx instance that you want to use, feel free to use that instead. Just make sure to configure Kibana so it is reachable by your Nginx server (you probably want to change the host value, in /opt/kibana/config/kibana.yml, to your Kibana server's private IP address or hostname). Also, it is recommended that you enable SSL/TLS.

    - Install Nginx and Apt-utils
    - `$ sudo apt-get install nginx apache2-utils`

    - Create an admin user and assign password to it
    - `$ sudo htpasswd -c /etc/nginx/htpasswd.users kibanaadmin`

    - Now open the nginx default server block
    - `$ sudo vim /etc/nginx/sites-available/default`

    - Delete the file's contents, and paste the following code block into the file.
    ```
    server {
    listen 80;
    server_name kibana;
    error_log /var/log/nginx/kibana.error.log;
    access_log /var/log/nginx/kibana.access.log;
    location / {
    rewrite ^/(.*) /$1 break;
    proxy_ignore_client_abort on;
    proxy_pass http://localhost:5601;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    }
    }
    ```

    - Save and exit. This configures Nginx to direct your server's HTTP traffic to the Kibana application, which is listening on `localhost:5601`.

    - Now restart Nginx to reflect your changes
    - `$ sudo service nginx restart`

    - Kibana is now accessible via your FQDN or the public IP address of your ELK Server i.e.
    - `http://elk\_server\_public\_ip/`
  2. Dev-Dipesh revised this gist Aug 7, 2016. 1 changed file with 6 additions and 6 deletions.
    12 changes: 6 additions & 6 deletions ELK with Nginx.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,11 +1,11 @@
    ## ELK (Elasticsearch Logstash Kibana)

    Though we're focused more on server setup procedure in this document, I will still give a very brief explanation in laymen terms for ELK. To those who are starting new in this stack, must have already heard of MVC (Model View Controller), so take it like this:
    ```
    Model => Elasticsearch
    View => Kibana
    Controller => Logstash
    ```

    - Model `=>` Elasticsearch (for Storage, Indexing & Search)
    - View `=>` Kibana (for DataViz & G-Man, yeah the one in half life :smirk:)
    - Controller `=>` Logstash (For Logs & Filtering)


    ---

    @@ -156,4 +156,4 @@ Delete the deb-src entry from the /etc/apt/sources.list.d/kibana.list file to cl
    - Start service Kibana
    - `$ sudo service kibana start`

    - Test: At this stage you can point your browser to `http://YOUR_ELASTIC_IP:5601` after Kibana is started.
    - Test: At this stage you can point your browser to **`http://YOUR_ELASTIC_IP:5601`** after Kibana is started.
  3. Dev-Dipesh revised this gist Aug 7, 2016. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion ELK with Nginx.md
    View file
    Original file line number Diff line number Diff line change
    @@ -156,4 +156,4 @@ Delete the deb-src entry from the /etc/apt/sources.list.d/kibana.list file to cl
    - Start service Kibana
    - `$ sudo service kibana start`

    - Test: At this stage you can point your browser to ‘http://YOUR_ELASTIC_IP:5601’ after Kibana is started.
    - Test: At this stage you can point your browser to `http://YOUR_ELASTIC_IP:5601` after Kibana is started.
  4. Dev-Dipesh revised this gist Aug 7, 2016. 1 changed file with 14 additions and 5 deletions.
    19 changes: 14 additions & 5 deletions ELK with Nginx.md
    View file
    Original file line number Diff line number Diff line change
    @@ -48,6 +48,10 @@ Canonical have recently stopped support for smaller instances and hence you can'

    ---

    ####0 - Preliminary (skip if already done)
    - `$ sudo apt-get update`
    - `$ sudo apt-get upgrade`

    ####1 - Java 8 Installation
    - Add Oracle Java PPA to apt:
    - `$ sudo add-apt-repository -y ppa:webupd8team/java`
    @@ -57,6 +61,9 @@ Canonical have recently stopped support for smaller instances and hence you can'

    - Install the latest version of Oracle Java 8
    - `$ sudo apt-get -y install oracle-java8-installer`

    - Check Java installation
    - `$ java -version`

    ---

    @@ -83,12 +90,12 @@ Canonical have recently stopped support for smaller instances and hence you can'
    ```json
    {
    "status" : 200,
    "name" : "Jigsaw",
    "name" : "Nuke - Frank Simpson",
    "cluster_name" : "elasticsearch",
    "version" : {
    "number" : "1.7.1",
    "build_hash" : "b88f43fc40b0bcd7f173a1f9ee2e97816de80b19",
    "build_timestamp" : "2015-07-29T09:54:16Z",
    "number" : "1.7.5",
    "build_hash" : "00f95f4ffca6de89d68b7ccaf80d148f1f70e4d4",
    "build_timestamp" : "2016-02-02T09:55:30Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
    },
    @@ -147,4 +154,6 @@ Delete the deb-src entry from the /etc/apt/sources.list.d/kibana.list file to cl
    - `$ sudo update-rc.d kibana defaults 95 10`

    - Start service Kibana
    - `$ sudo service kibana4 start`
    - `$ sudo service kibana start`

    - Test: At this stage you can point your browser to ‘http://YOUR_ELASTIC_IP:5601’ after Kibana is started.
  5. Dev-Dipesh revised this gist Aug 7, 2016. 1 changed file with 23 additions and 1 deletion.
    24 changes: 23 additions & 1 deletion ELK with Nginx.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,4 +1,14 @@
    ## ELK
    ## ELK (Elasticsearch Logstash Kibana)

    Though we're focused more on server setup procedure in this document, I will still give a very brief explanation in laymen terms for ELK. To those who are starting new in this stack, must have already heard of MVC (Model View Controller), so take it like this:
    ```
    Model => Elasticsearch
    View => Kibana
    Controller => Logstash
    ```

    ---

    Our ELK stack setup has four main components:
    - Logstash: The server component of Logstash that processes incoming logs
    - Elasticsearch: Stores all of the logs
    @@ -26,6 +36,18 @@ RAM - 4Gb
    CPU - 2
    ```

    The amount of CPU, RAM, and storage that your ELK Server will require depends on the volume of logs that you intend to gather.

    ####AWS
    Canonical have recently stopped support for smaller instances and hence you can't pick one from above configuration, so here is a workaround for that (only appreciated in test env) :wink:

    - Go to the following link
    - https://cloud-images.ubuntu.com/locator/
    - Enter the instance ID `ami-d05e75b8` in the search and once search populated, tap on the value of last column `ID` of table, which contains same AMI ID.
    - The next page you will land on is on AWS EC2 select instance type with all enabled.

    ---

    ####1 - Java 8 Installation
    - Add Oracle Java PPA to apt:
    - `$ sudo add-apt-repository -y ppa:webupd8team/java`
  6. Dev-Dipesh revised this gist Aug 7, 2016. 1 changed file with 8 additions and 0 deletions.
    8 changes: 8 additions & 0 deletions ELK with Nginx.md
    View file
    Original file line number Diff line number Diff line change
    @@ -36,6 +36,8 @@ CPU - 2
    - Install the latest version of Oracle Java 8
    - `$ sudo apt-get -y install oracle-java8-installer`

    ---

    ####2 - Elasticsearch Installation
    - Import Elasticsearch public GPG key into apt
    - `$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -`
    @@ -75,6 +77,10 @@ CPU - 2
    - Enable Elasticsearch to start on boot
    `$ sudo update-rc.d elasticsearch defaults 95 10`

    **Production tip:** DO NOT open any other ports, like 9200, to the world! There are many bots that search for 9200 and execute groovy scripts to overtake machines.

    ---

    ####3 - Logstash Installation
    - The Logstash package is available from the same repository as Elasticsearch, and public key is already installed, so let's create the Logstash source list:
    - `$ echo 'deb http://packages.elastic.co/logstash/2.2/debian stable main' | sudo tee /etc/apt/sources.list.d/logstash-2.2.x.list
    @@ -94,6 +100,8 @@ CPU - 2

    - We have yet to configure Logstash, but let leave it for later.

    ---

    ####4 - Kibana Installation
    - Download and install the Public Signing Key
    - `$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -`
  7. Dev-Dipesh renamed this gist Aug 7, 2016. 1 changed file with 0 additions and 0 deletions.
    0 ELK (E - v2.3.0) with Nginx.md → ELK with Nginx.md
    View file
    File renamed without changes.
  8. Dev-Dipesh revised this gist Aug 7, 2016. 1 changed file with 95 additions and 44 deletions.
    139 changes: 95 additions & 44 deletions ELK (E - v2.3.0) with Nginx.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,9 +1,9 @@
    ## ELK
    Our ELK stack setup has four main components:
    - Logstash: The server component of Logstash that processes incoming logs
    - Elasticsearch: Stores all of the logs
    - Kibana: Web interface for searching and visualizing logs, which will be proxied through Nginx
    - Filebeat: Installed on client servers that will send their logs to Logstash, Filebeat serves as a log shipping agent that utilizes the lumberjack networking protocol to communicate with Logstash
    - Logstash: The server component of Logstash that processes incoming logs
    - Elasticsearch: Stores all of the logs
    - Kibana: Web interface for searching and visualizing logs, which will be proxied through Nginx
    - Filebeat: Installed on client servers that will send their logs to Logstash, Filebeat serves as a log shipping agent that utilizes the lumberjack networking protocol to communicate with Logstash

    ![ELK+NGINX](https://assets.digitalocean.com/articles/elk/elk-infrastructure.png)

    @@ -16,7 +16,7 @@ Logstash - 2.2
    Kibana - 4.5
    ```

    #### Installation Steps
    ## Installation Steps

    For production go with three separate instances for each, elasticsearch, logstash and kibana. Currently we are setting it up on a single machine. You can opt for similar configuration on any cloud provider.

    @@ -26,44 +26,95 @@ RAM - 4Gb
    CPU - 2
    ```

    - **Install Java 8**
    - Add Oracle Java PPA to apt:
    `$ sudo add-apt-repository -y ppa:webupd8team/java`
    - Update your apt package database:
    `$ sudo apt-get update`
    - Install the latest version of Oracle Java 8
    `$ sudo apt-get -y install oracle-java8-installer`
    ####1 - Java 8 Installation
    - Add Oracle Java PPA to apt:
    - `$ sudo add-apt-repository -y ppa:webupd8team/java`

    - Update your apt package database:
    - `$ sudo apt-get update`

    - Install the latest version of Oracle Java 8
    - `$ sudo apt-get -y install oracle-java8-installer`

    **2 - Install Elasticsearch**
    - Import Elasticsearch public GPG key into apt
    `$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -`
    - Create the Elasticsearch source list
    `$ echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list`
    - Update your package database
    `$ sudo apt-get update`
    - Install Elasticsearch
    `$ sudo apt-get -y install elasticsearch`
    - Start Elasticsearch service
    `$ sudo service elasticsearch restart`
    - Test
    `$ curl localhost:9200`
    - If the output is similar to this, then you will know that Elasticsearch is running properly:
    ```json
    {
    "status" : 200,
    "name" : "Jigsaw",
    "cluster_name" : "elasticsearch",
    "version" : {
    "number" : "1.7.1",
    "build_hash" : "b88f43fc40b0bcd7f173a1f9ee2e97816de80b19",
    "build_timestamp" : "2015-07-29T09:54:16Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
    },
    "tagline" : "You Know, for Search"
    }
    ```
    - Enable Elasticsearch to start on boot
    `$ sudo update-rc.d elasticsearch defaults 95 10`
    ####2 - Elasticsearch Installation
    - Import Elasticsearch public GPG key into apt
    - `$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -`

    - Create the Elasticsearch source list
    - `$ echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list`

    - Update your package database
    - `$ sudo apt-get update`

    - Install Elasticsearch
    - `$ sudo apt-get -y install elasticsearch`

    - Start Elasticsearch service
    - `$ sudo service elasticsearch restart`

    - Test
    - `$ curl localhost:9200`

    - If the output is similar to this, then you will know that Elasticsearch is running properly:
    ```json
    {
    "status" : 200,
    "name" : "Jigsaw",
    "cluster_name" : "elasticsearch",
    "version" : {
    "number" : "1.7.1",
    "build_hash" : "b88f43fc40b0bcd7f173a1f9ee2e97816de80b19",
    "build_timestamp" : "2015-07-29T09:54:16Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
    },
    "tagline" : "You Know, for Search"
    }
    ```

    - Enable Elasticsearch to start on boot
    `$ sudo update-rc.d elasticsearch defaults 95 10`

    ####3 - Logstash Installation
    - The Logstash package is available from the same repository as Elasticsearch, and public key is already installed, so let's create the Logstash source list:
    - `$ echo 'deb http://packages.elastic.co/logstash/2.2/debian stable main' | sudo tee /etc/apt/sources.list.d/logstash-2.2.x.list
    `

    - Update apt package database:
    - `$ sudo apt-get update`

    - Install Logstash
    - `$ sudo apt-get install logstash`

    - Enable start on boot
    - `$ sudo update-rc.d logstash defaults 97 8`

    - Run service logstash
    - `$ sudo service logstash start`

    - We have yet to configure Logstash, but let leave it for later.

    ####4 - Kibana Installation
    - Download and install the Public Signing Key
    - `$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -`

    - Add the repository definition to your `/etc/apt/sources.list.d/kibana.list` file
    - `$ echo "deb http://packages.elastic.co/kibana/4.5/debian stable main" | sudo tee -a /etc/apt/sources.list
    `

    **WARNING**
    ```
    Use the echo method described above to add the Kibana repository. Do not use add-apt-repository, as that command adds a deb-src entry with no corresponding source package. When the deb-src entry, is present, the commands in this procedure generate an error similar to the following:
    Unable to find expected entry 'main/source/Sources' in Release file (Wrong sources.list entry or malformed file)
    Delete the deb-src entry from the /etc/apt/sources.list.d/kibana.list file to clear the error.
    ```

    **3 - Logstash Installation**
    - Update apt and install Kibana
    - `$ sudo apt-get update && sudo apt-get install kibana`

    - Enable start on boot
    - `$ sudo update-rc.d kibana defaults 95 10`

    - Start service Kibana
    - `$ sudo service kibana4 start`
  9. Dev-Dipesh revised this gist Aug 7, 2016. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion ELK (E - v2.3.0) with Nginx.md
    View file
    Original file line number Diff line number Diff line change
    @@ -26,7 +26,7 @@ RAM - 4Gb
    CPU - 2
    ```

    **1 - Install Java 8**
    - **Install Java 8**
    - Add Oracle Java PPA to apt:
    `$ sudo add-apt-repository -y ppa:webupd8team/java`
    - Update your apt package database:
  10. Dev-Dipesh revised this gist Aug 7, 2016. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions ELK (E - v2.3.0) with Nginx.md
    View file
    Original file line number Diff line number Diff line change
    @@ -25,6 +25,7 @@ OS - Ubuntu 14.04 LTS
    RAM - 4Gb
    CPU - 2
    ```

    **1 - Install Java 8**
    - Add Oracle Java PPA to apt:
    `$ sudo add-apt-repository -y ppa:webupd8team/java`
  11. Dev-Dipesh revised this gist Aug 7, 2016. 1 changed file with 61 additions and 1 deletion.
    62 changes: 61 additions & 1 deletion ELK (E - v2.3.0) with Nginx.md
    View file
    Original file line number Diff line number Diff line change
    @@ -5,4 +5,64 @@ Our ELK stack setup has four main components:
    - Kibana: Web interface for searching and visualizing logs, which will be proxied through Nginx
    - Filebeat: Installed on client servers that will send their logs to Logstash, Filebeat serves as a log shipping agent that utilizes the lumberjack networking protocol to communicate with Logstash

    ![ELK+NGINX](https://assets.digitalocean.com/articles/elk/elk-infrastructure.png)
    ![ELK+NGINX](https://assets.digitalocean.com/articles/elk/elk-infrastructure.png)

    **These are the versions we are currently setting up in this installment, please make note accordingly if you have specific version requirements.**

    ```
    JDK Version - 8
    Elasticsearch - 2.x
    Logstash - 2.2
    Kibana - 4.5
    ```

    #### Installation Steps

    For production go with three separate instances for each, elasticsearch, logstash and kibana. Currently we are setting it up on a single machine. You can opt for similar configuration on any cloud provider.

    ```
    OS - Ubuntu 14.04 LTS
    RAM - 4Gb
    CPU - 2
    ```
    **1 - Install Java 8**
    - Add Oracle Java PPA to apt:
    `$ sudo add-apt-repository -y ppa:webupd8team/java`
    - Update your apt package database:
    `$ sudo apt-get update`
    - Install the latest version of Oracle Java 8
    `$ sudo apt-get -y install oracle-java8-installer`

    **2 - Install Elasticsearch**
    - Import Elasticsearch public GPG key into apt
    `$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -`
    - Create the Elasticsearch source list
    `$ echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list`
    - Update your package database
    `$ sudo apt-get update`
    - Install Elasticsearch
    `$ sudo apt-get -y install elasticsearch`
    - Start Elasticsearch service
    `$ sudo service elasticsearch restart`
    - Test
    `$ curl localhost:9200`
    - If the output is similar to this, then you will know that Elasticsearch is running properly:
    ```json
    {
    "status" : 200,
    "name" : "Jigsaw",
    "cluster_name" : "elasticsearch",
    "version" : {
    "number" : "1.7.1",
    "build_hash" : "b88f43fc40b0bcd7f173a1f9ee2e97816de80b19",
    "build_timestamp" : "2015-07-29T09:54:16Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
    },
    "tagline" : "You Know, for Search"
    }
    ```
    - Enable Elasticsearch to start on boot
    `$ sudo update-rc.d elasticsearch defaults 95 10`

    **3 - Logstash Installation**
  12. Dev-Dipesh revised this gist Aug 7, 2016. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion ELK (E - v2.3.0) with Nginx.md
    View file
    Original file line number Diff line number Diff line change
    @@ -5,4 +5,4 @@ Our ELK stack setup has four main components:
    - Kibana: Web interface for searching and visualizing logs, which will be proxied through Nginx
    - Filebeat: Installed on client servers that will send their logs to Logstash, Filebeat serves as a log shipping agent that utilizes the lumberjack networking protocol to communicate with Logstash

    (ELK+NGINX)[https://assets.digitalocean.com/articles/elk/elk-infrastructure.png]
    ![ELK+NGINX](https://assets.digitalocean.com/articles/elk/elk-infrastructure.png)
  13. Dev-Dipesh created this gist Aug 7, 2016.
    8 changes: 8 additions & 0 deletions ELK (E - v2.3.0) with Nginx.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,8 @@
    ## ELK
    Our ELK stack setup has four main components:
    - Logstash: The server component of Logstash that processes incoming logs
    - Elasticsearch: Stores all of the logs
    - Kibana: Web interface for searching and visualizing logs, which will be proxied through Nginx
    - Filebeat: Installed on client servers that will send their logs to Logstash, Filebeat serves as a log shipping agent that utilizes the lumberjack networking protocol to communicate with Logstash

    (ELK+NGINX)[https://assets.digitalocean.com/articles/elk/elk-infrastructure.png]