Skip to content

Instantly share code, notes, and snippets.

@pydevops

pydevops/cheatsheet-elasticsearch.md

Forked from ruanbekker/cheatsheet-elasticsearch.md
Created October 2, 2018 23:58
Show Gist options
  • Save pydevops/8fefb652fe91b66d98ed300202f3752e to your computer and use it in GitHub Desktop.
Save pydevops/8fefb652fe91b66d98ed300202f3752e to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. @ruanbekker ruanbekker revised this gist Jun 20, 2018. 1 changed file with 9 additions and 0 deletions.
    9 changes: 9 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -221,6 +221,15 @@ green open index-name-2018.01.19 Vp1EBoeMQkS-a_upLzedhQ 5 1 1220
    green open index-name-2018.01.17 hSJMzFJIQrePifCfgb1rOA 5 1 2875 0 3.8mb 1.9mb
    ```

    View only the index name header:

    ```
    $ curl -XGET 'http://127.0.0.1:9200/_cat/indices/*2018.03.*?v&h=index'
    index
    index-name-2018.03.01
    index-name-2018.03.02
    ```

    ## How Many Documents in the ES Cluster (Across all Indices):

    ```
  2. @ruanbekker ruanbekker revised this gist Jun 20, 2018. 1 changed file with 16 additions and 7 deletions.
    23 changes: 16 additions & 7 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1294,12 +1294,21 @@ $ curl -XGET 'elasticsearch:9200/_tasks?detailed=true&actions=*/delete/byquery&p

    Setup the [S3 Snapshot Repository](https://sysadmins.co.za/aws-elasticsearch-register-s3-repository-for-snapshots-using-the-cli/?rbas_source=gist.github.com?rbas_sourcepage=cheatsheet-elasticsearch.md)

    Viw the Snapshot Repository:
    List the Snapshot Repositories:

    ```
    $ curl -XGET 'http://elasticsearch:9200/_snapshot/index-backups?pretty'
    $ curl -XGET 'http://127.0.0.1:9200/_cat/repositories?v'
    id type
    foo-bacups s3
    bar-backups s3
    ```

    View the Snapshot Repository:

    ```
    $ curl -XGET 'http://elasticsearch:9200/_snapshot/bar-backups?pretty'
    {
    "index-backups" : {
    "bar-backups" : {
    "type" : "s3",
    "settings" : {
    "bucket" : "my-es-snapshot-bucket",
    @@ -1316,7 +1325,7 @@ Create a Snapshot named `mysnapshot_ruan-test-2018-05-24_1` of the index: `ruan-

    ```
    $ curl -XPUT -H 'Content-Type: application/json' \
    'http://elasticsearch:9200/_snapshot/index-backups/mysnapshot_ruan-test-2018-05-24_1?wait_for_completion=true&pretty=true' -d '
    'http://elasticsearch:9200/_snapshot/bar-backups/mysnapshot_ruan-test-2018-05-24_1?wait_for_completion=true&pretty=true' -d '
    {
    "indices": "ruan-test-2018-05-24",
    "ignore_unavailable": true,
    @@ -1350,7 +1359,7 @@ $ curl -XPUT -H 'Content-Type: application/json' \
    Verify the Snapshot:

    ```
    $ curl -XGET 'http://elasticsearch:9200/_cat/snapshots/index-backups?v&s=id'
    $ curl -XGET 'http://elasticsearch:9200/_cat/snapshots/bar-backups?v&s=id'
    id status start_epoch start_time end_epoch end_time duration indices successful_shards failed_shards total_shards
    mysnapshot_ruan-test-2018-05-24_1 SUCCESS 1527254411 06:20:11 1527254411 06:20:11 389ms 1 5 0 5
    ```
    @@ -1360,7 +1369,7 @@ mysnapshot_ruan-test-2018-05-24_1 SUCCESS 1527254411 06:20:11 1527254411
    Get the Metadata of the Snapshot:

    ```
    $ curl -XGET 'http://elasticsearch:9200/_snapshot/index-backups/mysnapshot_ruan-test-2018-05-24_1?pretty'
    $ curl -XGET 'http://elasticsearch:9200/_snapshot/bar-backups/mysnapshot_ruan-test-2018-05-24_1?pretty'
    {
    "snapshots" : [ {
    "snapshot" : "mysnapshot_ruan-test-2018-05-24_1",
    @@ -1395,7 +1404,7 @@ $ aws s3 --profile es ls s3://my-es-snapshot-bucket/ | grep VRTF2942QCeqyEaMxPgb
    Execute the Restore:

    ```
    $ curl -XPOST -H 'Content-Type: application/json' 'http://elasticsearch:9200/_snapshot/index-backups/mysnapshot_ruan-test-2018-05-24_1/_restore -d '
    $ curl -XPOST -H 'Content-Type: application/json' 'http://elasticsearch:9200/_snapshot/bar-backups/mysnapshot_ruan-test-2018-05-24_1/_restore -d '
    {
    "indices": "ruan-test-2018-05-24",
    "ignore_unavailable": true,
  3. @ruanbekker ruanbekker revised this gist Jun 16, 2018. 1 changed file with 19 additions and 0 deletions.
    19 changes: 19 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -8,6 +8,8 @@
    - [Nodes Overview](#nodes-overview)
    - [Indices Overview](#indices-overview)
    - [Cluster Maintenance](#cluster-maintenance)
    - [Settings]()
    - [Cluster Settings](#cluster-settings)
    - [Ingest](#ingest-documents-into-elasticsearch)
    - [Mapping](#mapping)
    - [Check Fields in Mappings](#check-fields-in-mappings)
    @@ -315,6 +317,23 @@ $ curl -XGET http://127.0.0.1:9200/_cache/clear
    {"_shards":{"total":21,"successful":15,"failed":0}}
    ```

    # Settings

    ## Cluster Settings

    Search Timeout:

    Global Search Timeout, that applies to all search queries across the entire cluster -> search.default_search_timeout:

    ```
    PUT /_cluster/settings
    {
    "persistent" : {
    "search.default_search_timeout" : "50"
    }
    }
    ```

    # Index Info (Shards, Replicas, Allocation):

    ## Create Index:
  4. @ruanbekker ruanbekker revised this gist May 29, 2018. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1273,7 +1273,7 @@ $ curl -XGET 'elasticsearch:9200/_tasks?detailed=true&actions=*/delete/byquery&p

    ## Elasticsearch S3 Snapshot Repo

    Setup the [S3 Snapshot Repository](https://sysadmins.co.za/aws-elasticsearch-register-s3-repository-for-snapshots-using-the-cli/?rbas_source=gist.github.com?rbas_sourcepage=cheachsheet-elasticsearch.md)
    Setup the [S3 Snapshot Repository](https://sysadmins.co.za/aws-elasticsearch-register-s3-repository-for-snapshots-using-the-cli/?rbas_source=gist.github.com?rbas_sourcepage=cheatsheet-elasticsearch.md)

    Viw the Snapshot Repository:

  5. @ruanbekker ruanbekker revised this gist May 29, 2018. 1 changed file with 132 additions and 1 deletion.
    133 changes: 132 additions & 1 deletion cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -19,6 +19,10 @@
    - [Sort](#sort)
    - [Aggregate]()
    - [Delete](#delete)
    - [Snapshots](#snapshots)
    - [Create Snapshot Repository on S3](#elasticsearch-s3-snapshot-repo)
    - [Create a Snapshot](#elasticsearch-snapshots)
    - [Restore from a Snapshot](#elasticsearch-restore)

    # Resources
    - https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
    @@ -1263,4 +1267,131 @@ $ curl -XGET 'elasticsearch:9200/_tasks?detailed=true&actions=*/delete/byquery&p
    }
    }
    }
    ```
    ```

    # Snapshots

    ## Elasticsearch S3 Snapshot Repo

    Setup the [S3 Snapshot Repository](https://sysadmins.co.za/aws-elasticsearch-register-s3-repository-for-snapshots-using-the-cli/?rbas_source=gist.github.com?rbas_sourcepage=cheachsheet-elasticsearch.md)

    Viw the Snapshot Repository:

    ```
    $ curl -XGET 'http://elasticsearch:9200/_snapshot/index-backups?pretty'
    {
    "index-backups" : {
    "type" : "s3",
    "settings" : {
    "bucket" : "my-es-snapshot-bucket",
    "region" : "eu-west-1",
    "role_arn" : "arn:aws:iam::0123456789012:role/elasticsearch-snapshot-role"
    }
    }
    }
    ```

    ## Elasticsearch Snapshots

    Create a Snapshot named `mysnapshot_ruan-test-2018-05-24_1` of the index: `ruan-test-2018-05-24` and return the exit when the snapshot is done:

    ```
    $ curl -XPUT -H 'Content-Type: application/json' \
    'http://elasticsearch:9200/_snapshot/index-backups/mysnapshot_ruan-test-2018-05-24_1?wait_for_completion=true&pretty=true' -d '
    {
    "indices": "ruan-test-2018-05-24",
    "ignore_unavailable": true,
    "include_global_state": false
    }
    '
    {
    "snapshot" : {
    "snapshot" : "mysnapshot_ruan-test-2018-05-24_1",
    "uuid" : "YRTE5922QCeqyEaMxPqb1A",
    "version_id" : 6000199,
    "version" : "6.0.1",
    "indices" : [ "ruan-test-2018-05-24" ],
    "state" : "SUCCESS",
    "start_time" : "2018-05-25T13:20:11.497Z",
    "start_time_in_millis" : 1527254411497,
    "end_time" : "2018-05-25T13:20:11.886Z",
    "end_time_in_millis" : 1527254411886,
    "duration_in_millis" : 389,
    "failures" : [ ],
    "shards" : {
    "total" : 5,
    "failed" : 0,
    "successful" : 5
    }
    }
    }
    ```

    Verify the Snapshot:

    ```
    $ curl -XGET 'http://elasticsearch:9200/_cat/snapshots/index-backups?v&s=id'
    id status start_epoch start_time end_epoch end_time duration indices successful_shards failed_shards total_shards
    mysnapshot_ruan-test-2018-05-24_1 SUCCESS 1527254411 06:20:11 1527254411 06:20:11 389ms 1 5 0 5
    ```

    ## Elasticsearch Restore

    Get the Metadata of the Snapshot:

    ```
    $ curl -XGET 'http://elasticsearch:9200/_snapshot/index-backups/mysnapshot_ruan-test-2018-05-24_1?pretty'
    {
    "snapshots" : [ {
    "snapshot" : "mysnapshot_ruan-test-2018-05-24_1",
    "uuid" : "YRTE5922QCeqyEaMxPqb1A",
    "version_id" : 6000199,
    "version" : "6.0.1",
    "indices" : [ "ruan-test-2018-05-24" ],
    "state" : "SUCCESS",
    "start_time" : "2018-05-25T13:20:11.497Z",
    "start_time_in_millis" : 1527254411497,
    "end_time" : "2018-05-25T13:20:11.886Z",
    "end_time_in_millis" : 1527254411886,
    "duration_in_millis" : 389,
    "failures" : [ ],
    "shards" : {
    "total" : 5,
    "failed" : 0,
    "successful" : 5
    }
    } ]
    }
    ```

    Inspect the Snapshot on S3:

    ```
    $ aws s3 --profile es ls s3://my-es-snapshot-bucket/ | grep VRTF2942QCeqyEaMxPgb1B
    2018-05-25 15:20:12 90 meta-VRTF2942QCeqyEaMxPgb1B.dat
    2018-05-25 15:20:12 258 snap-VRTF2942QCeqyEaMxPgb1B.dat
    ```

    Execute the Restore:

    ```
    $ curl -XPOST -H 'Content-Type: application/json' 'http://elasticsearch:9200/_snapshot/index-backups/mysnapshot_ruan-test-2018-05-24_1/_restore -d '
    {
    "indices": "ruan-test-2018-05-24",
    "ignore_unavailable": true,
    "include_global_state": false,
    "rename_pattern": "index_(.+)",
    "rename_replacement": "restored_index_$1"
    }
    '
    ```

    or leave out the body for normal restore

    ## Elasticsearch Snapshot Resources:

    - https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html
    - https://www.youtube.com/watch?v=Otl-IcmbiDE
    - https://recology.info/2015/02/elasticsearch-backup-restore/
    - https://medium.com/@rcdexta/periodic-snapshotting-of-elasticsearch-indices-f6b6ca221a0c
  6. @ruanbekker ruanbekker revised this gist Mar 13, 2018. 1 changed file with 106 additions and 0 deletions.
    106 changes: 106 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -384,6 +384,112 @@ my2ndindex 0 p STARTED 0 130b 10.0.2.27 bDWFHuw
    my2ndindex 0 r STARTED 0 130b 10.0.2.24 rNDYCtL
    ```

    ## Create a Index:

    Create a Index with Default Settings:

    ```
    $ curl -XPUT -H 'Content-Type: application/json' 'http://127.0.0.1:9200/ruan-test-2018.03.12'
    ```

    View the settings of the created index:

    ```
    $ curl -XGET 'http://127.0.0.1:9200/ruan-test-2018.03.12/_settings?pretty'
    {
    "ruan-test-2018.03.12" : {
    "settings" : {
    "index" : {
    "creation_date" : "1520929659349",
    "number_of_shards" : "5",
    "number_of_replicas" : "1",
    "uuid" : "EwGz6y7XQkK0ZI08u8qdrQ",
    "version" : {
    "created" : "6000199"
    },
    "provided_name" : "ruan-test-2018.03.12"
    }
    }
    }
    }
    ```

    Remember that primary shard number can only be set on index creation. Change the settings of the index, let's update the index to: 2 replica shards, and the total_fields limit to: 2000

    ```
    $ curl -XPUT -H 'Content-Type: application/json' 'http://127.0.0.1:9200/ruan-test-2018.03.12/_settings' -d '{"number_of_replicas": 0, "index.mapping.total_fields.limit": 2000}'
    ```

    View the changes:

    ```
    $ curl -XGET 'http://127.0.0.1:9200/ruan-test-2018.03.12/_settings?pretty'
    {
    "ruan-test-2018.03.12" : {
    "settings" : {
    "index" : {
    "mapping" : {
    "total_fields" : {
    "limit" : "2000"
    }
    },
    "number_of_shards" : "5",
    "provided_name" : "ruan-test-2018.03.12",
    "creation_date" : "1520929659349",
    "number_of_replicas" : "0",
    "uuid" : "EwGz6y7XQkK0ZI08u8qdrQ",
    "version" : {
    "created" : "6000199"
    }
    }
    }
    }
    }
    ```

    Now, to set the settings on Index Creation:


    ```
    $ curl -XPUT -H 'Content-Type: application/json' 'http://127.0.0.1:9200/ruan-test-2018.03.13' -d '{"settings": {"number_of_replicas": 1, "number_of_shards": 2, "index.mapping.total_fields.limit": 2000}}'
    ```

    Verifying our settings:

    ```
    $ curl -XGET 'http://127.0.0.1:9200/ruan-test-2018.03.13/_settings?pretty'
    {
    "ruan-test-2018.03.13" : {
    "settings" : {
    "index" : {
    "mapping" : {
    "total_fields" : {
    "limit" : "2000"
    }
    },
    "number_of_shards" : "2",
    "provided_name" : "ruan-test-2018.03.13",
    "creation_date" : "1520929638792",
    "number_of_replicas" : "1",
    "uuid" : "hEY8HrlRTFuiYLwKVDAraQ",
    "version" : {
    "created" : "6000199"
    }
    }
    }
    }
    }
    ```

    Viewing our indexes:

    ```
    $ curl -XGET 'http://127.0.0.1:9200/_cat/indices/ruan-test-*?v'
    health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
    green open ruan-test-2018.03.12 EwGz6y7XQkK0ZI08u8qdrQ 5 1 2 0 15.7kb 7.8kb
    green open ruan-test-2018.03.13 hEY8HrlRTFuiYLwKVDAraQ 2 1 0 0 932b 466b
    ```

    ## Ingest Document into Elasticsearch:

    Let's ingest one docuemnt into Elasticsearch, and in this case we will specify the document id as `1`
  7. @ruanbekker ruanbekker revised this gist Mar 12, 2018. 1 changed file with 20 additions and 0 deletions.
    20 changes: 20 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -6,6 +6,7 @@
    - [Index Level](#cluster-health-index-level)
    - [Shard Level](#cluster-health-shard-level)
    - [Nodes Overview](#nodes-overview)
    - [Indices Overview](#indices-overview)
    - [Cluster Maintenance](#cluster-maintenance)
    - [Ingest](#ingest-documents-into-elasticsearch)
    - [Mapping](#mapping)
    @@ -189,12 +190,31 @@ mstWlaoyTM69xhSt-_rZAA 10.0.2.18 10.0.2.18 mstWlao

    ## Indices Overview:

    View all your indices in your cluster:

    ```
    $ curl -XGET http://elasticsearch:9200/_cat/indices?v
    health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
    green open ruan-test CrQZB2L4SaaYCkvYPx5vUA 5 1 38 0 131.9kb 78.6kb
    ```

    View one index:

    ```
    $ curl -XGET 'http://127.0.0.1:9200/_cat/indices/index-name-2018.01.01?v'
    health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
    green open index-name-2018.01.01 Nk8SMQvRSIaNm854bc3Zjg 5 1 395552 0 755.6mb 377.8mb
    ```

    View a range of indices:

    ```
    $ curl -XGET 'https://http://127.0.0.1:9200/_cat/indices/index-name-2018.01*?v'
    health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
    green open index-name-2018.01.19 Vp1EBoeMQkS-a_upLzedhQ 5 1 1220 0 2.6mb 1.3mb
    green open index-name-2018.01.17 hSJMzFJIQrePifCfgb1rOA 5 1 2875 0 3.8mb 1.9mb
    ```

    ## How Many Documents in the ES Cluster (Across all Indices):

    ```
  8. @ruanbekker ruanbekker revised this gist Mar 12, 2018. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -13,6 +13,8 @@
    - [Close API](#open--close-api)
    - [Search](#searching)
    - [Query](#query)
    - [Query by Match](#query-by-match)
    - [Query with Bool](#query-with-bool)
    - [Sort](#sort)
    - [Aggregate]()
    - [Delete](#delete)
  9. @ruanbekker ruanbekker revised this gist Mar 12, 2018. 1 changed file with 3 additions and 3 deletions.
    6 changes: 3 additions & 3 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -892,19 +892,19 @@ curl -XGET 'http://elasticsearch:9200/people/_search?q=age:30&pretty'
    }
    ```

    Query by Term and limit results by 2:
    #### Query by Term and limit results by 2:

    ```
    $ curl -XGET http://127.0.0.1:9200/scrape-sysadmins/_search?pretty -d '{"query": {"term": {"title": "traefik"}}, "size": 2}'
    ```

    Query by Match:
    #### Query by Match:

    ```
    $ curl -XGET http://127.0.0.1:9200/scrape-sysadmins/_search?pretty -d '{"query": {"match": {"title": "traefik"}}, "size": 10}'
    ```

    Query with Bool:
    #### Query with Bool:

    - Check if field exists in index:

  10. @ruanbekker ruanbekker revised this gist Mar 12, 2018. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -9,6 +9,7 @@
    - [Cluster Maintenance](#cluster-maintenance)
    - [Ingest](#ingest-documents-into-elasticsearch)
    - [Mapping](#mapping)
    - [Check Fields in Mappings](#check-fields-in-mappings)
    - [Close API](#open--close-api)
    - [Search](#searching)
    - [Query](#query)
  11. @ruanbekker ruanbekker revised this gist Mar 12, 2018. 1 changed file with 70 additions and 1 deletion.
    71 changes: 70 additions & 1 deletion cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -8,7 +8,8 @@
    - [Nodes Overview](#nodes-overview)
    - [Cluster Maintenance](#cluster-maintenance)
    - [Ingest](#ingest-documents-into-elasticsearch)
    - [Open/Close API](#open--close-api)
    - [Mapping](#mapping)
    - [Close API](#open--close-api)
    - [Search](#searching)
    - [Query](#query)
    - [Sort](#sort)
    @@ -621,6 +622,41 @@ Ingest using the Bulk Api:
    curl -XPOST 'http://elasticsearch:9200/info/_bulk?pretty' --data-binary @info.json
    ```

    # Mapping

    ## Create Mapping
    ## View Mappings

    ## Check Fields in Mappings:

    Check if a field exisists in your mapping:

    ```
    $ curl -XGET 'http://127.0.0.1:9200/index-name-2018.03.01/_mapping/docs/field/company?pretty'
    {
    "index-name-2018.03.01" : {
    "mappings" : {
    "docs" : {
    "company" : {
    "full_name" : "company",
    "mapping" : {
    "company" : {
    "type" : "text",
    "fields" : {
    "keyword" : {
    "type" : "keyword",
    "ignore_above" : 256
    }
    }
    }
    }
    }
    }
    }
    }
    }
    ```

    # Open / Close API:
    - https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-open-close.html

    @@ -867,6 +903,39 @@ Query by Match:
    $ curl -XGET http://127.0.0.1:9200/scrape-sysadmins/_search?pretty -d '{"query": {"match": {"title": "traefik"}}, "size": 10}'
    ```

    Query with Bool:

    - Check if field exists in index:

    ```
    $ curl http://127.0.0.1:9200/test4/_search?pretty -d '{"query": {"bool": {"must": [{"exists": {"field": "name"}}]}}}}'
    {
    "took" : 7,
    "timed_out" : false,
    "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
    },
    "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
    {
    "_index" : "test4",
    "_type" : "docs",
    "_id" : "2",
    "_score" : 1.0,
    "_source" : {
    "id" : "2",
    "name" : "ruan"
    }
    }
    ]
    }
    }
    ```

    - https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query.html

    ## Sort
  12. @ruanbekker ruanbekker revised this gist Mar 12, 2018. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -4,6 +4,7 @@

    - [Cluster Health](#cluster-health)
    - [Index Level](#cluster-health-index-level)
    - [Shard Level](#cluster-health-shard-level)
    - [Nodes Overview](#nodes-overview)
    - [Cluster Maintenance](#cluster-maintenance)
    - [Ingest](#ingest-documents-into-elasticsearch)
  13. @ruanbekker ruanbekker revised this gist Mar 12, 2018. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -3,7 +3,7 @@
    # Shortlinks:

    - [Cluster Health](#cluster-health)
    -- [Index Level](#cluster-health-index-level)
    - [Index Level](#cluster-health-index-level)
    - [Nodes Overview](#nodes-overview)
    - [Cluster Maintenance](#cluster-maintenance)
    - [Ingest](#ingest-documents-into-elasticsearch)
  14. @ruanbekker ruanbekker revised this gist Mar 12, 2018. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -3,6 +3,7 @@
    # Shortlinks:

    - [Cluster Health](#cluster-health)
    -- [Index Level](#cluster-health-index-level)
    - [Nodes Overview](#nodes-overview)
    - [Cluster Maintenance](#cluster-maintenance)
    - [Ingest](#ingest-documents-into-elasticsearch)
  15. @ruanbekker ruanbekker revised this gist Mar 12, 2018. 1 changed file with 8 additions and 0 deletions.
    8 changes: 8 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -14,6 +14,9 @@
    - [Delete](#delete)

    # Resources
    - https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
    - https://www.elastic.co/guide/en/elasticsearch/reference/current/docs.html
    - https://www.elastic.co/blog/managing-time-based-indices-efficiently
    - http://joelabrahamsson.com/elasticsearch-101/
    - https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html
    - https://chatbots.network/logstash-exclude-bots-from-result/
    @@ -958,6 +961,11 @@ $ curl -XPOST http://elasticsearch:9200/products/_search?pretty -d '{"query" : {

    # Delete

    References:

    - [Delete API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete.html)
    - [Delete by Query](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docs-delete-by-query.html)

    ## Delete Index:

    ```
  16. @ruanbekker ruanbekker revised this gist Mar 11, 2018. 1 changed file with 10 additions and 1 deletion.
    11 changes: 10 additions & 1 deletion cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -698,7 +698,8 @@ $ curl -XGET 'http://elasticsearch:9200/people/users/_search?q=age:28&explain&pr
    },
    {
    "value" : 1.0,
    "description" : "queryNorm",
    "description" : "
    Norm",
    "details" : [ ]
    }
    ]
    @@ -855,6 +856,14 @@ Query by Term and limit results by 2:
    $ curl -XGET http://127.0.0.1:9200/scrape-sysadmins/_search?pretty -d '{"query": {"term": {"title": "traefik"}}, "size": 2}'
    ```

    Query by Match:

    ```
    $ curl -XGET http://127.0.0.1:9200/scrape-sysadmins/_search?pretty -d '{"query": {"match": {"title": "traefik"}}, "size": 10}'
    ```

    - https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query.html

    ## Sort
    - https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html

  17. @ruanbekker ruanbekker revised this gist Sep 23, 2017. 1 changed file with 6 additions and 0 deletions.
    6 changes: 6 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -849,6 +849,12 @@ curl -XGET 'http://elasticsearch:9200/people/_search?q=age:30&pretty'
    }
    ```

    Query by Term and limit results by 2:

    ```
    $ curl -XGET http://127.0.0.1:9200/scrape-sysadmins/_search?pretty -d '{"query": {"term": {"title": "traefik"}}, "size": 2}'
    ```

    ## Sort
    - https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html

  18. @ruanbekker ruanbekker revised this gist Aug 24, 2017. 1 changed file with 7 additions and 0 deletions.
    7 changes: 7 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -275,6 +275,13 @@ insertOrder timeInQueue priority source
    1737 1.3s URGENT shard-started ([sysadmins-2017.06.02][3], node[WR3y31g1TnuufpNyrJnQtg], [R], v[91], s[INITIALIZING], a[id=JmrtwtYURMyQF6LspeJXLg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2017-08-11T07:50:56.550Z]]), reason [after recovery (replica) from node [{es01}{6ND8sZ_rTqaL42VdlxyW7Q}{10.79.2.193}{10.79.2.193:9300}]]
    ```

    ## Clear Cache:

    ```
    $ curl -XGET http://127.0.0.1:9200/_cache/clear
    {"_shards":{"total":21,"successful":15,"failed":0}}
    ```

    # Index Info (Shards, Replicas, Allocation):

    ## Create Index:
  19. @ruanbekker ruanbekker revised this gist Aug 14, 2017. 1 changed file with 3 additions and 0 deletions.
    3 changes: 3 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -239,6 +239,9 @@ $ curl -XPUT 'localhost:9200/_cluster/settings?pretty' -d'
    '
    ```

    ## Recovery Resources:
    - https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-recovery.html

    ## Recovering from Node Failure:

    At the moment one of the nodes were down, and up again:
  20. @ruanbekker ruanbekker revised this gist Aug 12, 2017. 1 changed file with 33 additions and 0 deletions.
    33 changes: 33 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -239,6 +239,39 @@ $ curl -XPUT 'localhost:9200/_cluster/settings?pretty' -d'
    '
    ```

    ## Recovering from Node Failure:

    At the moment one of the nodes were down, and up again:

    ```
    $ curl -XGET http://127.0.0.1:9200/_cat/allocation?v
    shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
    290 54.1mb 1gb 20mb 1gb 98 10.79.2.193 10.79.2.193 es01
    151 43.5mb 1gb 11.9gb 13gb 8 10.79.3.171 10.79.3.171 es02
    139 UNASSIGNED
    ```

    ## Recovery API:

    ```
    $ curl -XGET http://127.0.0.1:9200/_cat/recovery?v
    index shard time type stage source_host target_host repository snapshot files files_percent bytes bytes_percent total_files total_bytes translog translog_percent total_translog
    sysadmins-2017.06.19 0 1512 replica done 10.79.2.193 10.79.3.171 n/a n/a 31 100.0% 340020 100.0% 31 340020 0 100.0% 0
    sysadmins-2017.06.19 0 7739 store done 10.79.2.193 10.79.2.193 n/a n/a 0 100.0% 0 100.0% 31 340020 0 100.0% 0
    sysadmins-2017.06.19 1 2592 relocation done 10.79.2.193 10.79.3.171 n/a n/a 13 100.0% 246229 100.0% 13 246229 0 100.0% 0
    sysadmins-2017.06.19 1 613 replica done 10.79.3.171 10.79.2.193 n/a n/a 0 0.0% 0 0.0% 0 0 0 100.0% 0
    ```

    ## Pending Tasks:

    ```
    $ curl -XGET http://127.0.0.1:9200/_cat/pending_tasks?v
    insertOrder timeInQueue priority source
    1736 1.8s URGENT shard-started ([sysadmins-2017.06.02][2], node[WR3y31g1TnuufpNyrJnQtg], [R], v[91], s[INITIALIZING], a[id=wVTDn4nFSKKxvi07cU0uCg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2017-08-11T07:50:56.550Z]]), reason [after recovery (replica) from node [{es01}{6ND8sZ_rTqaL42VdlxyW7Q}{10.79.2.193}{10.79.2.193:9300}]]
    1737 1.3s URGENT shard-started ([sysadmins-2017.06.02][3], node[WR3y31g1TnuufpNyrJnQtg], [R], v[91], s[INITIALIZING], a[id=JmrtwtYURMyQF6LspeJXLg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2017-08-11T07:50:56.550Z]]), reason [after recovery (replica) from node [{es01}{6ND8sZ_rTqaL42VdlxyW7Q}{10.79.2.193}{10.79.2.193:9300}]]
    ```

    # Index Info (Shards, Replicas, Allocation):

    ## Create Index:
  21. @ruanbekker ruanbekker revised this gist Aug 12, 2017. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -16,7 +16,7 @@
    # Resources
    - http://joelabrahamsson.com/elasticsearch-101/
    - https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html

    - https://chatbots.network/logstash-exclude-bots-from-result/

    # Overview

  22. @ruanbekker ruanbekker revised this gist Aug 12, 2017. 1 changed file with 6 additions and 5 deletions.
    11 changes: 6 additions & 5 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -6,11 +6,12 @@
    - [Nodes Overview](#nodes-overview)
    - [Cluster Maintenance](#cluster-maintenance)
    - [Ingest](#ingest-documents-into-elasticsearch)
    - [](#open--close-api)
    - [](#searching)
    - [](#query)
    - [](#sort)
    - [](#delete)
    - [Open/Close API](#open--close-api)
    - [Search](#searching)
    - [Query](#query)
    - [Sort](#sort)
    - [Aggregate]()
    - [Delete](#delete)

    # Resources
    - http://joelabrahamsson.com/elasticsearch-101/
  23. @ruanbekker ruanbekker revised this gist Aug 12, 2017. 1 changed file with 5 additions and 1 deletion.
    6 changes: 5 additions & 1 deletion cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -6,7 +6,11 @@
    - [Nodes Overview](#nodes-overview)
    - [Cluster Maintenance](#cluster-maintenance)
    - [Ingest](#ingest-documents-into-elasticsearch)

    - [](#open--close-api)
    - [](#searching)
    - [](#query)
    - [](#sort)
    - [](#delete)

    # Resources
    - http://joelabrahamsson.com/elasticsearch-101/
  24. @ruanbekker ruanbekker revised this gist Aug 12, 2017. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -4,6 +4,8 @@

    - [Cluster Health](#cluster-health)
    - [Nodes Overview](#nodes-overview)
    - [Cluster Maintenance](#cluster-maintenance)
    - [Ingest](#ingest-documents-into-elasticsearch)


    # Resources
  25. @ruanbekker ruanbekker revised this gist Aug 12, 2017. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -3,6 +3,8 @@
    # Shortlinks:

    - [Cluster Health](#cluster-health)
    - [Nodes Overview](#nodes-overview)


    # Resources
    - http://joelabrahamsson.com/elasticsearch-101/
  26. @ruanbekker ruanbekker revised this gist Aug 12, 2017. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -2,7 +2,7 @@

    # Shortlinks:

    - [#]()
    - [Cluster Health](#cluster-health)

    # Resources
    - http://joelabrahamsson.com/elasticsearch-101/
  27. @ruanbekker ruanbekker revised this gist Aug 12, 2017. 1 changed file with 72 additions and 0 deletions.
    72 changes: 72 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,5 +1,9 @@
    (still a work-in-progress)

    # Shortlinks:

    - [#]()

    # Resources
    - http://joelabrahamsson.com/elasticsearch-101/
    - https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html
    @@ -916,4 +920,72 @@ curl -XPOST 'http://elasticsearch:9200/weblogs/_delete_by_query?pretty' -d' {"qu
    "throttled_until_millis" : 0,
    "failures" : [ ]
    }
    ```

    If routing is provided, then the routing is copied to the scroll query, limiting the process to the shards that match that routing value:

    ```
    $ curl -XPOST 'http://elasticsearch:9200/people/_delete_by_query?routing=1
    {
    "query": {
    "range" : {
    "age" : {
    "gte" : 10
    }
    }
    }
    }
    ```

    By default _delete_by_query uses scroll batches of 1000. You can change the batch size with the scroll_size URL parameter:

    ```
    $ curl -XPOST 'http://elasticsearch:9200/weblogs/_delete_by_query?scroll_size=5000
    {
    "query": {
    "term": {
    "category": "docker"
    }
    }
    }
    ```

    ## Delete Stats:

    ```
    $ curl -XGET 'elasticsearch:9200/_tasks?detailed=true&actions=*/delete/byquery&pretty'
    {
    "nodes" : {
    "s5A2CoRWrwKf512z6NEscF" : {
    "name" : "r4A5VoT",
    "transport_address" : "127.0.0.1:9300",
    "host" : "127.0.0.1",
    "ip" : "127.0.0.1:9300",
    "attributes" : {
    "testattr" : "test",
    "portsfile" : "true"
    },
    "tasks" : {
    "s5A2CoRWrwKf512z6NEscF" : {
    "node" : "s5A2CoRWrwKf512z6NEscF",
    "id" : 36619,
    "type" : "transport",
    "action" : "indices:data/write/delete/byquery",
    "status" : {
    "total" : 6154,
    "updated" : 0,
    "created" : 0,
    "deleted" : 3500,
    "batches" : 36,
    "version_conflicts" : 0,
    "noops" : 0,
    "retries": 0,
    "throttled_millis": 0
    },
    "description" : ""
    }
    }
    }
    }
    }
    ```
  28. @ruanbekker ruanbekker revised this gist Aug 11, 2017. 1 changed file with 16 additions and 0 deletions.
    16 changes: 16 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -210,6 +210,22 @@ shards disk.indices disk.used disk.avail disk.total disk.percent host ip
    5 3.1mb 15.5gb 30.1gb 45.7gb 34 10.0.2.22 10.0.2.22 ifgr6ym
    ```

    # Cluster Maintenance:

    ## Decomission Node from Shard Allocation

    This will move shards from the mentioned node

    ```
    $ curl -XPUT 'localhost:9200/_cluster/settings?pretty' -d'
    {
    "transient" : {
    "cluster.routing.allocation.exclude._ip" : "10.0.0.1"
    }
    }
    '
    ```

    # Index Info (Shards, Replicas, Allocation):

    ## Create Index:
  29. @ruanbekker ruanbekker revised this gist Aug 11, 2017. 1 changed file with 115 additions and 0 deletions.
    115 changes: 115 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -9,6 +9,9 @@

    ## Cluster Health:

    Resource:
    - https://www.elastic.co/guide/en/elasticsearch/guide/current/_cluster_health.html

    ```
    $ curl -XGET http://elasticsearch:9200/_cluster/health?pretty
    {
    @@ -30,6 +33,118 @@ $ curl -XGET http://elasticsearch:9200/_cluster/health?pretty
    }
    ```

    ## Cluster Health: Index Level:

    ```
    $ curl -XGET 'http://elasticsearch:9200/_cluster/health?level=indices&pretty'
    {
    "cluster_name" : "swarm-elasticsearch",
    "status" : "red",
    "timed_out" : false,
    "number_of_nodes" : 5,
    "number_of_data_nodes" : 5,
    "active_primary_shards" : 44,
    "active_shards" : 44,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 64,
    "delayed_unassigned_shards" : 0,
    "number_of_pending_tasks" : 0,
    "number_of_in_flight_fetch" : 0,
    "task_max_waiting_in_queue_millis" : 0,
    "active_shards_percent_as_number" : 40.74074074074074,
    "indices" : {
    "test" : {
    "status" : "yellow",
    "number_of_shards" : 5,
    "number_of_replicas" : 1,
    "active_primary_shards" : 5,
    "active_shards" : 5,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 5
    }
    }
    }
    ```

    ## Cluster Health: Shard Level:

    ```
    curl -XGET 'http://elasticsearch:9200/_cluster/health?level=shards&pretty'
    {
    "cluster_name" : "swarm-elasticsearch",
    "status" : "red",
    "timed_out" : false,
    "number_of_nodes" : 5,
    "number_of_data_nodes" : 5,
    "active_primary_shards" : 44,
    "active_shards" : 44,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 64,
    "delayed_unassigned_shards" : 0,
    "number_of_pending_tasks" : 0,
    "number_of_in_flight_fetch" : 0,
    "task_max_waiting_in_queue_millis" : 0,
    "active_shards_percent_as_number" : 40.74074074074074,
    "indices" : {
    "test" : {
    "status" : "yellow",
    "number_of_shards" : 5,
    "number_of_replicas" : 1,
    "active_primary_shards" : 5,
    "active_shards" : 5,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 5,
    "shards" : {
    "0" : {
    "status" : "yellow",
    "primary_active" : true,
    "active_shards" : 1,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 1
    },
    "1" : {
    "status" : "yellow",
    "primary_active" : true,
    "active_shards" : 1,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 1
    },
    "2" : {
    "status" : "yellow",
    "primary_active" : true,
    "active_shards" : 1,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 1
    },
    "3" : {
    "status" : "yellow",
    "primary_active" : true,
    "active_shards" : 1,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 1
    },
    "4" : {
    "status" : "yellow",
    "primary_active" : true,
    "active_shards" : 1,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 1
    }
    }
    }
    }
    }
    ```

    ## Nodes Overview:

    ```
  30. @ruanbekker ruanbekker revised this gist Aug 10, 2017. 1 changed file with 5 additions and 0 deletions.
    5 changes: 5 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,5 +1,10 @@
    (still a work-in-progress)

    # Resources
    - http://joelabrahamsson.com/elasticsearch-101/
    - https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html


    # Overview

    ## Cluster Health: