Skip to content

Instantly share code, notes, and snippets.

@naviat

naviat/cheatsheet-elasticsearch.md

Forked from ruanbekker/cheatsheet-elasticsearch.md
Last active August 18, 2021 06:23
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save naviat/9afd4a52de6cacdb42e8fc64e88697f0 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save naviat/9afd4a52de6cacdb42e8fc64e88697f0 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. naviat revised this gist Aug 18, 2021. 1 changed file with 54 additions and 54 deletions.
    108 changes: 54 additions & 54 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -40,7 +40,7 @@ Resource:
    - https://www.elastic.co/guide/en/elasticsearch/guide/current/_cluster_health.html

    ```
    $ curl -XGET http://elasticsearch:9200/_cluster/health?pretty
    $ curl -XGET http://localhost:9200/_cluster/health?pretty
    {
    "cluster_name" : "docker-cluster",
    "status" : "green",
    @@ -63,7 +63,7 @@ $ curl -XGET http://elasticsearch:9200/_cluster/health?pretty
    ## Cluster Health: Index Level:

    ```
    $ curl -XGET 'http://elasticsearch:9200/_cluster/health?level=indices&pretty'
    $ curl -XGET 'http://localhost:9200/_cluster/health?level=indices&pretty'
    {
    "cluster_name" : "swarm-elasticsearch",
    "status" : "red",
    @@ -98,7 +98,7 @@ $ curl -XGET 'http://elasticsearch:9200/_cluster/health?level=indices&pretty'
    ## Cluster Health: Shard Level:

    ```
    curl -XGET 'http://elasticsearch:9200/_cluster/health?level=shards&pretty'
    curl -XGET 'http://localhost:9200/_cluster/health?level=shards&pretty'
    {
    "cluster_name" : "swarm-elasticsearch",
    "status" : "red",
    @@ -175,7 +175,7 @@ curl -XGET 'http://elasticsearch:9200/_cluster/health?level=shards&pretty'
    ## Nodes Overview:

    ```
    $ curl -XGET http://elasticsearch:9200/_cat/nodes?v
    $ curl -XGET http://localhost:9200/_cat/nodes?v
    ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
    10.0.2.28 21 92 2 0.55 0.45 0.38 mdi - ea1q921
    10.0.2.24 27 95 5 0.17 0.24 0.22 mdi - rNDYCtL
    @@ -187,7 +187,7 @@ ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master
    ## Who is Master:

    ```
    $ curl -XGET http://elasticsearch:9200/_cat/master?v
    $ curl -XGET http://localhost:9200/_cat/master?v
    id host ip node
    mstWlaoyTM69xhSt-_rZAA 10.0.2.18 10.0.2.18 mstWlao
    ```
    @@ -197,7 +197,7 @@ mstWlaoyTM69xhSt-_rZAA 10.0.2.18 10.0.2.18 mstWlao
    View all your indices in your cluster:

    ```
    $ curl -XGET http://elasticsearch:9200/_cat/indices?v
    $ curl -XGET http://localhost:9200/_cat/indices?v
    health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
    green open ruan-test CrQZB2L4SaaYCkvYPx5vUA 5 1 38 0 131.9kb 78.6kb
    ```
    @@ -222,15 +222,15 @@ green open index-name-2018.01.17 hSJMzFJIQrePifCfgb1rOA 5 1 2875
    ## How Many Documents in the ES Cluster (Across all Indices):

    ```
    $ curl -XGET http://elasticsearch:9200/_cat/count?v
    $ curl -XGET http://localhost:9200/_cat/count?v
    epoch timestamp count
    1502288579 14:22:59 38
    ```

    ## Shards Info per Index:

    ```
    $ curl -XGET http://elasticsearch:9200/_cat/shards/ruan-test?v
    $ curl -XGET http://localhost:9200/_cat/shards/ruan-test?v
    index shard prirep state docs store ip node
    ruan-test 3 r STARTED 10 6.9kb 10.0.2.28 ea1q921
    ruan-test 3 p STARTED 10 6.9kb 10.0.2.24 rNDYCtL
    @@ -247,7 +247,7 @@ ruan-test 0 r STARTED 4 12.9kb 10.0.2.27 bDWFHuw
    ## Shard Allocation per Node:

    ```
    $ curl -XGET http://elasticsearch:9200/_cat/allocation?v
    $ curl -XGET http://localhost:9200/_cat/allocation?v
    shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
    4 60.6mb 15.7gb 29.9gb 45.7gb 34 10.0.2.24 10.0.2.24 rNDYCtL
    4 48.3kb 16.7gb 28.9gb 45.7gb 36 10.0.2.18 10.0.2.18 mstWlao
    @@ -322,14 +322,14 @@ $ curl -XGET http://127.0.0.1:9200/_cache/clear
    When you create an Index, 5 Primary Shards and 1 Replica Shard will assigned to the Index by Default.

    ```
    $ curl -XPUT http://elasticsearch:9200/my2ndindex
    $ curl -XPUT http://localhost:9200/my2ndindex
    {"acknowledged":true,"shards_acknowledged":true}
    ```

    To verify the behavior:

    ```
    curl -XGET -u http://elasticsearch:9200/_cat/indices?v
    curl -XGET -u http://localhost:9200/_cat/indices?v
    health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
    green open my2ndindex V32G9IOoTF6uq0DuNUIAMA 5 1 0 0 1.2kb 650b
    green open ruan-test CrQZB2L4SaaYCkvYPx5vUA 5 1 38 0 131.9kb 78.6kb
    @@ -343,7 +343,7 @@ You can ONLY set the number primary shards on index creation.
    While having 5 prmary shards and 1 replica shard, let's have a look at it:

    ```
    $ curl -XGET http://elasticsearch:9200/_cat/shards/my2ndindex?v
    $ curl -XGET http://localhost:9200/_cat/shards/my2ndindex?v
    index shard prirep state docs store ip node
    my2ndindex 3 p STARTED 0 130b 10.0.2.22 ifgr6ym
    my2ndindex 3 r STARTED 0 130b 10.0.2.27 bDWFHuw
    @@ -362,14 +362,14 @@ my2ndindex 0 r STARTED 0 130b 10.0.2.24 rNDYCtL
    Let's change the replica shard number to 2, meaning each primary shard will have 2 replica shards:

    ```
    $ curl -XPUT http://elasticsearch:9200/my2ndindex/_settings -d '{"settings": {"index": {"number_of_replicas": 2}}}'
    $ curl -XPUT http://localhost:9200/my2ndindex/_settings -d '{"settings": {"index": {"number_of_replicas": 2}}}'
    {"acknowledged":true}
    ```

    Let's have a look at the shard info after we have increased the replica shard number:

    ```
    $ curl -XGET http://elasticsearch:9200/_cat/shards/my2ndindex?v
    $ curl -XGET http://localhost:9200/_cat/shards/my2ndindex?v
    index shard prirep state docs store ip node
    my2ndindex 3 r STARTED 0 130b 10.0.2.28 ea1q921
    my2ndindex 3 p STARTED 0 130b 10.0.2.22 ifgr6ym
    @@ -499,22 +499,22 @@ green open ruan-test-2018.03.13 hEY8HrlRTFuiYLwKVDAraQ 2 1 0
    Let's ingest one docuemnt into Elasticsearch, and in this case we will specify the document id as `1`

    ```
    $ curl -XPUT http://elasticsearch:9200/my2ndindex/docs/1 -d '{"identity": {"name": "ruan", "surname": "bekker"}}'
    $ curl -XPUT http://localhost:9200/my2ndindex/docs/1 -d '{"identity": {"name": "ruan", "surname": "bekker"}}'
    {"_index":"my2ndindex","_type":"docs","_id":"1","_version":1,"result":"created","_shards":{"total":3,"successful":3,"failed":0},"created":true}
    ```

    View the index info:

    ```
    $ curl -XGET 'http://elasticsearch:9200/_cat/indices/my*?v'
    $ curl -XGET 'http://localhost:9200/_cat/indices/my*?v'
    health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
    green open my2ndindex V32G9IOoTF6uq0DuNUIAMA 5 2 1 0 13kb 4.3kb
    ```

    View the Shard information on our Index:

    ```
    $ curl -XGET http://elasticsearch:9200/_cat/shards/my2ndindex?v
    $ curl -XGET http://localhost:9200/_cat/shards/my2ndindex?v
    index shard prirep state docs store ip node
    my2ndindex 3 r STARTED 1 3.9kb 10.0.2.28 ea1q921
    my2ndindex 3 p STARTED 1 3.9kb 10.0.2.22 ifgr6ym
    @@ -562,22 +562,22 @@ To get the status back to green:
    Increase the replica shards to `5`:

    ```
    $ curl -XPUT http://elasticsearch:9200/my2ndindex/_settings -d '{"settings": {"number_of_replicas": 5}}'
    $ curl -XPUT http://localhost:9200/my2ndindex/_settings -d '{"settings": {"number_of_replicas": 5}}'
    {"acknowledged":true}
    ```

    Verify the Indices Overview:

    ```
    $ curl -XGET 'http://elasticsearch:9200/_cat/indices/my*?v'
    $ curl -XGET 'http://localhost:9200/_cat/indices/my*?v'
    health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
    yellow open my2ndindex V32G9IOoTF6uq0DuNUIAMA 5 5 1 0 22.2kb 4.4kb
    ```

    We can see that we have a YELLOW status, for more info let's have a look at the shards overview:

    ```
    $ curl -XGET http://elasticsearch:9200/_cat/shards/my2ndindex?v
    $ curl -XGET http://localhost:9200/_cat/shards/my2ndindex?v
    index shard prirep state docs store ip node
    my2ndindex 3 r STARTED 1 3.9kb 10.0.2.28 ea1q921
    my2ndindex 3 p STARTED 1 3.9kb 10.0.2.22 ifgr6ym
    @@ -614,7 +614,7 @@ my2ndindex 0 r UNASSIGNED
    Also, when we look at the allocation api, we can see that we have 5 shards that is unassigned:

    ```
    $ curl -XGET http://elasticsearch:9200/_cat/allocation?v
    $ curl -XGET http://localhost:9200/_cat/allocation?v
    shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
    9 59.2kb 16.8gb 28.8gb 45.7gb 36 10.0.2.18 10.0.2.18 mstWlao
    10 61.2mb 16.8gb 28.8gb 45.7gb 36 10.0.2.27 10.0.2.27 bDWFHuw
    @@ -629,14 +629,14 @@ shards disk.indices disk.used disk.avail disk.total disk.percent host ip
    Let's create an index with 10 primary shards and a replica count of 2:

    ```
    $ curl -XPUT http://elasticsearch:9200/my3rdindex -d '{"settings": {"index": {"number_of_shards": 10, "number_of_replicas": 2}}}'
    $ curl -XPUT http://localhost:9200/my3rdindex -d '{"settings": {"index": {"number_of_shards": 10, "number_of_replicas": 2}}}'
    {"acknowledged":true,"shards_acknowledged":true}/ #
    ```

    Verify:

    ```
    $ curl -XGET 'http://elasticsearch:9200/_cat/indices/my*?v'
    $ curl -XGET 'http://localhost:9200/_cat/indices/my*?v'
    health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
    green open my3rdindex ljovpse0RzCB5INxUBLBYg 10 2 0 0 2.4kb 650b
    green open my2ndindex V32G9IOoTF6uq0DuNUIAMA 5 2 1 0 13.3kb 4.4kb
    @@ -645,7 +645,7 @@ green open my2ndindex V32G9IOoTF6uq0DuNUIAMA 5 2 1 0
    View the shard info on our index:

    ```
    $ curl -XGET http://elasticsearch:9200/_cat/shards/my3rdindex?v
    $ curl -XGET http://localhost:9200/_cat/shards/my3rdindex?v
    index shard prirep state docs store ip node
    my3rdindex 8 r STARTED 0 130b 10.0.2.28 ea1q921
    my3rdindex 8 p STARTED 0 130b 10.0.2.22 ifgr6ym
    @@ -682,14 +682,14 @@ my3rdindex 0 r STARTED 0 130b 10.0.2.24 rNDYCtL
    Take note, with the configuration as above your index that you created will have 30 shards in your cluster:

    ```
    $ curl -s -XGET 'http://elasticsearch:9200/_cat/shards/my3rdindex?v' | grep -v 'node' | wc -l
    $ curl -s -XGET 'http://localhost:9200/_cat/shards/my3rdindex?v' | grep -v 'node' | wc -l
    30
    ```

    Number of Primary Shards per Node:

    ```
    $ curl -s -XGET 'http://elasticsearch:9200/_cat/shards/my3rdindex?v' | grep 'p STARTED' | awk '{print $7}' | sort | uniq -c
    $ curl -s -XGET 'http://localhost:9200/_cat/shards/my3rdindex?v' | grep 'p STARTED' | awk '{print $7}' | sort | uniq -c
    2 10.0.2.18
    3 10.0.2.22
    1 10.0.2.24
    @@ -717,15 +717,15 @@ When you do a `PUT` request, you need to specify the `id` of the document:
    Let's ingest a simple document with a random string as the document id:

    ```
    $ curl -XPUT http://elasticsearch:9200/people/users/abcd -d '{"name", "james", "age": 28}'
    $ curl -XPUT http://localhost:9200/people/users/abcd -d '{"name", "james", "age": 28}'
    {"_index":"people","_type":"users","_id":"abcd","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"created":true}
    ```

    If we have to repeat the same request with the same `id`, the docuement will be overwritten, ES will create a new document if
    the `id` is not present.

    ```
    $ curl -XPUT http://elasticsearch:9200/people/users/abcd -d '{"name": "james", "age": 28}'
    $ curl -XPUT http://localhost:9200/people/users/abcd -d '{"name": "james", "age": 28}'
    {"_index":"people","_type":"users","_id":"abcd","_version":2,"result":"updated","_shards":{"total":2,"successful":2,"failed":0},"created":false}
    ```

    @@ -734,7 +734,7 @@ $ curl -XPUT http://elasticsearch:9200/people/users/abcd -d '{"name": "james", "
    When you do a `POST` request, the service will automatically assign a `id` for your docuemt:

    ```
    $ curl -XPOST http://elasticsearch:9200/people/users/ -d '{"name": "susan", "age: 30}'
    $ curl -XPOST http://localhost:9200/people/users/ -d '{"name": "susan", "age: 30}'
    {"_index":"people","_type":"users","_id":"AV3H_9q6AH1phg1wCfDW","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"created":true}
    ```

    @@ -752,7 +752,7 @@ Our Sample Data: `info.json`:
    Ingest using the Bulk Api:

    ```
    curl -XPOST 'http://elasticsearch:9200/info/_bulk?pretty' --data-binary @info.json
    curl -XPOST 'http://localhost:9200/info/_bulk?pretty' --data-binary @info.json
    ```

    # Mapping
    @@ -796,21 +796,21 @@ $ curl -XGET 'http://127.0.0.1:9200/index-name-2018.03.01/_mapping/docs/field/co
    ## Close Index:

    ```
    $ curl -XPOST http://elasticsearch:9200/people/_close
    $ curl -XPOST http://localhost:9200/people/_close
    {"acknowledged":true}
    ```

    Trying to ingest while the index is closed:

    ```
    $ curl -XPOST http://elasticsearch:9200/people/users/ -d '{"name": "susan", "age": 30}'
    $ curl -XPOST http://localhost:9200/people/users/ -d '{"name": "susan", "age": 30}'
    {"error":{"root_cause":[{"type":"index_closed_exception","reason":"closed","index_uuid":"Yt31-EAwTOa-a6duElYRsQ","index":"people"}],"type":"index_closed_exception","reason":"closed","index_uuid":"Yt31-EAwTOa-a6duElYRsQ","index":"people"},"status":403}
    ```

    ## Open Index:

    ```
    $ curl -XPOST http://elasticsearch:9200/people/_open
    $ curl -XPOST http://localhost:9200/people/_open
    ```

    # Searching
    @@ -820,7 +820,7 @@ $ curl -XPOST http://elasticsearch:9200/people/_open
    We can get the document by passing the document `id`:

    ```
    $ curl -XGET http://elasticsearch:9200/people/users/abcd?pretty
    $ curl -XGET http://localhost:9200/people/users/abcd?pretty
    {
    "_index" : "people",
    "_type" : "users",
    @@ -837,7 +837,7 @@ $ curl -XGET http://elasticsearch:9200/people/users/abcd?pretty
    ## Determine which Shard a Document Reside on:

    ```
    $ curl -XGET 'http://elasticsearch:9200/people/users/_search?q=age:28&explain&pretty'
    $ curl -XGET 'http://localhost:9200/people/users/_search?q=age:28&explain&pretty'
    {
    "took" : 73,
    "timed_out" : false,
    @@ -890,7 +890,7 @@ $ curl -XGET 'http://elasticsearch:9200/people/users/_search?q=age:28&explain&pr
    Lets do a search on our index:

    ```
    $ curl -XGET http://elasticsearch:9200/people/_search?pretty
    $ curl -XGET http://localhost:9200/people/_search?pretty
    {
    "took" : 29,
    "timed_out" : false,
    @@ -931,7 +931,7 @@ $ curl -XGET http://elasticsearch:9200/people/_search?pretty
    By default the Search API returns 10 items, which can be changed using `size`

    ```
    curl -XGET 'http://elasticsearch:9200/shakespeare/_search?size=3&pretty'
    curl -XGET 'http://localhost:9200/shakespeare/_search?size=3&pretty'
    {
    "took" : 25,
    "timed_out" : false,
    @@ -996,7 +996,7 @@ curl -XGET 'http://elasticsearch:9200/shakespeare/_search?size=3&pretty'
    Query our index for people with the age of 28:

    ```
    curl -XGET 'http://elasticsearch:9200/people/_search?q=age:30&pretty'
    curl -XGET 'http://localhost:9200/people/_search?q=age:30&pretty'
    {
    "took" : 25,
    "timed_out" : false,
    @@ -1079,16 +1079,16 @@ Sort Per Field:
    Ingest a couple of example documents:

    ```
    $ curl -XPUT http://elasticsearch:9200/products/items/1 -d '{"product": "chocolate", "price": [20, 4]}'
    $ curl -XPUT http://elasticsearch:9200/products/items/2 -d '{"product": "apples", "price": [28, 6]}'
    $ curl -XPUT http://elasticsearch:9200/products/items/3 -d '{"product": "bananas", "price": [28, 22, 23, 20]}'
    $ curl -XPUT http://elasticsearch:9200/products/items/4 -d '{"product": "chips", "price": [14, 24, 22, 12]}'
    $ curl -XPUT http://localhost:9200/products/items/1 -d '{"product": "chocolate", "price": [20, 4]}'
    $ curl -XPUT http://localhost:9200/products/items/2 -d '{"product": "apples", "price": [28, 6]}'
    $ curl -XPUT http://localhost:9200/products/items/3 -d '{"product": "bananas", "price": [28, 22, 23, 20]}'
    $ curl -XPUT http://localhost:9200/products/items/4 -d '{"product": "chips", "price": [14, 24, 22, 12]}'
    ```

    Run a Sort Query on the term `bananas`, and show the `average` price. We can also use `min, max, avg, sum`:

    ```
    $ curl -XPOST http://elasticsearch:9200/products/_search?pretty -d '{"query" : {"term" : { "product" : "bananas" }}, "sort" : [{"price" : {"order" : "asc", "mode" : "avg"}}]}'
    $ curl -XPOST http://localhost:9200/products/_search?pretty -d '{"query" : {"term" : { "product" : "bananas" }}, "sort" : [{"price" : {"order" : "asc", "mode" : "avg"}}]}'
    {
    "took" : 9,
    "timed_out" : false,
    @@ -1127,7 +1127,7 @@ $ curl -XPOST http://elasticsearch:9200/products/_search?pretty -d '{"query" : {
    Running the same, but wanting to see the sum of all the prices:

    ```
    $ curl -XPOST http://elasticsearch:9200/products/_search?pretty -d '{"query" : {"term" : { "product" : "bananas" }}, "sort" : [{"price" : {"order" : "asc", "mode" : "sum"}}]}'
    $ curl -XPOST http://localhost:9200/products/_search?pretty -d '{"query" : {"term" : { "product" : "bananas" }}, "sort" : [{"price" : {"order" : "asc", "mode" : "sum"}}]}'
    {
    "took" : 34,
    "timed_out" : false,
    @@ -1173,15 +1173,15 @@ References:
    ## Delete Index:

    ```
    $ curl -XDELETE http://elasticsearch:9200/myindex
    $ curl -XDELETE http://localhost:9200/myindex
    ```

    ## Delete Documents on Query:

    We would like to delete all documents that has `"os_name": "Windows 10"`

    ```
    curl -XPOST 'http://elasticsearch:9200/weblogs/_delete_by_query?pretty' -d' {"query": {"match": {"os_name": "Windows 10"}}}'
    curl -XPOST 'http://localhost:9200/weblogs/_delete_by_query?pretty' -d' {"query": {"match": {"os_name": "Windows 10"}}}'
    {
    "took" : 1217,
    "timed_out" : false,
    @@ -1204,7 +1204,7 @@ curl -XPOST 'http://elasticsearch:9200/weblogs/_delete_by_query?pretty' -d' {"qu
    If routing is provided, then the routing is copied to the scroll query, limiting the process to the shards that match that routing value:

    ```
    $ curl -XPOST 'http://elasticsearch:9200/people/_delete_by_query?routing=1
    $ curl -XPOST 'http://localhost:9200/people/_delete_by_query?routing=1
    {
    "query": {
    "range" : {
    @@ -1219,7 +1219,7 @@ $ curl -XPOST 'http://elasticsearch:9200/people/_delete_by_query?routing=1
    By default _delete_by_query uses scroll batches of 1000. You can change the batch size with the scroll_size URL parameter:

    ```
    $ curl -XPOST 'http://elasticsearch:9200/weblogs/_delete_by_query?scroll_size=5000
    $ curl -XPOST 'http://localhost:9200/weblogs/_delete_by_query?scroll_size=5000
    {
    "query": {
    "term": {
    @@ -1232,7 +1232,7 @@ $ curl -XPOST 'http://elasticsearch:9200/weblogs/_delete_by_query?scroll_size=50
    ## Delete Stats:

    ```
    $ curl -XGET 'elasticsearch:9200/_tasks?detailed=true&actions=*/delete/byquery&pretty'
    $ curl -XGET 'localhost:9200/_tasks?detailed=true&actions=*/delete/byquery&pretty'
    {
    "nodes" : {
    "s5A2CoRWrwKf512z6NEscF" : {
    @@ -1278,7 +1278,7 @@ Setup the [S3 Snapshot Repository](https://sysadmins.co.za/aws-elasticsearch-reg
    Viw the Snapshot Repository:

    ```
    $ curl -XGET 'http://elasticsearch:9200/_snapshot/index-backups?pretty'
    $ curl -XGET 'http://localhost:9200/_snapshot/index-backups?pretty'
    {
    "index-backups" : {
    "type" : "s3",
    @@ -1297,7 +1297,7 @@ Create a Snapshot named `mysnapshot_ruan-test-2018-05-24_1` of the index: `ruan-

    ```
    $ curl -XPUT -H 'Content-Type: application/json' \
    'http://elasticsearch:9200/_snapshot/index-backups/mysnapshot_ruan-test-2018-05-24_1?wait_for_completion=true&pretty=true' -d '
    'http://localhost:9200/_snapshot/index-backups/mysnapshot_ruan-test-2018-05-24_1?wait_for_completion=true&pretty=true' -d '
    {
    "indices": "ruan-test-2018-05-24",
    "ignore_unavailable": true,
    @@ -1331,7 +1331,7 @@ $ curl -XPUT -H 'Content-Type: application/json' \
    Verify the Snapshot:

    ```
    $ curl -XGET 'http://elasticsearch:9200/_cat/snapshots/index-backups?v&s=id'
    $ curl -XGET 'http://localhost:9200/_cat/snapshots/index-backups?v&s=id'
    id status start_epoch start_time end_epoch end_time duration indices successful_shards failed_shards total_shards
    mysnapshot_ruan-test-2018-05-24_1 SUCCESS 1527254411 06:20:11 1527254411 06:20:11 389ms 1 5 0 5
    ```
    @@ -1341,7 +1341,7 @@ mysnapshot_ruan-test-2018-05-24_1 SUCCESS 1527254411 06:20:11 1527254411
    Get the Metadata of the Snapshot:

    ```
    $ curl -XGET 'http://elasticsearch:9200/_snapshot/index-backups/mysnapshot_ruan-test-2018-05-24_1?pretty'
    $ curl -XGET 'http://localhost:9200/_snapshot/index-backups/mysnapshot_ruan-test-2018-05-24_1?pretty'
    {
    "snapshots" : [ {
    "snapshot" : "mysnapshot_ruan-test-2018-05-24_1",
    @@ -1376,7 +1376,7 @@ $ aws s3 --profile es ls s3://my-es-snapshot-bucket/ | grep VRTF2942QCeqyEaMxPgb
    Execute the Restore:

    ```
    $ curl -XPOST -H 'Content-Type: application/json' 'http://elasticsearch:9200/_snapshot/index-backups/mysnapshot_ruan-test-2018-05-24_1/_restore -d '
    $ curl -XPOST -H 'Content-Type: application/json' 'http://localhost:9200/_snapshot/index-backups/mysnapshot_ruan-test-2018-05-24_1/_restore -d '
    {
    "indices": "ruan-test-2018-05-24",
    "ignore_unavailable": true,
  2. @ruanbekker ruanbekker revised this gist May 29, 2018. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1273,7 +1273,7 @@ $ curl -XGET 'elasticsearch:9200/_tasks?detailed=true&actions=*/delete/byquery&p

    ## Elasticsearch S3 Snapshot Repo

    Setup the [S3 Snapshot Repository](https://sysadmins.co.za/aws-elasticsearch-register-s3-repository-for-snapshots-using-the-cli/?rbas_source=gist.github.com?rbas_sourcepage=cheachsheet-elasticsearch.md)
    Setup the [S3 Snapshot Repository](https://sysadmins.co.za/aws-elasticsearch-register-s3-repository-for-snapshots-using-the-cli/?rbas_source=gist.github.com?rbas_sourcepage=cheatsheet-elasticsearch.md)

    Viw the Snapshot Repository:

  3. @ruanbekker ruanbekker revised this gist May 29, 2018. 1 changed file with 132 additions and 1 deletion.
    133 changes: 132 additions & 1 deletion cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -19,6 +19,10 @@
    - [Sort](#sort)
    - [Aggregate]()
    - [Delete](#delete)
    - [Snapshots](#snapshots)
    - [Create Snapshot Repository on S3](#elasticsearch-s3-snapshot-repo)
    - [Create a Snapshot](#elasticsearch-snapshots)
    - [Restore from a Snapshot](#elasticsearch-restore)

    # Resources
    - https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
    @@ -1263,4 +1267,131 @@ $ curl -XGET 'elasticsearch:9200/_tasks?detailed=true&actions=*/delete/byquery&p
    }
    }
    }
    ```
    ```

    # Snapshots

    ## Elasticsearch S3 Snapshot Repo

    Setup the [S3 Snapshot Repository](https://sysadmins.co.za/aws-elasticsearch-register-s3-repository-for-snapshots-using-the-cli/?rbas_source=gist.github.com?rbas_sourcepage=cheachsheet-elasticsearch.md)

    Viw the Snapshot Repository:

    ```
    $ curl -XGET 'http://elasticsearch:9200/_snapshot/index-backups?pretty'
    {
    "index-backups" : {
    "type" : "s3",
    "settings" : {
    "bucket" : "my-es-snapshot-bucket",
    "region" : "eu-west-1",
    "role_arn" : "arn:aws:iam::0123456789012:role/elasticsearch-snapshot-role"
    }
    }
    }
    ```

    ## Elasticsearch Snapshots

    Create a Snapshot named `mysnapshot_ruan-test-2018-05-24_1` of the index: `ruan-test-2018-05-24` and return the exit when the snapshot is done:

    ```
    $ curl -XPUT -H 'Content-Type: application/json' \
    'http://elasticsearch:9200/_snapshot/index-backups/mysnapshot_ruan-test-2018-05-24_1?wait_for_completion=true&pretty=true' -d '
    {
    "indices": "ruan-test-2018-05-24",
    "ignore_unavailable": true,
    "include_global_state": false
    }
    '
    {
    "snapshot" : {
    "snapshot" : "mysnapshot_ruan-test-2018-05-24_1",
    "uuid" : "YRTE5922QCeqyEaMxPqb1A",
    "version_id" : 6000199,
    "version" : "6.0.1",
    "indices" : [ "ruan-test-2018-05-24" ],
    "state" : "SUCCESS",
    "start_time" : "2018-05-25T13:20:11.497Z",
    "start_time_in_millis" : 1527254411497,
    "end_time" : "2018-05-25T13:20:11.886Z",
    "end_time_in_millis" : 1527254411886,
    "duration_in_millis" : 389,
    "failures" : [ ],
    "shards" : {
    "total" : 5,
    "failed" : 0,
    "successful" : 5
    }
    }
    }
    ```

    Verify the Snapshot:

    ```
    $ curl -XGET 'http://elasticsearch:9200/_cat/snapshots/index-backups?v&s=id'
    id status start_epoch start_time end_epoch end_time duration indices successful_shards failed_shards total_shards
    mysnapshot_ruan-test-2018-05-24_1 SUCCESS 1527254411 06:20:11 1527254411 06:20:11 389ms 1 5 0 5
    ```

    ## Elasticsearch Restore

    Get the Metadata of the Snapshot:

    ```
    $ curl -XGET 'http://elasticsearch:9200/_snapshot/index-backups/mysnapshot_ruan-test-2018-05-24_1?pretty'
    {
    "snapshots" : [ {
    "snapshot" : "mysnapshot_ruan-test-2018-05-24_1",
    "uuid" : "YRTE5922QCeqyEaMxPqb1A",
    "version_id" : 6000199,
    "version" : "6.0.1",
    "indices" : [ "ruan-test-2018-05-24" ],
    "state" : "SUCCESS",
    "start_time" : "2018-05-25T13:20:11.497Z",
    "start_time_in_millis" : 1527254411497,
    "end_time" : "2018-05-25T13:20:11.886Z",
    "end_time_in_millis" : 1527254411886,
    "duration_in_millis" : 389,
    "failures" : [ ],
    "shards" : {
    "total" : 5,
    "failed" : 0,
    "successful" : 5
    }
    } ]
    }
    ```

    Inspect the Snapshot on S3:

    ```
    $ aws s3 --profile es ls s3://my-es-snapshot-bucket/ | grep VRTF2942QCeqyEaMxPgb1B
    2018-05-25 15:20:12 90 meta-VRTF2942QCeqyEaMxPgb1B.dat
    2018-05-25 15:20:12 258 snap-VRTF2942QCeqyEaMxPgb1B.dat
    ```

    Execute the Restore:

    ```
    $ curl -XPOST -H 'Content-Type: application/json' 'http://elasticsearch:9200/_snapshot/index-backups/mysnapshot_ruan-test-2018-05-24_1/_restore -d '
    {
    "indices": "ruan-test-2018-05-24",
    "ignore_unavailable": true,
    "include_global_state": false,
    "rename_pattern": "index_(.+)",
    "rename_replacement": "restored_index_$1"
    }
    '
    ```

    or leave out the body for normal restore

    ## Elasticsearch Snapshot Resources:

    - https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html
    - https://www.youtube.com/watch?v=Otl-IcmbiDE
    - https://recology.info/2015/02/elasticsearch-backup-restore/
    - https://medium.com/@rcdexta/periodic-snapshotting-of-elasticsearch-indices-f6b6ca221a0c
  4. @ruanbekker ruanbekker revised this gist Mar 13, 2018. 1 changed file with 106 additions and 0 deletions.
    106 changes: 106 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -384,6 +384,112 @@ my2ndindex 0 p STARTED 0 130b 10.0.2.27 bDWFHuw
    my2ndindex 0 r STARTED 0 130b 10.0.2.24 rNDYCtL
    ```

    ## Create a Index:

    Create a Index with Default Settings:

    ```
    $ curl -XPUT -H 'Content-Type: application/json' 'http://127.0.0.1:9200/ruan-test-2018.03.12'
    ```

    View the settings of the created index:

    ```
    $ curl -XGET 'http://127.0.0.1:9200/ruan-test-2018.03.12/_settings?pretty'
    {
    "ruan-test-2018.03.12" : {
    "settings" : {
    "index" : {
    "creation_date" : "1520929659349",
    "number_of_shards" : "5",
    "number_of_replicas" : "1",
    "uuid" : "EwGz6y7XQkK0ZI08u8qdrQ",
    "version" : {
    "created" : "6000199"
    },
    "provided_name" : "ruan-test-2018.03.12"
    }
    }
    }
    }
    ```

    Remember that primary shard number can only be set on index creation. Change the settings of the index, let's update the index to: 2 replica shards, and the total_fields limit to: 2000

    ```
    $ curl -XPUT -H 'Content-Type: application/json' 'http://127.0.0.1:9200/ruan-test-2018.03.12/_settings' -d '{"number_of_replicas": 0, "index.mapping.total_fields.limit": 2000}'
    ```

    View the changes:

    ```
    $ curl -XGET 'http://127.0.0.1:9200/ruan-test-2018.03.12/_settings?pretty'
    {
    "ruan-test-2018.03.12" : {
    "settings" : {
    "index" : {
    "mapping" : {
    "total_fields" : {
    "limit" : "2000"
    }
    },
    "number_of_shards" : "5",
    "provided_name" : "ruan-test-2018.03.12",
    "creation_date" : "1520929659349",
    "number_of_replicas" : "0",
    "uuid" : "EwGz6y7XQkK0ZI08u8qdrQ",
    "version" : {
    "created" : "6000199"
    }
    }
    }
    }
    }
    ```

    Now, to set the settings on Index Creation:


    ```
    $ curl -XPUT -H 'Content-Type: application/json' 'http://127.0.0.1:9200/ruan-test-2018.03.13' -d '{"settings": {"number_of_replicas": 1, "number_of_shards": 2, "index.mapping.total_fields.limit": 2000}}'
    ```

    Verifying our settings:

    ```
    $ curl -XGET 'http://127.0.0.1:9200/ruan-test-2018.03.13/_settings?pretty'
    {
    "ruan-test-2018.03.13" : {
    "settings" : {
    "index" : {
    "mapping" : {
    "total_fields" : {
    "limit" : "2000"
    }
    },
    "number_of_shards" : "2",
    "provided_name" : "ruan-test-2018.03.13",
    "creation_date" : "1520929638792",
    "number_of_replicas" : "1",
    "uuid" : "hEY8HrlRTFuiYLwKVDAraQ",
    "version" : {
    "created" : "6000199"
    }
    }
    }
    }
    }
    ```

    Viewing our indexes:

    ```
    $ curl -XGET 'http://127.0.0.1:9200/_cat/indices/ruan-test-*?v'
    health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
    green open ruan-test-2018.03.12 EwGz6y7XQkK0ZI08u8qdrQ 5 1 2 0 15.7kb 7.8kb
    green open ruan-test-2018.03.13 hEY8HrlRTFuiYLwKVDAraQ 2 1 0 0 932b 466b
    ```

    ## Ingest Document into Elasticsearch:

    Let's ingest one docuemnt into Elasticsearch, and in this case we will specify the document id as `1`
  5. @ruanbekker ruanbekker revised this gist Mar 12, 2018. 1 changed file with 20 additions and 0 deletions.
    20 changes: 20 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -6,6 +6,7 @@
    - [Index Level](#cluster-health-index-level)
    - [Shard Level](#cluster-health-shard-level)
    - [Nodes Overview](#nodes-overview)
    - [Indices Overview](#indices-overview)
    - [Cluster Maintenance](#cluster-maintenance)
    - [Ingest](#ingest-documents-into-elasticsearch)
    - [Mapping](#mapping)
    @@ -189,12 +190,31 @@ mstWlaoyTM69xhSt-_rZAA 10.0.2.18 10.0.2.18 mstWlao

    ## Indices Overview:

    View all your indices in your cluster:

    ```
    $ curl -XGET http://elasticsearch:9200/_cat/indices?v
    health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
    green open ruan-test CrQZB2L4SaaYCkvYPx5vUA 5 1 38 0 131.9kb 78.6kb
    ```

    View one index:

    ```
    $ curl -XGET 'http://127.0.0.1:9200/_cat/indices/index-name-2018.01.01?v'
    health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
    green open index-name-2018.01.01 Nk8SMQvRSIaNm854bc3Zjg 5 1 395552 0 755.6mb 377.8mb
    ```

    View a range of indices:

    ```
    $ curl -XGET 'https://http://127.0.0.1:9200/_cat/indices/index-name-2018.01*?v'
    health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
    green open index-name-2018.01.19 Vp1EBoeMQkS-a_upLzedhQ 5 1 1220 0 2.6mb 1.3mb
    green open index-name-2018.01.17 hSJMzFJIQrePifCfgb1rOA 5 1 2875 0 3.8mb 1.9mb
    ```

    ## How Many Documents in the ES Cluster (Across all Indices):

    ```
  6. @ruanbekker ruanbekker revised this gist Mar 12, 2018. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -13,6 +13,8 @@
    - [Close API](#open--close-api)
    - [Search](#searching)
    - [Query](#query)
    - [Query by Match](#query-by-match)
    - [Query with Bool](#query-with-bool)
    - [Sort](#sort)
    - [Aggregate]()
    - [Delete](#delete)
  7. @ruanbekker ruanbekker revised this gist Mar 12, 2018. 1 changed file with 3 additions and 3 deletions.
    6 changes: 3 additions & 3 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -892,19 +892,19 @@ curl -XGET 'http://elasticsearch:9200/people/_search?q=age:30&pretty'
    }
    ```

    Query by Term and limit results by 2:
    #### Query by Term and limit results by 2:

    ```
    $ curl -XGET http://127.0.0.1:9200/scrape-sysadmins/_search?pretty -d '{"query": {"term": {"title": "traefik"}}, "size": 2}'
    ```

    Query by Match:
    #### Query by Match:

    ```
    $ curl -XGET http://127.0.0.1:9200/scrape-sysadmins/_search?pretty -d '{"query": {"match": {"title": "traefik"}}, "size": 10}'
    ```

    Query with Bool:
    #### Query with Bool:

    - Check if field exists in index:

  8. @ruanbekker ruanbekker revised this gist Mar 12, 2018. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -9,6 +9,7 @@
    - [Cluster Maintenance](#cluster-maintenance)
    - [Ingest](#ingest-documents-into-elasticsearch)
    - [Mapping](#mapping)
    - [Check Fields in Mappings](#check-fields-in-mappings)
    - [Close API](#open--close-api)
    - [Search](#searching)
    - [Query](#query)
  9. @ruanbekker ruanbekker revised this gist Mar 12, 2018. 1 changed file with 70 additions and 1 deletion.
    71 changes: 70 additions & 1 deletion cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -8,7 +8,8 @@
    - [Nodes Overview](#nodes-overview)
    - [Cluster Maintenance](#cluster-maintenance)
    - [Ingest](#ingest-documents-into-elasticsearch)
    - [Open/Close API](#open--close-api)
    - [Mapping](#mapping)
    - [Close API](#open--close-api)
    - [Search](#searching)
    - [Query](#query)
    - [Sort](#sort)
    @@ -621,6 +622,41 @@ Ingest using the Bulk Api:
    curl -XPOST 'http://elasticsearch:9200/info/_bulk?pretty' --data-binary @info.json
    ```

    # Mapping

    ## Create Mapping
    ## View Mappings

    ## Check Fields in Mappings:

    Check if a field exisists in your mapping:

    ```
    $ curl -XGET 'http://127.0.0.1:9200/index-name-2018.03.01/_mapping/docs/field/company?pretty'
    {
    "index-name-2018.03.01" : {
    "mappings" : {
    "docs" : {
    "company" : {
    "full_name" : "company",
    "mapping" : {
    "company" : {
    "type" : "text",
    "fields" : {
    "keyword" : {
    "type" : "keyword",
    "ignore_above" : 256
    }
    }
    }
    }
    }
    }
    }
    }
    }
    ```

    # Open / Close API:
    - https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-open-close.html

    @@ -867,6 +903,39 @@ Query by Match:
    $ curl -XGET http://127.0.0.1:9200/scrape-sysadmins/_search?pretty -d '{"query": {"match": {"title": "traefik"}}, "size": 10}'
    ```

    Query with Bool:

    - Check if field exists in index:

    ```
    $ curl http://127.0.0.1:9200/test4/_search?pretty -d '{"query": {"bool": {"must": [{"exists": {"field": "name"}}]}}}}'
    {
    "took" : 7,
    "timed_out" : false,
    "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
    },
    "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
    {
    "_index" : "test4",
    "_type" : "docs",
    "_id" : "2",
    "_score" : 1.0,
    "_source" : {
    "id" : "2",
    "name" : "ruan"
    }
    }
    ]
    }
    }
    ```

    - https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query.html

    ## Sort
  10. @ruanbekker ruanbekker revised this gist Mar 12, 2018. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -4,6 +4,7 @@

    - [Cluster Health](#cluster-health)
    - [Index Level](#cluster-health-index-level)
    - [Shard Level](#cluster-health-shard-level)
    - [Nodes Overview](#nodes-overview)
    - [Cluster Maintenance](#cluster-maintenance)
    - [Ingest](#ingest-documents-into-elasticsearch)
  11. @ruanbekker ruanbekker revised this gist Mar 12, 2018. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -3,7 +3,7 @@
    # Shortlinks:

    - [Cluster Health](#cluster-health)
    -- [Index Level](#cluster-health-index-level)
    - [Index Level](#cluster-health-index-level)
    - [Nodes Overview](#nodes-overview)
    - [Cluster Maintenance](#cluster-maintenance)
    - [Ingest](#ingest-documents-into-elasticsearch)
  12. @ruanbekker ruanbekker revised this gist Mar 12, 2018. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -3,6 +3,7 @@
    # Shortlinks:

    - [Cluster Health](#cluster-health)
    -- [Index Level](#cluster-health-index-level)
    - [Nodes Overview](#nodes-overview)
    - [Cluster Maintenance](#cluster-maintenance)
    - [Ingest](#ingest-documents-into-elasticsearch)
  13. @ruanbekker ruanbekker revised this gist Mar 12, 2018. 1 changed file with 8 additions and 0 deletions.
    8 changes: 8 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -14,6 +14,9 @@
    - [Delete](#delete)

    # Resources
    - https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
    - https://www.elastic.co/guide/en/elasticsearch/reference/current/docs.html
    - https://www.elastic.co/blog/managing-time-based-indices-efficiently
    - http://joelabrahamsson.com/elasticsearch-101/
    - https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html
    - https://chatbots.network/logstash-exclude-bots-from-result/
    @@ -958,6 +961,11 @@ $ curl -XPOST http://elasticsearch:9200/products/_search?pretty -d '{"query" : {

    # Delete

    References:

    - [Delete API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete.html)
    - [Delete by Query](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docs-delete-by-query.html)

    ## Delete Index:

    ```
  14. @ruanbekker ruanbekker revised this gist Mar 11, 2018. 1 changed file with 10 additions and 1 deletion.
    11 changes: 10 additions & 1 deletion cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -698,7 +698,8 @@ $ curl -XGET 'http://elasticsearch:9200/people/users/_search?q=age:28&explain&pr
    },
    {
    "value" : 1.0,
    "description" : "queryNorm",
    "description" : "
    Norm",
    "details" : [ ]
    }
    ]
    @@ -855,6 +856,14 @@ Query by Term and limit results by 2:
    $ curl -XGET http://127.0.0.1:9200/scrape-sysadmins/_search?pretty -d '{"query": {"term": {"title": "traefik"}}, "size": 2}'
    ```

    Query by Match:

    ```
    $ curl -XGET http://127.0.0.1:9200/scrape-sysadmins/_search?pretty -d '{"query": {"match": {"title": "traefik"}}, "size": 10}'
    ```

    - https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query.html

    ## Sort
    - https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html

  15. @ruanbekker ruanbekker revised this gist Sep 23, 2017. 1 changed file with 6 additions and 0 deletions.
    6 changes: 6 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -849,6 +849,12 @@ curl -XGET 'http://elasticsearch:9200/people/_search?q=age:30&pretty'
    }
    ```

    Query by Term and limit results by 2:

    ```
    $ curl -XGET http://127.0.0.1:9200/scrape-sysadmins/_search?pretty -d '{"query": {"term": {"title": "traefik"}}, "size": 2}'
    ```

    ## Sort
    - https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html

  16. @ruanbekker ruanbekker revised this gist Aug 24, 2017. 1 changed file with 7 additions and 0 deletions.
    7 changes: 7 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -275,6 +275,13 @@ insertOrder timeInQueue priority source
    1737 1.3s URGENT shard-started ([sysadmins-2017.06.02][3], node[WR3y31g1TnuufpNyrJnQtg], [R], v[91], s[INITIALIZING], a[id=JmrtwtYURMyQF6LspeJXLg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2017-08-11T07:50:56.550Z]]), reason [after recovery (replica) from node [{es01}{6ND8sZ_rTqaL42VdlxyW7Q}{10.79.2.193}{10.79.2.193:9300}]]
    ```

    ## Clear Cache:

    ```
    $ curl -XGET http://127.0.0.1:9200/_cache/clear
    {"_shards":{"total":21,"successful":15,"failed":0}}
    ```

    # Index Info (Shards, Replicas, Allocation):

    ## Create Index:
  17. @ruanbekker ruanbekker revised this gist Aug 14, 2017. 1 changed file with 3 additions and 0 deletions.
    3 changes: 3 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -239,6 +239,9 @@ $ curl -XPUT 'localhost:9200/_cluster/settings?pretty' -d'
    '
    ```

    ## Recovery Resources:
    - https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-recovery.html

    ## Recovering from Node Failure:

    At the moment one of the nodes were down, and up again:
  18. @ruanbekker ruanbekker revised this gist Aug 12, 2017. 1 changed file with 33 additions and 0 deletions.
    33 changes: 33 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -239,6 +239,39 @@ $ curl -XPUT 'localhost:9200/_cluster/settings?pretty' -d'
    '
    ```

    ## Recovering from Node Failure:

    At the moment one of the nodes were down, and up again:

    ```
    $ curl -XGET http://127.0.0.1:9200/_cat/allocation?v
    shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
    290 54.1mb 1gb 20mb 1gb 98 10.79.2.193 10.79.2.193 es01
    151 43.5mb 1gb 11.9gb 13gb 8 10.79.3.171 10.79.3.171 es02
    139 UNASSIGNED
    ```

    ## Recovery API:

    ```
    $ curl -XGET http://127.0.0.1:9200/_cat/recovery?v
    index shard time type stage source_host target_host repository snapshot files files_percent bytes bytes_percent total_files total_bytes translog translog_percent total_translog
    sysadmins-2017.06.19 0 1512 replica done 10.79.2.193 10.79.3.171 n/a n/a 31 100.0% 340020 100.0% 31 340020 0 100.0% 0
    sysadmins-2017.06.19 0 7739 store done 10.79.2.193 10.79.2.193 n/a n/a 0 100.0% 0 100.0% 31 340020 0 100.0% 0
    sysadmins-2017.06.19 1 2592 relocation done 10.79.2.193 10.79.3.171 n/a n/a 13 100.0% 246229 100.0% 13 246229 0 100.0% 0
    sysadmins-2017.06.19 1 613 replica done 10.79.3.171 10.79.2.193 n/a n/a 0 0.0% 0 0.0% 0 0 0 100.0% 0
    ```

    ## Pending Tasks:

    ```
    $ curl -XGET http://127.0.0.1:9200/_cat/pending_tasks?v
    insertOrder timeInQueue priority source
    1736 1.8s URGENT shard-started ([sysadmins-2017.06.02][2], node[WR3y31g1TnuufpNyrJnQtg], [R], v[91], s[INITIALIZING], a[id=wVTDn4nFSKKxvi07cU0uCg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2017-08-11T07:50:56.550Z]]), reason [after recovery (replica) from node [{es01}{6ND8sZ_rTqaL42VdlxyW7Q}{10.79.2.193}{10.79.2.193:9300}]]
    1737 1.3s URGENT shard-started ([sysadmins-2017.06.02][3], node[WR3y31g1TnuufpNyrJnQtg], [R], v[91], s[INITIALIZING], a[id=JmrtwtYURMyQF6LspeJXLg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2017-08-11T07:50:56.550Z]]), reason [after recovery (replica) from node [{es01}{6ND8sZ_rTqaL42VdlxyW7Q}{10.79.2.193}{10.79.2.193:9300}]]
    ```

    # Index Info (Shards, Replicas, Allocation):

    ## Create Index:
  19. @ruanbekker ruanbekker revised this gist Aug 12, 2017. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -16,7 +16,7 @@
    # Resources
    - http://joelabrahamsson.com/elasticsearch-101/
    - https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html

    - https://chatbots.network/logstash-exclude-bots-from-result/

    # Overview

  20. @ruanbekker ruanbekker revised this gist Aug 12, 2017. 1 changed file with 6 additions and 5 deletions.
    11 changes: 6 additions & 5 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -6,11 +6,12 @@
    - [Nodes Overview](#nodes-overview)
    - [Cluster Maintenance](#cluster-maintenance)
    - [Ingest](#ingest-documents-into-elasticsearch)
    - [](#open--close-api)
    - [](#searching)
    - [](#query)
    - [](#sort)
    - [](#delete)
    - [Open/Close API](#open--close-api)
    - [Search](#searching)
    - [Query](#query)
    - [Sort](#sort)
    - [Aggregate]()
    - [Delete](#delete)

    # Resources
    - http://joelabrahamsson.com/elasticsearch-101/
  21. @ruanbekker ruanbekker revised this gist Aug 12, 2017. 1 changed file with 5 additions and 1 deletion.
    6 changes: 5 additions & 1 deletion cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -6,7 +6,11 @@
    - [Nodes Overview](#nodes-overview)
    - [Cluster Maintenance](#cluster-maintenance)
    - [Ingest](#ingest-documents-into-elasticsearch)

    - [](#open--close-api)
    - [](#searching)
    - [](#query)
    - [](#sort)
    - [](#delete)

    # Resources
    - http://joelabrahamsson.com/elasticsearch-101/
  22. @ruanbekker ruanbekker revised this gist Aug 12, 2017. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -4,6 +4,8 @@

    - [Cluster Health](#cluster-health)
    - [Nodes Overview](#nodes-overview)
    - [Cluster Maintenance](#cluster-maintenance)
    - [Ingest](#ingest-documents-into-elasticsearch)


    # Resources
  23. @ruanbekker ruanbekker revised this gist Aug 12, 2017. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -3,6 +3,8 @@
    # Shortlinks:

    - [Cluster Health](#cluster-health)
    - [Nodes Overview](#nodes-overview)


    # Resources
    - http://joelabrahamsson.com/elasticsearch-101/
  24. @ruanbekker ruanbekker revised this gist Aug 12, 2017. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -2,7 +2,7 @@

    # Shortlinks:

    - [#]()
    - [Cluster Health](#cluster-health)

    # Resources
    - http://joelabrahamsson.com/elasticsearch-101/
  25. @ruanbekker ruanbekker revised this gist Aug 12, 2017. 1 changed file with 72 additions and 0 deletions.
    72 changes: 72 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,5 +1,9 @@
    (still a work-in-progress)

    # Shortlinks:

    - [#]()

    # Resources
    - http://joelabrahamsson.com/elasticsearch-101/
    - https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html
    @@ -916,4 +920,72 @@ curl -XPOST 'http://elasticsearch:9200/weblogs/_delete_by_query?pretty' -d' {"qu
    "throttled_until_millis" : 0,
    "failures" : [ ]
    }
    ```

    If routing is provided, then the routing is copied to the scroll query, limiting the process to the shards that match that routing value:

    ```
    $ curl -XPOST 'http://elasticsearch:9200/people/_delete_by_query?routing=1
    {
    "query": {
    "range" : {
    "age" : {
    "gte" : 10
    }
    }
    }
    }
    ```

    By default _delete_by_query uses scroll batches of 1000. You can change the batch size with the scroll_size URL parameter:

    ```
    $ curl -XPOST 'http://elasticsearch:9200/weblogs/_delete_by_query?scroll_size=5000
    {
    "query": {
    "term": {
    "category": "docker"
    }
    }
    }
    ```

    ## Delete Stats:

    ```
    $ curl -XGET 'elasticsearch:9200/_tasks?detailed=true&actions=*/delete/byquery&pretty'
    {
    "nodes" : {
    "s5A2CoRWrwKf512z6NEscF" : {
    "name" : "r4A5VoT",
    "transport_address" : "127.0.0.1:9300",
    "host" : "127.0.0.1",
    "ip" : "127.0.0.1:9300",
    "attributes" : {
    "testattr" : "test",
    "portsfile" : "true"
    },
    "tasks" : {
    "s5A2CoRWrwKf512z6NEscF" : {
    "node" : "s5A2CoRWrwKf512z6NEscF",
    "id" : 36619,
    "type" : "transport",
    "action" : "indices:data/write/delete/byquery",
    "status" : {
    "total" : 6154,
    "updated" : 0,
    "created" : 0,
    "deleted" : 3500,
    "batches" : 36,
    "version_conflicts" : 0,
    "noops" : 0,
    "retries": 0,
    "throttled_millis": 0
    },
    "description" : ""
    }
    }
    }
    }
    }
    ```
  26. @ruanbekker ruanbekker revised this gist Aug 11, 2017. 1 changed file with 16 additions and 0 deletions.
    16 changes: 16 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -210,6 +210,22 @@ shards disk.indices disk.used disk.avail disk.total disk.percent host ip
    5 3.1mb 15.5gb 30.1gb 45.7gb 34 10.0.2.22 10.0.2.22 ifgr6ym
    ```

    # Cluster Maintenance:

    ## Decomission Node from Shard Allocation

    This will move shards from the mentioned node

    ```
    $ curl -XPUT 'localhost:9200/_cluster/settings?pretty' -d'
    {
    "transient" : {
    "cluster.routing.allocation.exclude._ip" : "10.0.0.1"
    }
    }
    '
    ```

    # Index Info (Shards, Replicas, Allocation):

    ## Create Index:
  27. @ruanbekker ruanbekker revised this gist Aug 11, 2017. 1 changed file with 115 additions and 0 deletions.
    115 changes: 115 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -9,6 +9,9 @@

    ## Cluster Health:

    Resource:
    - https://www.elastic.co/guide/en/elasticsearch/guide/current/_cluster_health.html

    ```
    $ curl -XGET http://elasticsearch:9200/_cluster/health?pretty
    {
    @@ -30,6 +33,118 @@ $ curl -XGET http://elasticsearch:9200/_cluster/health?pretty
    }
    ```

    ## Cluster Health: Index Level:

    ```
    $ curl -XGET 'http://elasticsearch:9200/_cluster/health?level=indices&pretty'
    {
    "cluster_name" : "swarm-elasticsearch",
    "status" : "red",
    "timed_out" : false,
    "number_of_nodes" : 5,
    "number_of_data_nodes" : 5,
    "active_primary_shards" : 44,
    "active_shards" : 44,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 64,
    "delayed_unassigned_shards" : 0,
    "number_of_pending_tasks" : 0,
    "number_of_in_flight_fetch" : 0,
    "task_max_waiting_in_queue_millis" : 0,
    "active_shards_percent_as_number" : 40.74074074074074,
    "indices" : {
    "test" : {
    "status" : "yellow",
    "number_of_shards" : 5,
    "number_of_replicas" : 1,
    "active_primary_shards" : 5,
    "active_shards" : 5,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 5
    }
    }
    }
    ```

    ## Cluster Health: Shard Level:

    ```
    curl -XGET 'http://elasticsearch:9200/_cluster/health?level=shards&pretty'
    {
    "cluster_name" : "swarm-elasticsearch",
    "status" : "red",
    "timed_out" : false,
    "number_of_nodes" : 5,
    "number_of_data_nodes" : 5,
    "active_primary_shards" : 44,
    "active_shards" : 44,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 64,
    "delayed_unassigned_shards" : 0,
    "number_of_pending_tasks" : 0,
    "number_of_in_flight_fetch" : 0,
    "task_max_waiting_in_queue_millis" : 0,
    "active_shards_percent_as_number" : 40.74074074074074,
    "indices" : {
    "test" : {
    "status" : "yellow",
    "number_of_shards" : 5,
    "number_of_replicas" : 1,
    "active_primary_shards" : 5,
    "active_shards" : 5,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 5,
    "shards" : {
    "0" : {
    "status" : "yellow",
    "primary_active" : true,
    "active_shards" : 1,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 1
    },
    "1" : {
    "status" : "yellow",
    "primary_active" : true,
    "active_shards" : 1,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 1
    },
    "2" : {
    "status" : "yellow",
    "primary_active" : true,
    "active_shards" : 1,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 1
    },
    "3" : {
    "status" : "yellow",
    "primary_active" : true,
    "active_shards" : 1,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 1
    },
    "4" : {
    "status" : "yellow",
    "primary_active" : true,
    "active_shards" : 1,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 1
    }
    }
    }
    }
    }
    ```

    ## Nodes Overview:

    ```
  28. @ruanbekker ruanbekker revised this gist Aug 10, 2017. 1 changed file with 5 additions and 0 deletions.
    5 changes: 5 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,5 +1,10 @@
    (still a work-in-progress)

    # Resources
    - http://joelabrahamsson.com/elasticsearch-101/
    - https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html


    # Overview

    ## Cluster Health:
  29. @ruanbekker ruanbekker revised this gist Aug 10, 2017. 1 changed file with 33 additions and 0 deletions.
    33 changes: 33 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -747,4 +747,37 @@ $ curl -XPOST http://elasticsearch:9200/products/_search?pretty -d '{"query" : {
    ]
    }
    }
    ```

    # Delete

    ## Delete Index:

    ```
    $ curl -XDELETE http://elasticsearch:9200/myindex
    ```

    ## Delete Documents on Query:

    We would like to delete all documents that has `"os_name": "Windows 10"`

    ```
    curl -XPOST 'http://elasticsearch:9200/weblogs/_delete_by_query?pretty' -d' {"query": {"match": {"os_name": "Windows 10"}}}'
    {
    "took" : 1217,
    "timed_out" : false,
    "total" : 48,
    "deleted" : 48,
    "batches" : 1,
    "version_conflicts" : 0,
    "noops" : 0,
    "retries" : {
    "bulk" : 0,
    "search" : 0
    },
    "throttled_millis" : 0,
    "requests_per_second" : -1.0,
    "throttled_until_millis" : 0,
    "failures" : [ ]
    }
    ```
  30. @ruanbekker ruanbekker revised this gist Aug 9, 2017. 1 changed file with 39 additions and 0 deletions.
    39 changes: 39 additions & 0 deletions cheatsheet-elasticsearch.md
    View file
    Original file line number Diff line number Diff line change
    @@ -708,4 +708,43 @@ $ curl -XPOST http://elasticsearch:9200/products/_search?pretty -d '{"query" : {
    ]
    }
    }
    ```

    Running the same, but wanting to see the sum of all the prices:

    ```
    $ curl -XPOST http://elasticsearch:9200/products/_search?pretty -d '{"query" : {"term" : { "product" : "bananas" }}, "sort" : [{"price" : {"order" : "asc", "mode" : "sum"}}]}'
    {
    "took" : 34,
    "timed_out" : false,
    "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
    },
    "hits" : {
    "total" : 1,
    "max_score" : null,
    "hits" : [
    {
    "_index" : "products",
    "_type" : "items",
    "_id" : "3",
    "_score" : null,
    "_source" : {
    "product" : "bananas",
    "price" : [
    28,
    22,
    23,
    20
    ]
    },
    "sort" : [
    93
    ]
    }
    ]
    }
    }
    ```