bidhanahdib · March 16, 2020 17:34 · Jul 13, 2019 · Jul 13, 2019 · Jul 13, 2019 · Jun 20, 2018
diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -1,4 +1,4 @@
-(still a work-in-progress)
+My Elasticsearch cheatsheet with example usage via rest api (still a work-in-progress)
 
 # Shortlinks:
 
@@ -18,6 +18,7 @@
 - [Query](#query)
   - [Query by Match](#query-by-match)
   - [Query with Bool](#query-with-bool)
+  - [Other Examples with Query](#other-examples-of-query)
 - [Sort](#sort)
 - [Aggregate]()
 - [Delete](#delete)

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -1127,6 +1127,117 @@ $ curl http://127.0.0.1:9200/test4/_search?pretty -d '
 }
 ```
 
+#### Other Examples of Query:
+
+Match:
+
+```
+{
+  "query": {
+    "match": {
+      "title": "something"
+    }
+  }
+}
+```
+
+Multi match with boost on title:
+
+```
+# ^ boosts the score 4 times on title
+{
+  "query": {
+    "multi_match": {
+      "query": "something",
+      "fields": ["title^4", "plot"]
+    }
+  }
+}
+```
+
+Match phrase:
+
+```
+{
+  "query": {
+    "match_phrase": {
+      "title": "somethings got to give"
+    }
+  }
+}
+```
+
+Common terms:
+
+```
+{
+  "query": {
+    "common": {
+      "title": {
+        "query": "the something word"
+      }
+    }
+  }
+}
+```
+
+Query string:
+
+```
+{
+  "query": {
+    "query_string": {
+      "query": "the something AND (gives OR gave)"
+    }
+  }
+}
+```
+
+Simple query string:
+
+```
+{
+  "query": {
+    "simple_query_string": {
+      "query": "\"give got to\"~4 | *thing~2",
+      "fields": ["title"]
+    }
+  }
+}
+```
+
+More info on above:
+
+```
++    -> Acts as the AND operator
+|    -> Acts as the OR operator
+*    -> Acts as a wildcard.
+""   -> Wraps several terms into a phrase.
+()   -> Wraps a clause for precedence.
+~n   -> When used after a term (e.g. thign~3), sets fuzziness. When used after a phrase, sets slop. See Options.
+-    -> Negates the term.
+```
+
+Match all:
+
+```
+{
+  "query": {
+    "match_all": {}
+  }
+}
+```
+
+Match none:
+
+```
+{
+  "query": {
+    "match_none": {}
+  }
+}
+```
+
 - https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query.html
 
 ## Sort

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -1055,21 +1055,51 @@ curl -XGET 'http://elasticsearch:9200/people/_search?q=age:30&pretty'
 #### Query by Term and limit results by 2:
 
 ```
-$ curl -XGET http://127.0.0.1:9200/scrape-sysadmins/_search?pretty -d '{"query": {"term": {"title": "traefik"}}, "size": 2}'
+$ curl -XGET http://127.0.0.1:9200/scrape-sysadmins/_search?pretty -d '
+{
+  "query": {
+    "term": {
+      "title": "traefik"
+    }
+  }, 
+  "size": 2
+}
+'
 ```
 
 #### Query by Match:
 
 ```
-$ curl -XGET http://127.0.0.1:9200/scrape-sysadmins/_search?pretty -d '{"query": {"match": {"title": "traefik"}}, "size": 10}'
+$ curl -XGET http://127.0.0.1:9200/scrape-sysadmins/_search?pretty -d '
+{
+  "query": {
+    "match": {
+      "title": "traefik"
+    }
+  }, 
+  "size": 10
+}
+'
 ```
 
 #### Query with Bool:
 
 - Check if field exists in index:
 
 ```
-$ curl http://127.0.0.1:9200/test4/_search?pretty -d '{"query": {"bool": {"must": [{"exists": {"field": "name"}}]}}}}'
+$ curl http://127.0.0.1:9200/test4/_search?pretty -d '
+{
+  "query": {
+    "bool": {
+      "must": [{
+        "exists": {
+	  "field": "name"
+	}
+      }]
+    }
+  }
+}'
+
 {
   "took" : 7,
   "timed_out" : false,
@@ -1116,7 +1146,21 @@ $ curl -XPUT http://elasticsearch:9200/products/items/4 -d '{"product": "chips",
 Run a Sort Query on the term `bananas`, and show the `average` price. We can also use `min, max, avg, sum`:
 
 ```
-$ curl -XPOST http://elasticsearch:9200/products/_search?pretty -d '{"query" : {"term" : { "product" : "bananas" }}, "sort" : [{"price" : {"order" : "asc", "mode" : "avg"}}]}'
+$ curl -XPOST http://elasticsearch:9200/products/_search?pretty -d '
+{
+  "query" : {
+    "term" : {
+      "product" : "bananas"
+    }
+  }, 
+  "sort" : [{
+    "price" : {
+      "order" : "asc", 
+      "mode" : "avg"
+    }
+  }]
+}'
+
 {
   "took" : 9,
   "timed_out" : false,
@@ -1155,7 +1199,21 @@ $ curl -XPOST http://elasticsearch:9200/products/_search?pretty -d '{"query" : {
 Running the same, but wanting to see the sum of all the prices:
 
 ```
-$ curl -XPOST http://elasticsearch:9200/products/_search?pretty -d '{"query" : {"term" : { "product" : "bananas" }}, "sort" : [{"price" : {"order" : "asc", "mode" : "sum"}}]}'
+$ curl -XPOST http://elasticsearch:9200/products/_search?pretty -d '
+{
+  "query" : {
+    "term" : {
+      "product" : "bananas"
+    }
+  }, 
+  "sort" : [{
+    "price" : {
+      "order" : "asc", 
+      "mode" : "sum"
+    }
+  }]
+}'
+
 {
   "took" : 34,
   "timed_out" : false,
@@ -1209,7 +1267,15 @@ $ curl -XDELETE http://elasticsearch:9200/myindex
 We would like to delete all documents that has `"os_name": "Windows 10"`
 
 ```
-curl -XPOST 'http://elasticsearch:9200/weblogs/_delete_by_query?pretty' -d' {"query": {"match": {"os_name": "Windows 10"}}}'
+curl -XPOST 'http://elasticsearch:9200/weblogs/_delete_by_query?pretty' -d '
+{
+  "query": {
+    "match": {
+      "os_name": "Windows 10"
+    }
+  }
+}'
+
 {
   "took" : 1217,
   "timed_out" : false,

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -221,6 +221,15 @@ green  open   index-name-2018.01.19 Vp1EBoeMQkS-a_upLzedhQ   5   1       1220
 green  open   index-name-2018.01.17 hSJMzFJIQrePifCfgb1rOA   5   1       2875            0      3.8mb          1.9mb
 ```
 
+View only the index name header:
+
+```
+$ curl -XGET 'http://127.0.0.1:9200/_cat/indices/*2018.03.*?v&h=index'
+index
+index-name-2018.03.01
+index-name-2018.03.02
+```
+
 ## How Many Documents in the ES Cluster (Across all Indices):
 
 ```

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -1294,12 +1294,21 @@ $ curl -XGET 'elasticsearch:9200/_tasks?detailed=true&actions=*/delete/byquery&p
 
 Setup the [S3 Snapshot Repository](https://sysadmins.co.za/aws-elasticsearch-register-s3-repository-for-snapshots-using-the-cli/?rbas_source=gist.github.com?rbas_sourcepage=cheatsheet-elasticsearch.md)
 
-Viw the Snapshot Repository:
+List the Snapshot Repositories:
 
 ```
-$ curl -XGET 'http://elasticsearch:9200/_snapshot/index-backups?pretty'
+$ curl -XGET 'http://127.0.0.1:9200/_cat/repositories?v'
+id            type
+foo-bacups    s3
+bar-backups   s3
+```
+
+View the Snapshot Repository:
+
+```
+$ curl -XGET 'http://elasticsearch:9200/_snapshot/bar-backups?pretty'
 {
-  "index-backups" : {
+  "bar-backups" : {
     "type" : "s3",
     "settings" : {
       "bucket" : "my-es-snapshot-bucket",
@@ -1316,7 +1325,7 @@ Create a Snapshot named `mysnapshot_ruan-test-2018-05-24_1` of the index: `ruan-
 
 ```
 $ curl -XPUT -H 'Content-Type: application/json' \
-  'http://elasticsearch:9200/_snapshot/index-backups/mysnapshot_ruan-test-2018-05-24_1?wait_for_completion=true&pretty=true' -d '
+  'http://elasticsearch:9200/_snapshot/bar-backups/mysnapshot_ruan-test-2018-05-24_1?wait_for_completion=true&pretty=true' -d '
 {
 	"indices": "ruan-test-2018-05-24", 
 	"ignore_unavailable": true, 
@@ -1350,7 +1359,7 @@ $ curl -XPUT -H 'Content-Type: application/json' \
 Verify the Snapshot:
 
 ```
-$ curl -XGET 'http://elasticsearch:9200/_cat/snapshots/index-backups?v&s=id'
+$ curl -XGET 'http://elasticsearch:9200/_cat/snapshots/bar-backups?v&s=id'
 id                                      status start_epoch start_time end_epoch  end_time duration indices successful_shards failed_shards total_shards
 mysnapshot_ruan-test-2018-05-24_1      SUCCESS 1527254411  06:20:11   1527254411 06:20:11    389ms       1                 5             0            5
 ```
@@ -1360,7 +1369,7 @@ mysnapshot_ruan-test-2018-05-24_1      SUCCESS 1527254411  06:20:11   1527254411
 Get the Metadata of the Snapshot:
 
 ```
-$ curl -XGET 'http://elasticsearch:9200/_snapshot/index-backups/mysnapshot_ruan-test-2018-05-24_1?pretty'
+$ curl -XGET 'http://elasticsearch:9200/_snapshot/bar-backups/mysnapshot_ruan-test-2018-05-24_1?pretty'
 {
   "snapshots" : [ {
     "snapshot" : "mysnapshot_ruan-test-2018-05-24_1",
@@ -1395,7 +1404,7 @@ $ aws s3 --profile es ls s3://my-es-snapshot-bucket/ | grep VRTF2942QCeqyEaMxPgb
 Execute the Restore:
 
 ```
-$ curl -XPOST -H 'Content-Type: application/json' 'http://elasticsearch:9200/_snapshot/index-backups/mysnapshot_ruan-test-2018-05-24_1/_restore -d '
+$ curl -XPOST -H 'Content-Type: application/json' 'http://elasticsearch:9200/_snapshot/bar-backups/mysnapshot_ruan-test-2018-05-24_1/_restore -d '
 {
   "indices": "ruan-test-2018-05-24",
   "ignore_unavailable": true,

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -8,6 +8,8 @@
 - [Nodes Overview](#nodes-overview)
 - [Indices Overview](#indices-overview)
 - [Cluster Maintenance](#cluster-maintenance)
+- [Settings]()
+  - [Cluster Settings](#cluster-settings)
 - [Ingest](#ingest-documents-into-elasticsearch)
 - [Mapping](#mapping)
   - [Check Fields in Mappings](#check-fields-in-mappings)
@@ -315,6 +317,23 @@ $  curl -XGET http://127.0.0.1:9200/_cache/clear
 {"_shards":{"total":21,"successful":15,"failed":0}}
 ```
 
+# Settings
+
+## Cluster Settings
+
+Search Timeout:
+
+Global Search Timeout, that applies to all search queries across the entire cluster -> search.default_search_timeout:
+
+```
+PUT /_cluster/settings
+{
+    "persistent" : {
+        "search.default_search_timeout" : "50"
+    }
+}
+```
+
 # Index Info (Shards, Replicas, Allocation):
 
 ## Create Index:

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -1273,7 +1273,7 @@ $ curl -XGET 'elasticsearch:9200/_tasks?detailed=true&actions=*/delete/byquery&p
 
 ## Elasticsearch S3 Snapshot Repo
 
-Setup the [S3 Snapshot Repository](https://sysadmins.co.za/aws-elasticsearch-register-s3-repository-for-snapshots-using-the-cli/?rbas_source=gist.github.com?rbas_sourcepage=cheachsheet-elasticsearch.md)
+Setup the [S3 Snapshot Repository](https://sysadmins.co.za/aws-elasticsearch-register-s3-repository-for-snapshots-using-the-cli/?rbas_source=gist.github.com?rbas_sourcepage=cheatsheet-elasticsearch.md)
 
 Viw the Snapshot Repository:
 

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -19,6 +19,10 @@
 - [Sort](#sort)
 - [Aggregate]()
 - [Delete](#delete)
+- [Snapshots](#snapshots)
+  - [Create Snapshot Repository on S3](#elasticsearch-s3-snapshot-repo)
+  - [Create a Snapshot](#elasticsearch-snapshots)
+  - [Restore from a Snapshot](#elasticsearch-restore)
 
 # Resources 
 - https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
@@ -1263,4 +1267,131 @@ $ curl -XGET 'elasticsearch:9200/_tasks?detailed=true&actions=*/delete/byquery&p
     }
   }
 }
-```
+```
+
+# Snapshots
+
+## Elasticsearch S3 Snapshot Repo
+
+Setup the [S3 Snapshot Repository](https://sysadmins.co.za/aws-elasticsearch-register-s3-repository-for-snapshots-using-the-cli/?rbas_source=gist.github.com?rbas_sourcepage=cheachsheet-elasticsearch.md)
+
+Viw the Snapshot Repository:
+
+```
+$ curl -XGET 'http://elasticsearch:9200/_snapshot/index-backups?pretty'
+{
+  "index-backups" : {
+    "type" : "s3",
+    "settings" : {
+      "bucket" : "my-es-snapshot-bucket",
+      "region" : "eu-west-1",
+      "role_arn" : "arn:aws:iam::0123456789012:role/elasticsearch-snapshot-role"
+    }
+  }
+}
+```
+
+## Elasticsearch Snapshots
+
+Create a Snapshot named `mysnapshot_ruan-test-2018-05-24_1` of the index: `ruan-test-2018-05-24` and return the exit when the snapshot is done:
+
+```
+$ curl -XPUT -H 'Content-Type: application/json' \
+  'http://elasticsearch:9200/_snapshot/index-backups/mysnapshot_ruan-test-2018-05-24_1?wait_for_completion=true&pretty=true' -d '
+{
+	"indices": "ruan-test-2018-05-24", 
+	"ignore_unavailable": true, 
+	"include_global_state": false
+}
+'
+
+{
+  "snapshot" : {
+    "snapshot" : "mysnapshot_ruan-test-2018-05-24_1",
+    "uuid" : "YRTE5922QCeqyEaMxPqb1A",
+    "version_id" : 6000199,
+    "version" : "6.0.1",
+    "indices" : [ "ruan-test-2018-05-24" ],
+    "state" : "SUCCESS",
+    "start_time" : "2018-05-25T13:20:11.497Z",
+    "start_time_in_millis" : 1527254411497,
+    "end_time" : "2018-05-25T13:20:11.886Z",
+    "end_time_in_millis" : 1527254411886,
+    "duration_in_millis" : 389,
+    "failures" : [ ],
+    "shards" : {
+      "total" : 5,
+      "failed" : 0,
+      "successful" : 5
+    }
+  }
+}
+```
+
+Verify the Snapshot:
+
+```
+$ curl -XGET 'http://elasticsearch:9200/_cat/snapshots/index-backups?v&s=id'
+id                                      status start_epoch start_time end_epoch  end_time duration indices successful_shards failed_shards total_shards
+mysnapshot_ruan-test-2018-05-24_1      SUCCESS 1527254411  06:20:11   1527254411 06:20:11    389ms       1                 5             0            5
+```
+
+## Elasticsearch Restore
+
+Get the Metadata of the Snapshot:
+
+```
+$ curl -XGET 'http://elasticsearch:9200/_snapshot/index-backups/mysnapshot_ruan-test-2018-05-24_1?pretty'
+{
+  "snapshots" : [ {
+    "snapshot" : "mysnapshot_ruan-test-2018-05-24_1",
+    "uuid" : "YRTE5922QCeqyEaMxPqb1A",
+    "version_id" : 6000199,
+    "version" : "6.0.1",
+    "indices" : [ "ruan-test-2018-05-24" ],
+    "state" : "SUCCESS",
+    "start_time" : "2018-05-25T13:20:11.497Z",
+    "start_time_in_millis" : 1527254411497,
+    "end_time" : "2018-05-25T13:20:11.886Z",
+    "end_time_in_millis" : 1527254411886,
+    "duration_in_millis" : 389,
+    "failures" : [ ],
+    "shards" : {
+      "total" : 5,
+      "failed" : 0,
+      "successful" : 5
+    }
+  } ]
+}
+```
+
+Inspect the Snapshot on S3:
+
+```
+$ aws s3 --profile es ls s3://my-es-snapshot-bucket/ | grep VRTF2942QCeqyEaMxPgb1B
+2018-05-25 15:20:12         90 meta-VRTF2942QCeqyEaMxPgb1B.dat
+2018-05-25 15:20:12        258 snap-VRTF2942QCeqyEaMxPgb1B.dat
+```
+
+Execute the Restore:
+
+```
+$ curl -XPOST -H 'Content-Type: application/json' 'http://elasticsearch:9200/_snapshot/index-backups/mysnapshot_ruan-test-2018-05-24_1/_restore -d '
+{
+  "indices": "ruan-test-2018-05-24",
+  "ignore_unavailable": true,
+  "include_global_state": false,
+  "rename_pattern": "index_(.+)",
+  "rename_replacement": "restored_index_$1"
+}
+'
+```
+
+or leave out the body for normal restore
+
+## Elasticsearch Snapshot Resources:
+
+- https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html
+- https://www.youtube.com/watch?v=Otl-IcmbiDE
+- https://recology.info/2015/02/elasticsearch-backup-restore/
+- https://medium.com/@rcdexta/periodic-snapshotting-of-elasticsearch-indices-f6b6ca221a0c
diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -384,6 +384,112 @@ my2ndindex 0     p      STARTED    0  130b 10.0.2.27 bDWFHuw
 my2ndindex 0     r      STARTED    0  130b 10.0.2.24 rNDYCtL
 ```
 
+## Create a Index:
+
+Create a Index with Default Settings:
+
+```
+$ curl -XPUT -H 'Content-Type: application/json' 'http://127.0.0.1:9200/ruan-test-2018.03.12'
+```
+
+View the settings of the created index:
+
+```
+$ curl -XGET 'http://127.0.0.1:9200/ruan-test-2018.03.12/_settings?pretty'
+{
+  "ruan-test-2018.03.12" : {
+    "settings" : {
+      "index" : {
+        "creation_date" : "1520929659349",
+        "number_of_shards" : "5",
+        "number_of_replicas" : "1",
+        "uuid" : "EwGz6y7XQkK0ZI08u8qdrQ",
+        "version" : {
+          "created" : "6000199"
+        },
+        "provided_name" : "ruan-test-2018.03.12"
+      }
+    }
+  }
+}
+```
+
+Remember that primary shard number can only be set on index creation. Change the settings of the index, let's update the index to: 2 replica shards, and the total_fields limit to: 2000
+
+```
+$ curl -XPUT -H 'Content-Type: application/json' 'http://127.0.0.1:9200/ruan-test-2018.03.12/_settings' -d '{"number_of_replicas": 0, "index.mapping.total_fields.limit": 2000}'
+```
+
+View the changes:
+
+```
+$ curl -XGET 'http://127.0.0.1:9200/ruan-test-2018.03.12/_settings?pretty'
+{
+  "ruan-test-2018.03.12" : {
+    "settings" : {
+      "index" : {
+        "mapping" : {
+          "total_fields" : {
+            "limit" : "2000"
+          }
+        },
+        "number_of_shards" : "5",
+        "provided_name" : "ruan-test-2018.03.12",
+        "creation_date" : "1520929659349",
+        "number_of_replicas" : "0",
+        "uuid" : "EwGz6y7XQkK0ZI08u8qdrQ",
+        "version" : {
+          "created" : "6000199"
+        }
+      }
+    }
+  }
+}
+```
+
+Now, to set the settings on Index Creation:
+
+
+```
+$ curl -XPUT -H 'Content-Type: application/json' 'http://127.0.0.1:9200/ruan-test-2018.03.13' -d '{"settings": {"number_of_replicas": 1, "number_of_shards": 2, "index.mapping.total_fields.limit": 2000}}'
+```
+
+Verifying our settings:
+
+```
+$ curl -XGET 'http://127.0.0.1:9200/ruan-test-2018.03.13/_settings?pretty'
+{
+  "ruan-test-2018.03.13" : {
+    "settings" : {
+      "index" : {
+        "mapping" : {
+          "total_fields" : {
+            "limit" : "2000"
+          }
+        },
+        "number_of_shards" : "2",
+        "provided_name" : "ruan-test-2018.03.13",
+        "creation_date" : "1520929638792",
+        "number_of_replicas" : "1",
+        "uuid" : "hEY8HrlRTFuiYLwKVDAraQ",
+        "version" : {
+          "created" : "6000199"
+        }
+      }
+    }
+  }
+}
+```
+
+Viewing our indexes:
+
+```
+$ curl -XGET 'http://127.0.0.1:9200/_cat/indices/ruan-test-*?v'
+health status index                uuid                   pri rep docs.count docs.deleted store.size pri.store.size
+green  open   ruan-test-2018.03.12 EwGz6y7XQkK0ZI08u8qdrQ   5   1          2            0     15.7kb          7.8kb
+green  open   ruan-test-2018.03.13 hEY8HrlRTFuiYLwKVDAraQ   2   1          0            0       932b           466b
+```
+
 ## Ingest Document into Elasticsearch:
 
 Let's ingest one docuemnt into Elasticsearch, and in this case we will specify the document id as `1`

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -6,6 +6,7 @@
   - [Index Level](#cluster-health-index-level)
   - [Shard Level](#cluster-health-shard-level)
 - [Nodes Overview](#nodes-overview)
+- [Indices Overview](#indices-overview)
 - [Cluster Maintenance](#cluster-maintenance)
 - [Ingest](#ingest-documents-into-elasticsearch)
 - [Mapping](#mapping)
@@ -189,12 +190,31 @@ mstWlaoyTM69xhSt-_rZAA 10.0.2.18 10.0.2.18 mstWlao
 
 ## Indices Overview:
 
+View all your indices in your cluster:
+
 ```
 $ curl -XGET http://elasticsearch:9200/_cat/indices?v
 health status index                         uuid                   pri rep docs.count docs.deleted store.size pri.store.size
 green  open   ruan-test                     CrQZB2L4SaaYCkvYPx5vUA   5   1         38            0    131.9kb         78.6kb
 ```
 
+View one index:
+
+```
+$ curl -XGET 'http://127.0.0.1:9200/_cat/indices/index-name-2018.01.01?v'
+health status index                   uuid                   pri rep docs.count docs.deleted store.size pri.store.size
+green  open   index-name-2018.01.01 Nk8SMQvRSIaNm854bc3Zjg   5   1     395552            0    755.6mb        377.8mb
+```
+
+View a range of indices:
+
+```
+$ curl -XGET 'https://http://127.0.0.1:9200/_cat/indices/index-name-2018.01*?v'
+health status index                   uuid                   pri rep docs.count docs.deleted store.size pri.store.size
+green  open   index-name-2018.01.19 Vp1EBoeMQkS-a_upLzedhQ   5   1       1220            0      2.6mb          1.3mb
+green  open   index-name-2018.01.17 hSJMzFJIQrePifCfgb1rOA   5   1       2875            0      3.8mb          1.9mb
+```
+
 ## How Many Documents in the ES Cluster (Across all Indices):
 
 ```

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -13,6 +13,8 @@
 - [Close API](#open--close-api)
 - [Search](#searching)
 - [Query](#query)
+  - [Query by Match](#query-by-match)
+  - [Query with Bool](#query-with-bool)
 - [Sort](#sort)
 - [Aggregate]()
 - [Delete](#delete)

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -892,19 +892,19 @@ curl -XGET 'http://elasticsearch:9200/people/_search?q=age:30&pretty'
 }
 ```
 
-Query by Term and limit results by 2:
+#### Query by Term and limit results by 2:
 
 ```
 $ curl -XGET http://127.0.0.1:9200/scrape-sysadmins/_search?pretty -d '{"query": {"term": {"title": "traefik"}}, "size": 2}'
 ```
 
-Query by Match:
+#### Query by Match:
 
 ```
 $ curl -XGET http://127.0.0.1:9200/scrape-sysadmins/_search?pretty -d '{"query": {"match": {"title": "traefik"}}, "size": 10}'
 ```
 
-Query with Bool:
+#### Query with Bool:
 
 - Check if field exists in index:
 

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -9,6 +9,7 @@
 - [Cluster Maintenance](#cluster-maintenance)
 - [Ingest](#ingest-documents-into-elasticsearch)
 - [Mapping](#mapping)
+  - [Check Fields in Mappings](#check-fields-in-mappings)
 - [Close API](#open--close-api)
 - [Search](#searching)
 - [Query](#query)

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -8,7 +8,8 @@
 - [Nodes Overview](#nodes-overview)
 - [Cluster Maintenance](#cluster-maintenance)
 - [Ingest](#ingest-documents-into-elasticsearch)
-- [Open/Close API](#open--close-api)
+- [Mapping](#mapping)
+- [Close API](#open--close-api)
 - [Search](#searching)
 - [Query](#query)
 - [Sort](#sort)
@@ -621,6 +622,41 @@ Ingest using the Bulk Api:
 curl -XPOST 'http://elasticsearch:9200/info/_bulk?pretty' --data-binary @info.json
 ```
 
+# Mapping
+
+## Create Mapping
+## View Mappings
+
+## Check Fields in Mappings:
+
+Check if a field exisists in your mapping:
+
+```
+$ curl -XGET 'http://127.0.0.1:9200/index-name-2018.03.01/_mapping/docs/field/company?pretty'
+{
+  "index-name-2018.03.01" : {
+    "mappings" : {
+      "docs" : {
+        "company" : {
+          "full_name" : "company",
+          "mapping" : {
+            "company" : {
+              "type" : "text",
+              "fields" : {
+                "keyword" : {
+                  "type" : "keyword",
+                  "ignore_above" : 256
+                }
+              }
+            }
+          }
+        }
+      }
+    }
+  }
+}
+```
+
 # Open / Close API:
 - https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-open-close.html
 
@@ -867,6 +903,39 @@ Query by Match:
 $ curl -XGET http://127.0.0.1:9200/scrape-sysadmins/_search?pretty -d '{"query": {"match": {"title": "traefik"}}, "size": 10}'
 ```
 
+Query with Bool:
+
+- Check if field exists in index:
+
+```
+$ curl http://127.0.0.1:9200/test4/_search?pretty -d '{"query": {"bool": {"must": [{"exists": {"field": "name"}}]}}}}'
+{
+  "took" : 7,
+  "timed_out" : false,
+  "_shards" : {
+    "total" : 5,
+    "successful" : 5,
+    "failed" : 0
+  },
+  "hits" : {
+    "total" : 1,
+    "max_score" : 1.0,
+    "hits" : [
+      {
+        "_index" : "test4",
+        "_type" : "docs",
+        "_id" : "2",
+        "_score" : 1.0,
+        "_source" : {
+          "id" : "2",
+          "name" : "ruan"
+        }
+      }
+    ]
+  }
+}
+```
+
 - https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query.html
 
 ## Sort

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -4,6 +4,7 @@
 
 - [Cluster Health](#cluster-health)
   - [Index Level](#cluster-health-index-level)
+  - [Shard Level](#cluster-health-shard-level)
 - [Nodes Overview](#nodes-overview)
 - [Cluster Maintenance](#cluster-maintenance)
 - [Ingest](#ingest-documents-into-elasticsearch)

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -3,7 +3,7 @@
 # Shortlinks:
 
 - [Cluster Health](#cluster-health)
--- [Index Level](#cluster-health-index-level)
+  - [Index Level](#cluster-health-index-level)
 - [Nodes Overview](#nodes-overview)
 - [Cluster Maintenance](#cluster-maintenance)
 - [Ingest](#ingest-documents-into-elasticsearch)

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -3,6 +3,7 @@
 # Shortlinks:
 
 - [Cluster Health](#cluster-health)
+-- [Index Level](#cluster-health-index-level)
 - [Nodes Overview](#nodes-overview)
 - [Cluster Maintenance](#cluster-maintenance)
 - [Ingest](#ingest-documents-into-elasticsearch)

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -14,6 +14,9 @@
 - [Delete](#delete)
 
 # Resources 
+- https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
+- https://www.elastic.co/guide/en/elasticsearch/reference/current/docs.html
+- https://www.elastic.co/blog/managing-time-based-indices-efficiently
 - http://joelabrahamsson.com/elasticsearch-101/
 - https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html
 - https://chatbots.network/logstash-exclude-bots-from-result/
@@ -958,6 +961,11 @@ $ curl -XPOST http://elasticsearch:9200/products/_search?pretty -d '{"query" : {
 
 # Delete
 
+References:
+
+- [Delete API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete.html)
+- [Delete by Query](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docs-delete-by-query.html)
+
 ## Delete Index:
 
 ```

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -698,7 +698,8 @@ $ curl -XGET 'http://elasticsearch:9200/people/users/_search?q=age:28&explain&pr
             },
             {
               "value" : 1.0,
-              "description" : "queryNorm",
+              "description" : "
+              Norm",
               "details" : [ ]
             }
           ]
@@ -855,6 +856,14 @@ Query by Term and limit results by 2:
 $ curl -XGET http://127.0.0.1:9200/scrape-sysadmins/_search?pretty -d '{"query": {"term": {"title": "traefik"}}, "size": 2}'
 ```
 
+Query by Match:
+
+```
+$ curl -XGET http://127.0.0.1:9200/scrape-sysadmins/_search?pretty -d '{"query": {"match": {"title": "traefik"}}, "size": 10}'
+```
+
+- https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query.html
+
 ## Sort
 - https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html
 

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -849,6 +849,12 @@ curl -XGET 'http://elasticsearch:9200/people/_search?q=age:30&pretty'
 }
 ```
 
+Query by Term and limit results by 2:
+
+```
+$ curl -XGET http://127.0.0.1:9200/scrape-sysadmins/_search?pretty -d '{"query": {"term": {"title": "traefik"}}, "size": 2}'
+```
+
 ## Sort
 - https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html
 

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -275,6 +275,13 @@ insertOrder timeInQueue priority source
        1737        1.3s URGENT   shard-started ([sysadmins-2017.06.02][3], node[WR3y31g1TnuufpNyrJnQtg], [R], v[91], s[INITIALIZING], a[id=JmrtwtYURMyQF6LspeJXLg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2017-08-11T07:50:56.550Z]]), reason [after recovery (replica) from node [{es01}{6ND8sZ_rTqaL42VdlxyW7Q}{10.79.2.193}{10.79.2.193:9300}]]
 ```
 
+## Clear Cache:
+
+```
+$  curl -XGET http://127.0.0.1:9200/_cache/clear
+{"_shards":{"total":21,"successful":15,"failed":0}}
+```
+
 # Index Info (Shards, Replicas, Allocation):
 
 ## Create Index:

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -239,6 +239,9 @@ $ curl -XPUT 'localhost:9200/_cluster/settings?pretty' -d'
 '
 ```
 
+## Recovery Resources:
+- https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-recovery.html
+
 ## Recovering from Node Failure:
 
 At the moment one of the nodes were down, and up again:

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -239,6 +239,39 @@ $ curl -XPUT 'localhost:9200/_cluster/settings?pretty' -d'
 '
 ```
 
+## Recovering from Node Failure:
+
+At the moment one of the nodes were down, and up again:
+
+```
+$ curl -XGET http://127.0.0.1:9200/_cat/allocation?v
+shards disk.indices disk.used disk.avail disk.total disk.percent host        ip          node
+   290       54.1mb       1gb       20mb        1gb           98 10.79.2.193 10.79.2.193 es01
+   151       43.5mb       1gb     11.9gb       13gb            8 10.79.3.171 10.79.3.171 es02
+   139                                                                                   UNASSIGNED
+```
+
+## Recovery API:
+
+```
+$ curl -XGET http://127.0.0.1:9200/_cat/recovery?v
+index                     shard time   type       stage source_host target_host repository snapshot files files_percent bytes  bytes_percent total_files total_bytes translog translog_percent total_translog
+sysadmins-2017.06.19      0     1512   replica    done  10.79.2.193 10.79.3.171 n/a        n/a      31    100.0%        340020 100.0%        31          340020      0        100.0%           0
+sysadmins-2017.06.19      0     7739   store      done  10.79.2.193 10.79.2.193 n/a        n/a      0     100.0%        0      100.0%        31          340020      0        100.0%           0
+sysadmins-2017.06.19      1     2592   relocation done  10.79.2.193 10.79.3.171 n/a        n/a      13    100.0%        246229 100.0%        13          246229      0        100.0%           0
+sysadmins-2017.06.19      1     613    replica    done  10.79.3.171 10.79.2.193 n/a        n/a      0     0.0%          0      0.0%          0           0           0        100.0%           0
+
+```
+
+## Pending Tasks:
+
+```
+$ curl -XGET http://127.0.0.1:9200/_cat/pending_tasks?v
+insertOrder timeInQueue priority source
+       1736        1.8s URGENT   shard-started ([sysadmins-2017.06.02][2], node[WR3y31g1TnuufpNyrJnQtg], [R], v[91], s[INITIALIZING], a[id=wVTDn4nFSKKxvi07cU0uCg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2017-08-11T07:50:56.550Z]]), reason [after recovery (replica) from node [{es01}{6ND8sZ_rTqaL42VdlxyW7Q}{10.79.2.193}{10.79.2.193:9300}]]
+       1737        1.3s URGENT   shard-started ([sysadmins-2017.06.02][3], node[WR3y31g1TnuufpNyrJnQtg], [R], v[91], s[INITIALIZING], a[id=JmrtwtYURMyQF6LspeJXLg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2017-08-11T07:50:56.550Z]]), reason [after recovery (replica) from node [{es01}{6ND8sZ_rTqaL42VdlxyW7Q}{10.79.2.193}{10.79.2.193:9300}]]
+```
+
 # Index Info (Shards, Replicas, Allocation):
 
 ## Create Index:

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -16,7 +16,7 @@
 # Resources 
 - http://joelabrahamsson.com/elasticsearch-101/
 - https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html
-
+- https://chatbots.network/logstash-exclude-bots-from-result/
 
 # Overview
 

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -6,11 +6,12 @@
 - [Nodes Overview](#nodes-overview)
 - [Cluster Maintenance](#cluster-maintenance)
 - [Ingest](#ingest-documents-into-elasticsearch)
-- [](#open--close-api)
-- [](#searching)
-- [](#query)
-- [](#sort)
-- [](#delete)
+- [Open/Close API](#open--close-api)
+- [Search](#searching)
+- [Query](#query)
+- [Sort](#sort)
+- [Aggregate]()
+- [Delete](#delete)
 
 # Resources 
 - http://joelabrahamsson.com/elasticsearch-101/

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -6,7 +6,11 @@
 - [Nodes Overview](#nodes-overview)
 - [Cluster Maintenance](#cluster-maintenance)
 - [Ingest](#ingest-documents-into-elasticsearch)
-
+- [](#open--close-api)
+- [](#searching)
+- [](#query)
+- [](#sort)
+- [](#delete)
 
 # Resources 
 - http://joelabrahamsson.com/elasticsearch-101/

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -4,6 +4,8 @@
 
 - [Cluster Health](#cluster-health)
 - [Nodes Overview](#nodes-overview)
+- [Cluster Maintenance](#cluster-maintenance)
+- [Ingest](#ingest-documents-into-elasticsearch)
 
 
 # Resources 

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -3,6 +3,8 @@
 # Shortlinks:
 
 - [Cluster Health](#cluster-health)
+- [Nodes Overview](#nodes-overview)
+
 
 # Resources 
 - http://joelabrahamsson.com/elasticsearch-101/

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -2,7 +2,7 @@
 
 # Shortlinks:
 
-- [#]()
+- [Cluster Health](#cluster-health)
 
 # Resources 
 - http://joelabrahamsson.com/elasticsearch-101/

diff --git a/cheatsheet-elasticsearch.md b/cheatsheet-elasticsearch.md
@@ -1,5 +1,9 @@
 (still a work-in-progress)
 
+# Shortlinks:
+
+- [#]()
+
 # Resources 
 - http://joelabrahamsson.com/elasticsearch-101/
 - https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html
@@ -916,4 +920,72 @@ curl -XPOST 'http://elasticsearch:9200/weblogs/_delete_by_query?pretty' -d' {"qu
   "throttled_until_millis" : 0,
   "failures" : [ ]
 }
+```
+
+If routing is provided, then the routing is copied to the scroll query, limiting the process to the shards that match that routing value:
+
+```
+$ curl -XPOST 'http://elasticsearch:9200/people/_delete_by_query?routing=1
+{
+  "query": {
+    "range" : {
+        "age" : {
+           "gte" : 10
+        }
+    }
+  }
+}
+```
+
+By default _delete_by_query uses scroll batches of 1000. You can change the batch size with the scroll_size URL parameter:
+
+```
+$ curl -XPOST 'http://elasticsearch:9200/weblogs/_delete_by_query?scroll_size=5000
+{
+  "query": {
+    "term": {
+      "category": "docker"
+    }
+  }
+}
+```
+
+## Delete Stats:
+
+```
+$ curl -XGET 'elasticsearch:9200/_tasks?detailed=true&actions=*/delete/byquery&pretty'
+{
+  "nodes" : {
+    "s5A2CoRWrwKf512z6NEscF" : {
+      "name" : "r4A5VoT",
+      "transport_address" : "127.0.0.1:9300",
+      "host" : "127.0.0.1",
+      "ip" : "127.0.0.1:9300",
+      "attributes" : {
+        "testattr" : "test",
+        "portsfile" : "true"
+      },
+      "tasks" : {
+        "s5A2CoRWrwKf512z6NEscF" : {
+          "node" : "s5A2CoRWrwKf512z6NEscF",
+          "id" : 36619,
+          "type" : "transport",
+          "action" : "indices:data/write/delete/byquery",
+          "status" : {    
+            "total" : 6154,
+            "updated" : 0,
+            "created" : 0,
+            "deleted" : 3500,
+            "batches" : 36,
+            "version_conflicts" : 0,
+            "noops" : 0,
+            "retries": 0,
+            "throttled_millis": 0
+          },
+          "description" : ""
+        }
+      }
+    }
+  }
+}
 ```