Forked from lucidlibrary/elasticsearch-snapshot-restore.md
Created
September 17, 2016 10:18
-
-
Save sunyi00/be64d5c55bbcc4c03e38a6858e7031bf to your computer and use it in GitHub Desktop.
Revisions
-
lucidlibrary revised this gist
Apr 19, 2015 . 1 changed file with 1 addition and 2 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,5 +1,4 @@ [Elasticsearch](http://www.lucidlibrary.com/subject/Elasticsearch/128) introduced [Snapshot and Restore](http://blog.lucidlibrary.com/elasticsearch-snapshot-and-restore/) API in Elasticsearch 1.0. With this module you can take backup/restore of the data easily. <p>To take snapshots or to restore them, first you need to create a repository. A repository is a just like allocating an address to your snapshots.</p> -
lucidlibrary revised this gist
Apr 19, 2015 . 1 changed file with 2 additions and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,4 +1,5 @@ Elasticsearch introduced [Snapshot and Restore](http://blog.lucidlibrary.com/elasticsearch-snapshot-and-restore/) API in Elasticsearch 1.0. With this module you can take backup/restore of the data easily. <p>To take snapshots or to restore them, first you need to create a repository. A repository is a just like allocating an address to your snapshots.</p> -
lucidlibrary created this gist
Apr 19, 2015 .There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,126 @@ Elasticsearch introduced Snapshot and Restore API in Elasticsearch 1.0. With this module you can take backup/restore of the data easily. <p>To take snapshots or to restore them, first you need to create a repository. A repository is a just like allocating an address to your snapshots.</p> <p>A repository can contain as many snapshots as you would like, and you can create any number of repositories.</p> <p>Each repository is mapped to a location where your snapshot files will be stored. Elasticsearch first started with snapshots to FileSystem location, now you can take snapshots to Remove locations like AWS S3 directly.</p> <p>Elasticsearch currently supports snapshtos to <a title="AWS S3" href="https://github.com/elasticsearch/elasticsearch-cloud-aws#s3-repository" target="_blank">AWS S3</a>, <a title="HDFS" href="https://github.com/elasticsearch/elasticsearch-hadoop/tree/master/repository-hdfs" target="_blank">HDFS</a>, <a href="https://github.com/elasticsearch/elasticsearch-cloud-azure#azure-repository" target="_blank">Azure</a> .</p> <h3>Creating a file system repository</h3> ``` curl -XPUT 'http://localhost:9200/_snapshot/my_fs_repository' -d '{ "type": "fs", "settings": { "location": "/data/es-backups/fs-repository", "compress": true } }' ``` where `my_fs_repository` is repository name and `/data/es-backups/fs-reposiotry` is the location of the repository You need to create this location and elasticsearch user should have permissions to access this location. ``` sudo chown elasticsearch /data/es-backups/fs-repository sudo chgrp elasticsearch /data/es-backups/fs-repository ``` #### Creating a snapshot ``` curl -XPUT "localhost:9200/_snapshot/my_fs_repository/snapshot_name?wait_for_completion=true" -d '{ "indices": "indice_1,indice_2,indice_3", "ignore_unavailable": "true", "include_global_state": "false" }' ``` This way you can specify which indices you want to take snapshot. All indices will be part of the snapshot if `indices` parameter not specified. `include_global_state` parameter will decide whether to store the cluster meta data or not, which includes persistent cluster settings and index templates #### Restore a snapshot ``` curl -XPOST "localhost:9200/_snapshot/my_fs_repository/snapshot_name/_restore" -d '{ "indices": "indice_1,indice_2", "ignore_unavailable": "true" }' ``` #### Restore a snapshot with different indice name ``` curl -XPOST "localhost:9200/_snapshot/my_fs_repository/snapshot_name/_restore" -d '{ "indices": "index_1,index_2", "ignore_unavailable": "true", "include_global_state": "false", "rename_pattern": "index_(.+)", "rename_replacement": "restored_index_$1" }' ``` If you have got the snapshot data from a different machine, you will have to place those files in the repository location, snapshot name can be found from looking at the files ``` $ ls -lh /data/es-backups/fs-repository -rw-r--r-- 1 elasticsearch elasticsearch 34 Apr 19 17:01 index drwxr-xr-x 3 elasticsearch elasticsearch 4.0K Apr 19 17:01 indices -rw-r--r-- 1 elasticsearch elasticsearch 61 Apr 19 17:01 metadata-snapshot_name -rw-r--r-- 1 elasticsearch elasticsearch 193 Apr 19 17:01 snapshot-snapshot_name ``` #### Get the list of all snapshots Or, you can get the list of all available snapshots in a repository by `curl -XGET "localhost:9200/_snapshot/my_fs_repository/_all?pretty"` #### Delete a snapshot `curl -XDELETE "localhost:9200/_snapshot/my_fs_repository/snapshot_name"` #### Create S3 repository [elsticsearch-cloud-aws](https://github.com/elastic/elasticsearch-cloud-aws) plugin needs to be installed to be able to use S3 for snapshot location. /etc/elasticsearch/elasticsearch.yml ``` cloud: aws: access_key: AWS_ACCESS_KEY_ID secret_key: AWS_SECRET_ACCESS_KEY repositories: s3: bucket: "es-snapshots" region: "us-west-1" ``` S3 bucket policy ``` { "Statement": [ { "Action": [ "s3:ListBucket" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::es-snapshots" ] }, { "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::es-snapshots/*" ] } ], "Version": "2012-10-17" } ``` create a bucket and a directory (optional) to use as the location for S3 ``` curl -XPUT 'http://localhost:9200/_snapshot/my_s3_repository' -d '{ "type": "s3", "settings": { "bucket": "es-snapshots", "region": "us-west-1", "base_path": "my_cluster" } }' ```