Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save sunyi00/be64d5c55bbcc4c03e38a6858e7031bf to your computer and use it in GitHub Desktop.
Save sunyi00/be64d5c55bbcc4c03e38a6858e7031bf to your computer and use it in GitHub Desktop.

Revisions

  1. @lucidlibrary lucidlibrary revised this gist Apr 19, 2015. 1 changed file with 1 addition and 2 deletions.
    3 changes: 1 addition & 2 deletions elasticsearch-snapshot-restore.md
    Original file line number Diff line number Diff line change
    @@ -1,5 +1,4 @@

    Elasticsearch introduced [Snapshot and Restore](http://blog.lucidlibrary.com/elasticsearch-snapshot-and-restore/) API in Elasticsearch 1.0. With this module you can take backup/restore of the data easily.
    [Elasticsearch](http://www.lucidlibrary.com/subject/Elasticsearch/128) introduced [Snapshot and Restore](http://blog.lucidlibrary.com/elasticsearch-snapshot-and-restore/) API in Elasticsearch 1.0. With this module you can take backup/restore of the data easily.

    <p>To take snapshots or to restore them, first you need to create a repository. A repository is a just like allocating an address to your snapshots.</p>

  2. @lucidlibrary lucidlibrary revised this gist Apr 19, 2015. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion elasticsearch-snapshot-restore.md
    Original file line number Diff line number Diff line change
    @@ -1,4 +1,5 @@
    Elasticsearch introduced Snapshot and Restore API in Elasticsearch 1.0. With this module you can take backup/restore of the data easily.

    Elasticsearch introduced [Snapshot and Restore](http://blog.lucidlibrary.com/elasticsearch-snapshot-and-restore/) API in Elasticsearch 1.0. With this module you can take backup/restore of the data easily.

    <p>To take snapshots or to restore them, first you need to create a repository. A repository is a just like allocating an address to your snapshots.</p>

  3. @lucidlibrary lucidlibrary created this gist Apr 19, 2015.
    126 changes: 126 additions & 0 deletions elasticsearch-snapshot-restore.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,126 @@
    Elasticsearch introduced Snapshot and Restore API in Elasticsearch 1.0. With this module you can take backup/restore of the data easily.

    <p>To take snapshots or to restore them, first you need to create a repository. A repository is a just like allocating an address to your snapshots.</p>

    <p>A repository can contain as many snapshots as you would like, and you can create any number of repositories.</p>

    <p>Each repository is mapped to a location where your snapshot files will be stored. Elasticsearch first started with snapshots to FileSystem location, now you can take snapshots to Remove locations like AWS S3 directly.</p>

    <p>Elasticsearch currently supports snapshtos to <a title="AWS S3" href="https://github.com/elasticsearch/elasticsearch-cloud-aws#s3-repository" target="_blank">AWS S3</a>, <a title="HDFS" href="https://github.com/elasticsearch/elasticsearch-hadoop/tree/master/repository-hdfs" target="_blank">HDFS</a>, <a href="https://github.com/elasticsearch/elasticsearch-cloud-azure#azure-repository" target="_blank">Azure</a> .</p>

    <h3>Creating a file system repository</h3>
    ```
    curl -XPUT 'http://localhost:9200/_snapshot/my_fs_repository' -d '{
    "type": "fs",
    "settings": {
    "location": "/data/es-backups/fs-repository",
    "compress": true
    }
    }'
    ```

    where `my_fs_repository` is repository name and `/data/es-backups/fs-reposiotry` is the location of the repository

    You need to create this location and elasticsearch user should have permissions to access this location.
    ```
    sudo chown elasticsearch /data/es-backups/fs-repository
    sudo chgrp elasticsearch /data/es-backups/fs-repository
    ```

    #### Creating a snapshot
    ```
    curl -XPUT "localhost:9200/_snapshot/my_fs_repository/snapshot_name?wait_for_completion=true" -d '{
    "indices": "indice_1,indice_2,indice_3",
    "ignore_unavailable": "true",
    "include_global_state": "false"
    }'
    ```
    This way you can specify which indices you want to take snapshot. All indices will be part of the snapshot if `indices` parameter not specified.
    `include_global_state` parameter will decide whether to store the cluster meta data or not, which includes persistent cluster settings and index templates

    #### Restore a snapshot
    ```
    curl -XPOST "localhost:9200/_snapshot/my_fs_repository/snapshot_name/_restore" -d '{
    "indices": "indice_1,indice_2",
    "ignore_unavailable": "true"
    }'
    ```
    #### Restore a snapshot with different indice name
    ```
    curl -XPOST "localhost:9200/_snapshot/my_fs_repository/snapshot_name/_restore" -d '{
    "indices": "index_1,index_2",
    "ignore_unavailable": "true",
    "include_global_state": "false",
    "rename_pattern": "index_(.+)",
    "rename_replacement": "restored_index_$1"
    }'
    ```

    If you have got the snapshot data from a different machine, you will have to place those files in the repository location, snapshot name can be found from looking at the files
    ```
    $ ls -lh /data/es-backups/fs-repository
    -rw-r--r-- 1 elasticsearch elasticsearch 34 Apr 19 17:01 index
    drwxr-xr-x 3 elasticsearch elasticsearch 4.0K Apr 19 17:01 indices
    -rw-r--r-- 1 elasticsearch elasticsearch 61 Apr 19 17:01 metadata-snapshot_name
    -rw-r--r-- 1 elasticsearch elasticsearch 193 Apr 19 17:01 snapshot-snapshot_name
    ```
    #### Get the list of all snapshots
    Or, you can get the list of all available snapshots in a repository by
    `curl -XGET "localhost:9200/_snapshot/my_fs_repository/_all?pretty"`

    #### Delete a snapshot
    `curl -XDELETE "localhost:9200/_snapshot/my_fs_repository/snapshot_name"`
    #### Create S3 repository
    [elsticsearch-cloud-aws](https://github.com/elastic/elasticsearch-cloud-aws) plugin needs to be installed to be able to use S3 for snapshot location.

    /etc/elasticsearch/elasticsearch.yml
    ```
    cloud:
    aws:
    access_key: AWS_ACCESS_KEY_ID
    secret_key: AWS_SECRET_ACCESS_KEY
    repositories:
    s3:
    bucket: "es-snapshots"
    region: "us-west-1"
    ```
    S3 bucket policy
    ```
    {
    "Statement": [
    {
    "Action": [
    "s3:ListBucket"
    ],
    "Effect": "Allow",
    "Resource": [
    "arn:aws:s3:::es-snapshots"
    ]
    },
    {
    "Action": [
    "s3:GetObject",
    "s3:PutObject",
    "s3:DeleteObject"
    ],
    "Effect": "Allow",
    "Resource": [
    "arn:aws:s3:::es-snapshots/*"
    ]
    }
    ],
    "Version": "2012-10-17"
    }
    ```
    create a bucket and a directory (optional) to use as the location for S3

    ```
    curl -XPUT 'http://localhost:9200/_snapshot/my_s3_repository' -d '{
    "type": "s3",
    "settings": {
    "bucket": "es-snapshots",
    "region": "us-west-1",
    "base_path": "my_cluster"
    }
    }'
    ```