Skip to content

Instantly share code, notes, and snippets.

@prasanthj

prasanthj/s3-sync-debian

Last active May 19, 2020 01:44
Show Gist options
  • Save prasanthj/a463bc1c78b8aed57752e13e8cc39c56 to your computer and use it in GitHub Desktop.
Save prasanthj/a463bc1c78b8aed57752e13e8cc39c56 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. prasanthj revised this gist May 19, 2020. 1 changed file with 6 additions and 0 deletions.
    6 changes: 6 additions & 0 deletions s3-sync-debian
    View file
    Original file line number Diff line number Diff line change
    @@ -20,3 +20,9 @@ NOTE: the aws credentials above should have sufficient permissions to write to d
    should also have permissions for encrypting the data in destination bucket if SSE KMS is specified.

    aws s3 sync s3://<SRC-BUCKET>/<PATH-PREFIX>/tpcds_bin_partitioned_orc_10000.db/ s3://<DEST-BUCKET>/<PATH-PREFIX>/tpcds_bin_partitioned_orc_10000.db/ --sse aws:kms --sse-kms-key-id <KMS-KEY-ID>

    Completed 2.2 TiB/2.2 TiB (3.1 GiB/s) with 0 file(s) remaining
    real 12m0.114s
    user 13m29.787s
    sys 5m44.783s
    [1]+ Done time aws s3 sync s3://source/bucket/path/to/warehouse-id/warehouse/tablespace/managed/hive/tpcds_copy_orc_partitioned_10000.db/ s3://destination/bucket/path/to/warehouse-id/warehouse/tablespace/managed/hive/tpcds_copy_orc_partitioned_10000.db/ --sse aws:kms --sse-kms-key-id <destination-encryption-key>
  2. prasanthj revised this gist Jun 6, 2019. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion s3-sync-debian
    View file
    Original file line number Diff line number Diff line change
    @@ -1,9 +1,9 @@
    apt-get update
    apt-get install -y python curl vim
    curl -O https://bootstrap.pypa.io/get-pip.py
    export PATH=~/.local/bin:$PATH
    python get-pip.py --user
    pip install awscli --upgrade --user
    export PATH=~/.local/bin:$PATH

    mkdir -p ~/.aws
    vim ~/.aws/config
  3. prasanthj created this gist May 7, 2019.
    22 changes: 22 additions & 0 deletions s3-sync-debian
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,22 @@
    apt-get update
    apt-get install -y python curl vim
    curl -O https://bootstrap.pypa.io/get-pip.py
    python get-pip.py --user
    pip install awscli --upgrade --user
    export PATH=~/.local/bin:$PATH

    mkdir -p ~/.aws
    vim ~/.aws/config
    [default]
    aws_access_key_id=<AWS-ACCESS-KEY-HERE>
    aws_secret_access_key=<AWS-SECRET-KEY-HERE>
    s3 =
    max_concurrent_requests = 200
    max_queue_size = 10000
    multipart_threshold = 64MB
    multipart_chunksize = 16MB

    NOTE: the aws credentials above should have sufficient permissions to write to destination bucket and
    should also have permissions for encrypting the data in destination bucket if SSE KMS is specified.

    aws s3 sync s3://<SRC-BUCKET>/<PATH-PREFIX>/tpcds_bin_partitioned_orc_10000.db/ s3://<DEST-BUCKET>/<PATH-PREFIX>/tpcds_bin_partitioned_orc_10000.db/ --sse aws:kms --sse-kms-key-id <KMS-KEY-ID>