Skip to content

Instantly share code, notes, and snippets.

@feelinc

feelinc/UploadDirS3.py

Last active October 8, 2024 20:55
Show Gist options
  • Save feelinc/d1f541af4f31d09a2ec3 to your computer and use it in GitHub Desktop.
Save feelinc/d1f541af4f31d09a2ec3 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. feelinc revised this gist Aug 25, 2015. 1 changed file with 3 additions and 0 deletions.
    3 changes: 3 additions & 0 deletions usage.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,3 @@
    UploadDirS3.py /path/to/local/folder thebucketname /path/to/s3/folder

    Make sure to read the boto3 credentials placement doc
  2. feelinc created this gist Aug 25, 2015.
    38 changes: 38 additions & 0 deletions UploadDirS3.py
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,38 @@
    #!/usr/bin/python

    import os
    import sys
    import boto3

    # get an access token, local (from) directory, and S3 (to) directory
    # from the command-line
    local_directory, bucket, destination = sys.argv[1:4]

    client = boto3.client('s3')

    # enumerate local files recursively
    for root, dirs, files in os.walk(local_directory):

    for filename in files:

    # construct the full local path
    local_path = os.path.join(root, filename)

    # construct the full Dropbox path
    relative_path = os.path.relpath(local_path, local_directory)
    s3_path = os.path.join(destination, relative_path)

    # relative_path = os.path.relpath(os.path.join(root, filename))

    print 'Searching "%s" in "%s"' % (s3_path, bucket)
    try:
    client.head_object(Bucket=bucket, Key=s3_path)
    print "Path found on S3! Skipping %s..." % s3_path

    # try:
    # client.delete_object(Bucket=bucket, Key=s3_path)
    # except:
    # print "Unable to delete %s..." % s3_path
    except:
    print "Uploading %s..." % s3_path
    client.upload_file(local_path, bucket, s3_path)