-
-
Save SavvyGuard/6115006 to your computer and use it in GitHub Desktop.
| import boto | |
| # Fill these in - you get them when you sign up for S3 | |
| AWS_USER_NAME = '' | |
| AWS_ACCESS_KEY_ID = '' | |
| AWS_SECRET_ACCESS_KEY = '' | |
| bucket_name = 'dvargas' | |
| dirpath = 'data/Standard/' | |
| conn = boto.connect_s3(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) | |
| import boto.s3 | |
| bucket = conn.create_bucket(bucket_name, | |
| location=boto.s3.connection.Location.DEFAULT) | |
| from os import walk | |
| testfilenames = [] | |
| for (dirpath, dirnames, filenames) in walk(dirpath): | |
| testfilenames.extend(filenames) | |
| break | |
| for testfile in testfilenames: | |
| print 'Uploading %s to Amazon S3 bucket %s' % \ | |
| (dirpath + testfile, bucket_name) | |
| import sys | |
| def percent_cb(complete, total): | |
| sys.stdout.write('.') | |
| sys.stdout.flush() | |
| from boto.s3.key import Key | |
| k = Key(bucket) | |
| k.key = 'data/Standard/' + testfile | |
| k.set_contents_from_filename(dirpath + testfile, | |
| cb=percent_cb, num_cb=10) |
This also creates the bucket, and errors out if the bucket already exists. Would I simply comment out:
bucket = conn.create_bucket(bucket_name,
location=boto.s3.connection.Location.DEFAULT)
to bypass that?
Ah, got it figured out. Simply replace:
bucket = conn.create_bucket(bucket_name,
location=boto.s3.connection.Location.DEFAULT)
with
bucket = conn.get_bucket(bucket_name)
Using this technique can you upload files within a sub directory inside a directory?
This is great, just what I'm after. But how can I extract the AWS_ACCESS_KEY_ID and AWS_ACCESS_KEY_SECRET in my Python script running within Bitbucket Pipelines? Other scripts I have seen make no reference to these two properties, but connect like this:
import boto3
client = boto3.client('s3')
Although the methods are different on boto3 so it's a bit tricky...advice?
@mark-norgate The example from Amazon for Bitbucket pipelines says to set environment variables which are automatically picked up by the boto3 library. linky
Also, thanks for the script!
You also need to add '/'
sourcepath = os.path.join(sourceDir + '/' + filename)
Won't do it recursively for sub-directories
While syncing directory to aws server by using this code only one file is uploading where as this directory is contains 3 files. Please help me to solve this problem.
Thanks Man!
This code is amazing! Thank you @SavvyGuard !
bucket = conn.create_bucket(bucket_name, location='https://s3.eu-central-1.wasabisys.com')
S3ResponseError: S3ResponseError: 403 Forbidden
InvalidAccessKeyIdThe AWS Access Key Id you provided does not exist in our records.NOCC4UFL6U659XNJHGFME3M762MSK2KPQJMElWA0cWuCCEAOw9ObIyTn8GGe1ErsEdJeTw8aHfPX5T09QSDYT3jElLqAsGv/LPcIJhH+5ncuBdU=
what's wrong?
Thank you!