Skip to content

Instantly share code, notes, and snippets.

@pseudokool

pseudokool/google-drive-to-aws-s-using-rclone-helper.sh

Created March 19, 2025 08:47
Show Gist options
  • Save pseudokool/badf1fe766ed5df4ca970582537cfca9 to your computer and use it in GitHub Desktop.
Save pseudokool/badf1fe766ed5df4ca970582537cfca9 to your computer and use it in GitHub Desktop.
Download ZIP
google-drive-to-aws-s-using-rclone-helper
Raw
google-drive-to-aws-s-using-rclone-helper.sh
# Function to log messages with timestamp and level
log() {
local timestamp=$(date +"%Y-%m-%d %H:%M:%S")
local level="$1"
local message="$2"
echo "[$timestamp] [$level] $message"
}
# Google Drive Relative Path
SOURCE_FOLDER="<google-drive-folder-path>"
# S3 Path
BUCKET="<s3-bucket-name>"
DESTINATION_FOLDER="<bucket-folder-path>"
log "DEBUG" "Copying from SOURCE_FOLDER $SOURCE_FOLDER"
log "DEBUG" "Copying to DESTINATION_FOLDER $BUCKET$DESTINATION_FOLDER"
# Do transfer
log "INFO" "Initiating COPY"
rclone copy \
--tpslimit 200 \
--transfers 200 \
--buffer-size 200M \
--checkers 400 \
--s3-upload-cutoff 100M \
--s3-chunk-size 100M \
--s3-upload-concurrency 50 \
gdrive-remote:"${SOURCE_FOLDER}" \
s3-remote:"${BUCKET}${DESTINATION_FOLDER}" \
-P
log "INFO" "Transfer complete"
# Do checks
log "INFO" "Performing checks based on file sizes at source & destination"
rclone check \
--tpslimit 200 \
--transfers 200 \
--buffer-size 200M \
--checkers 400 \
--s3-upload-cutoff 100M \
--s3-chunk-size 100M \
--s3-upload-concurrency 50 \
gdrive-remote:"${SOURCE_FOLDER}" \
s3-remote:"${BUCKET}${DESTINATION_FOLDER}" \
-P
log "INFO" "Operations complete!"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment