Skip to content

Instantly share code, notes, and snippets.

@prakashsvmx

prakashsvmx/mino-batch-job-to-move-objects-and-versions-between-minio-instances-old-to-new-version.yaml

Last active April 12, 2023 09:51
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save prakashsvmx/a2be7038b9ca0337adf4191c87e3bdf1 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save prakashsvmx/a2be7038b9ca0337adf4191c87e3bdf1 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. prakashsvmx renamed this gist Apr 12, 2023. 1 changed file with 0 additions and 0 deletions.
    0 ...n-minio-instances-old-to-new-version.yaml → ...n-minio-instances-old-to-new-version.yaml
    View file
    File renamed without changes.
  2. prakashsvmx revised this gist Apr 12, 2023. 1 changed file with 3 additions and 6 deletions.
    9 changes: 3 additions & 6 deletions mino-batch-job-to-move-objects-between-minio-instances-old-to-new-version.yaml
    View file
    Original file line number Diff line number Diff line change
    @@ -3,11 +3,11 @@ replicate:
    # source of the objects to be replicated
    source:
    type: minio # valid values are "minio"
    bucket: test-v-bucket
    bucket: old-minio-versioned-bucket
    prefix: "" # 'PREFIX' is optional
    # NOTE: if source is remote then target must be "local"
    # Source is an old version of MinIO
    endpoint: "http://localhost:22001"
    endpoint: "http://localhost:15000"
    credentials:
    accessKey: minio
    secretKey: minio123
    @@ -16,7 +16,7 @@ replicate:
    # target where the objects must be replicated
    target:
    type: minio # valid values are "minio"
    bucket: test-vs-bucket
    bucket: new-minio-versioned bucket
    prefix: "" # 'PREFIX' is optional
    # NOTE: if target is remote then source must be "local"
    # Target is a new version of MinIO - This is where the objects would land.
    @@ -70,6 +70,3 @@ replicate:
    # Note the batch job id for checking status
    # mc batch status local22 <jopbid>
    # mc batch list local22
    # NOTE: If the target bucket is UN-VERSIONED, DEL markers will NOT be copied, but all OBJECT Versions WILL BE copied.
    # NOTE: If the target bucket is VERSIONED/SUSPENDED, DEL markers WILL BE copied along with other object versions.

  3. prakashsvmx created this gist Apr 3, 2023.
    75 changes: 75 additions & 0 deletions mino-batch-job-to-move-objects-between-minio-instances-old-to-new-version.yaml
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,75 @@
    replicate:
    apiVersion: v1
    # source of the objects to be replicated
    source:
    type: minio # valid values are "minio"
    bucket: test-v-bucket
    prefix: "" # 'PREFIX' is optional
    # NOTE: if source is remote then target must be "local"
    # Source is an old version of MinIO
    endpoint: "http://localhost:22001"
    credentials:
    accessKey: minio
    secretKey: minio123
    # sessionToken: SESSION-TOKEN # Optional only available when rotating credentials are used

    # target where the objects must be replicated
    target:
    type: minio # valid values are "minio"
    bucket: test-vs-bucket
    prefix: "" # 'PREFIX' is optional
    # NOTE: if target is remote then source must be "local"
    # Target is a new version of MinIO - This is where the objects would land.
    endpoint: "http://localhost:22000"
    credentials:
    accessKey: minio
    secretKey: minio123
    # sessionToken: SESSION-TOKEN # Optional only available when rotating credentials are used

    # NOTE: All flags are optional
    # - filtering criteria only applies for all source objects match the criteria
    # - configurable notification endpoints
    # - configurable retries for the job (each retry skips successfully previously replaced objects)
    #flags:
    #filter:
    #newerThan: "7d" # match objects newer than this value (e.g. 7d10h31s)
    #olderThan: "7d" # match objects older than this value (e.g. 7d10h31s)
    #createdAfter: "date" # match objects created after "date"
    #createdBefore: "date" # match objects created before "date"

    ## NOTE: tags are not supported when "source" is remote.
    # tags:
    # - key: "name"
    # value: "pick*" # match objects with tag 'name', with all values starting with 'pick'

    ## NOTE: metadata filter not supported when "source" is non MinIO.
    # metadata:
    # - key: "content-type"
    # value: "image/*" # match objects with 'content-type', with all values starting with 'image/'

    notify:
    endpoint: "http://localhost:8080" # notification endpoint to receive job status events
    #token: "Bearer xxxxx" # optional authentication token for the notification endpoint

    retry:
    attempts: 10 # number of retries for the job before giving up
    delay: "500ms" # least amount of delay between each retry


    # Running migration between 2 MinIO instances
    # Create alias for both and note the API endpoints
    # oldminio - 2022-01-08T03:11:54Z
    # local22 - 2023-03-31T21:38:10Z

    # run the commands on the local22 instance of MinIO

    # mc version RELEASE.2023-03-23T20-03-04Z
    # mc batch generate local22 replicate > replication.yaml
    # Update as per the above configuration . comment un wanted blocks like filter
    # mc batch start local22 ./replication.yaml
    # Note the batch job id for checking status
    # mc batch status local22 <jopbid>
    # mc batch list local22
    # NOTE: If the target bucket is UN-VERSIONED, DEL markers will NOT be copied, but all OBJECT Versions WILL BE copied.
    # NOTE: If the target bucket is VERSIONED/SUSPENDED, DEL markers WILL BE copied along with other object versions.