Skip to content

Instantly share code, notes, and snippets.

@fragolinux

fragolinux/velero-csi-plugin-demo.md

Forked from ashish-amarnath/velero-csi-plugin-demo.md
Created July 15, 2021 09:20
Show Gist options
  • Save fragolinux/ffd037421d422dc90d4af515961ccd0e to your computer and use it in GitHub Desktop.
Save fragolinux/ffd037421d422dc90d4af515961ccd0e to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. @ashish-amarnath ashish-amarnath revised this gist Oct 29, 2020. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions velero-csi-plugin-demo.md
    View file
    Original file line number Diff line number Diff line change
    @@ -16,11 +16,11 @@ Below are the updated set of instructions to
    VELERO_IMAGE=velero/velero:v1.5.2
    VELERO_PLUGINS=velero/velero-plugin-for-microsoft-azure:v1.1.1,velero/velero-plugin-for-csi:v0.1.2
    source $HOME/bin/velero-dev/aks-env
    /Users/ashisham/devnull/velero-1.5/velero-v1.5.2-darwin-amd64/velero install \
    velero install \
    --provider azure \
    --plugins ${VELERO_PLUGINS} \
    --bucket $BLOB_CONTAINER \
    --secret-file /Users/ashisham/bin/velero-dev/aks-creds \
    --secret-file aks-creds \
    --backup-location-config resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID \
    --snapshot-location-config apiTimeout=5m,resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID \
    --image ${VELERO_IMAGE} \
  2. @ashish-amarnath ashish-amarnath revised this gist Oct 29, 2020. No changes.
  3. @ashish-amarnath ashish-amarnath created this gist Oct 29, 2020.
    249 changes: 249 additions & 0 deletions velero-csi-plugin-demo.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,249 @@
    This gist is a follow-up to the [CSI Integration blog post](https://velero.io/blog/csi-integration/0) on the Velero website.
    Updates here include:
    1. Using [`v0.1.2` release of `velero-plugin-for-csi`](https://github.com/vmware-tanzu/velero-plugin-for-csi/releases/tag/v0.1.2)
    1. Using `[v1.1.1` release of `velero-plugin-for-microsoft-azure`](https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/releases/tag/v1.1.1)
    1. Using [`v1.5.2` release of Velero](https://github.com/vmware-tanzu/velero/releases/tag/v1.5.2)


    Below are the updated set of instructions to
    1. Create AKS cluster with kubernetes version v1.17 or later. In this example an AKS cluster running `v1.18.6` is used.
    1. Install CSI driver
    ```bash
    curl -skSL https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/deploy/install-driver.sh | bash -s master snapshot --
    ```
    1. Install Velero
    ```bash
    VELERO_IMAGE=velero/velero:v1.5.2
    VELERO_PLUGINS=velero/velero-plugin-for-microsoft-azure:v1.1.1,velero/velero-plugin-for-csi:v0.1.2
    source $HOME/bin/velero-dev/aks-env
    /Users/ashisham/devnull/velero-1.5/velero-v1.5.2-darwin-amd64/velero install \
    --provider azure \
    --plugins ${VELERO_PLUGINS} \
    --bucket $BLOB_CONTAINER \
    --secret-file /Users/ashisham/bin/velero-dev/aks-creds \
    --backup-location-config resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID \
    --snapshot-location-config apiTimeout=5m,resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID \
    --image ${VELERO_IMAGE} \
    --features=EnableCSI
    ```
    1. Verify installed Velero version and the version of the CLI
    ```bash
    $ velero version
    Client:
    Version: v1.5.2
    Git commit: e115e5a191b1fdb5d379b62a35916115e77124a4
    Server:
    Version: v1.5.2
    ```
    1. Verify installation of `velero-plugin-for-microsfot-azure` and `velero-plugin-for-csi`
    ```bash
    $ velero plugin get | grep csi
    velero.io/csi-pvc-backupper BackupItemAction
    velero.io/csi-volumesnapshot-backupper BackupItemAction
    velero.io/csi-volumesnapshotclass-backupper BackupItemAction
    velero.io/csi-volumesnapshotcontent-backupper BackupItemAction
    velero.io/csi-pvc-restorer RestoreItemAction
    velero.io/csi-volumesnapshot-restorer RestoreItemAction
    velero.io/csi-volumesnapshotclass-restorer RestoreItemAction
    velero.io/csi-volumesnapshotcontent-restorer RestoreItemAction
    $ velero plugin get | grep azure
    velero.io/azure ObjectStore
    velero.io/azure VolumeSnapshotter
    ```
    1. Create storage class and volumesnapshot class
    ```yaml
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
    name: disk.csi.azure.com
    provisioner: disk.csi.azure.com
    parameters:
    skuname: StandardSSD_LRS
    reclaimPolicy: Delete
    volumeBindingMode: Immediate
    allowVolumeExpansion: true
    ---
    apiVersion: snapshot.storage.k8s.io/v1beta1
    kind: VolumeSnapshotClass
    metadata:
    name: csi-azuredisk-vsc
    labels:
    velero.io/csi-volumesnapshot-class: "true"
    driver: disk.csi.azure.com
    deletionPolicy: Retain
    parameters:
    tags: 'foo=aaa,bar=bbb'
    ```
    From [StorageClass](https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/deploy/example/storageclass-azuredisk-csi.yaml) and [VolumeSnapshotClass](https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/deploy/example/snapshot/storageclass-azuredisk-snapshot.yaml).
    NOTE: The volumesnapshotclass was edited to add the `velero.io/csi-volumesnapshot-class: "true"` annotation.
    1. Deploy a sample CSI app
    ```yaml
    apiVersion: v1
    kind: Namespace
    metadata:
    creationTimestamp: null
    name: csi-app
    ---
    kind: Pod
    apiVersion: v1
    metadata:
    namespace: csi-app
    name: csi-nginx
    spec:
    nodeSelector:
    kubernetes.io/os: linux
    containers:
    - image: nginx
    name: nginx
    command: [ "sleep", "1000000" ]
    volumeMounts:
    - name: azuredisk01
    mountPath: "/mnt/azuredisk"
    volumes:
    - name: azuredisk01
    persistentVolumeClaim:
    claimName: pvc-azuredisk
    ---
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
    namespace: csi-app
    name: pvc-azuredisk
    spec:
    accessModes:
    - ReadWriteOnce
    resources:
    requests:
    storage: 1Gi
    storageClassName: disk.csi.azure.com
    ---
    ```
    1. Make our sample app generate some data into the PV
    ```bash
    $ kubectl -n csi-app exec -ti csi-nginx -- bash
    root@csi-nginx:/# fallocate -l 750M /mnt/azuredisk/velero-data
    root@csi-nginx:/# cksum /mnt/azuredisk/velero-data
    3090901991 786432000 /mnt/azuredisk/velero-data
    root@csi-nginx:/#
    ```
    1. Take a velero backup
    ```bash
    $ velero backup create csi-b1 --include-namespaces csi-app --wait
    Backup request "csi-b1" submitted successfully.
    Waiting for backup to complete. You may safely press ctrl-c to stop waiting - your backup will continue in the background.
    .......
    Backup completed with status: Completed. You may check for more information using the commands `velero backup describe csi-b1` and `velero backup logs csi-b1`.
    ```
    1. Describe the backup to confirm that the CSI volumesnapshots were included in the backup
    ```bash
    $ velero backup describe csi-b1 --details --features=EnableCSI
    Name: csi-b1
    Namespace: velero
    Labels: velero.io/storage-location=default
    Annotations: velero.io/source-cluster-k8s-gitversion=v1.18.6
    velero.io/source-cluster-k8s-major-version=1
    velero.io/source-cluster-k8s-minor-version=18
    Phase: Completed
    Errors: 0
    Warnings: 0
    Namespaces:
    Included: csi-app
    Excluded: <none>
    Resources:
    Included: *
    Excluded: <none>
    Cluster-scoped: auto
    Label selector: <none>
    Storage Location: default
    Velero-Native Snapshot PVs: auto
    TTL: 720h0m0s
    Hooks: <none>
    Backup Format Version: 1.1.0
    Started: 2020-10-29 15:20:57 -0700 PDT
    Completed: 2020-10-29 15:21:03 -0700 PDT
    Expiration: 2020-11-28 14:20:57 -0800 PST
    Total items to be backed up: 20
    Items backed up: 20
    Resource List:
    snapshot.storage.k8s.io/v1beta1/VolumeSnapshot:
    - csi-app/velero-pvc-azuredisk-7dbp7
    snapshot.storage.k8s.io/v1beta1/VolumeSnapshotClass:
    - csi-azuredisk-vsc
    snapshot.storage.k8s.io/v1beta1/VolumeSnapshotContent:
    - snapcontent-2934eb73-6dd3-494a-91ef-7e52ad8115b9
    v1/Event:
    - csi-app/csi-nginx.1642958b3a639423
    - csi-app/csi-nginx.1642958bd742db20
    - csi-app/csi-nginx.1642958df050f87b
    - csi-app/csi-nginx.16429592eb598457
    - csi-app/csi-nginx.1642959651292777
    - csi-app/csi-nginx.164295970bedfe18
    - csi-app/csi-nginx.16429597e182a29f
    - csi-app/csi-nginx.16429597eae448f1
    - csi-app/pvc-azuredisk.1642958b4da4a67a
    - csi-app/pvc-azuredisk.1642958b4dce84d8
    - csi-app/pvc-azuredisk.1642958bf941e107
    v1/Namespace:
    - csi-app
    v1/PersistentVolume:
    - pvc-ba8cc2e9-8255-4bda-bace-013f1036d693
    v1/PersistentVolumeClaim:
    - csi-app/pvc-azuredisk
    v1/Pod:
    - csi-app/csi-nginx
    v1/Secret:
    - csi-app/default-token-vsh5n
    v1/ServiceAccount:
    - csi-app/default
    Velero-Native Snapshots: <none included>
    CSI Volume Snapshots:
    Snapshot Content Name: snapcontent-2934eb73-6dd3-494a-91ef-7e52ad8115b9
    Storage Snapshot ID: /subscriptions/489d6686-11b0-4662-a8e5-4143cd7cd6af/resourceGroups/mc_ashisha-aks_velero-dev_eastus/providers/Microsoft.Compute/snapshots/snapshot-2934eb73-6dd3-494a-91ef-7e52ad8115b9
    Snapshot Size (bytes): 1073741824
    Ready to use: true
    ```
    1. Simulate a disaster, Delete the `csi-app` namespace
    ```bash
    $ kubectl delete ns csi-app
    namespace "csi-app" deleted
    $ kubectl get ns csi-app
    Error from server (NotFound): namespaces "csi-app" not found
    ```
    1. Restore from the backup taken in above
    ```bash
    $ velero create restore --from-backup csi-b1 --wait
    Restore request "csi-b1-20201029153424" submitted successfully.
    Waiting for restore to complete. You may safely press ctrl-c to stop waiting - your restore will continue in the background.
    .
    Restore completed with status: Completed. You may check for more information using the commands `velero restore describe csi-b1-20201029153424` and `velero restore logs csi-b1-20201029153424`.
    ```
    1. Verify that the `csi-app` was restored completely with the data intact
    ```bash
    $ kubectl get ns csi-app
    NAME STATUS AGE
    csi-app Active 81s
    $ kubectl -n csi-app exec -ti csi-nginx -- bash
    root@csi-nginx:/# ls /mnt/azuredisk/
    lost+found/ velero-data
    root@csi-nginx:/# ls /mnt/azuredisk/velero-data
    /mnt/azuredisk/velero-data
    root@csi-nginx:/# cksum /mnt/azuredisk/velero-data
    3090901991 786432000 /mnt/azuredisk/velero-data
    root@csi-nginx:/#
    ```