Skip to content

Instantly share code, notes, and snippets.

@notmatt

notmatt/manatee_brain_transplant.md

Last active August 29, 2015 14:14
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save notmatt/b24bb67ffda33b61504a to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save notmatt/b24bb67ffda33b61504a to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. notmatt revised this gist Apr 9, 2015. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion manatee_brain_transplant.md
    View file
    Original file line number Diff line number Diff line change
    @@ -164,7 +164,7 @@ A slight modification of the steps outlined [here](https://github.com/joyent/man
    - `sapiadm update $V2_PRIMARY metadata.DISABLE_SITTER=false`
    - halt primary: `sdc-oneachnode -n $CN_V2_PRIMARY "vmadm stop $V2_PRIMARY"`
    - run manatee-adm unfreeze in the reprovisioned sync: `sdc-oneachnode -n $CN_SYNC "zlogin $SYNC 'manatee-adm unfreeze' < /dev/null"`
    - start v2 primary: `sdc-oneachnode $CN_V2_PRIMARY "vmadm start $V2_PRIMARY"`
    - start v2 primary: `sdc-oneachnode -n $CN_V2_PRIMARY "vmadm start $V2_PRIMARY"`

    The modification takes advantage of the fact that manatee-sitter does not immediately pick up new config, so we can use SAPI to write the new config file, but it will not take effect until after manatee is stopped and restarted.

  2. notmatt revised this gist Apr 9, 2015. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions manatee_brain_transplant.md
    View file
    Original file line number Diff line number Diff line change
    @@ -162,8 +162,8 @@ A slight modification of the steps outlined [here](https://github.com/joyent/man
    - remove metadata from sapi:
    - `sapiadm update $V2_PRIMARY metadata.ONE_NODE_WRITE_MODE=false`
    - `sapiadm update $V2_PRIMARY metadata.DISABLE_SITTER=false`
    - halt primary: `sdc-oneachnode $CN_V2_PRIMARY "vamdm stop $V2_PRIMARY"`
    - run manatee-adm unfreeze in the reprovisioned sync: `sdc-oneachnode $CN_SYNC "zlogin $SYNC 'manatee-adm unfreeze' < /dev/null"`
    - halt primary: `sdc-oneachnode -n $CN_V2_PRIMARY "vmadm stop $V2_PRIMARY"`
    - run manatee-adm unfreeze in the reprovisioned sync: `sdc-oneachnode -n $CN_SYNC "zlogin $SYNC 'manatee-adm unfreeze' < /dev/null"`
    - start v2 primary: `sdc-oneachnode $CN_V2_PRIMARY "vmadm start $V2_PRIMARY"`

    The modification takes advantage of the fact that manatee-sitter does not immediately pick up new config, so we can use SAPI to write the new config file, but it will not take effect until after manatee is stopped and restarted.
  3. notmatt revised this gist Apr 9, 2015. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion manatee_brain_transplant.md
    View file
    Original file line number Diff line number Diff line change
    @@ -89,7 +89,7 @@ cat > manatee3.json <<EOF
    "service_uuid": "$MANATEE_SVC",
    "params": {
    "alias": "manatee3",
    "server_uuid: "$CN_PRIMARY"
    "server_uuid": "$CN_PRIMARY"
    },
    "metadata": {
    "DISABLE_SITTER": true,
  4. notmatt revised this gist Mar 30, 2015. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion manatee_brain_transplant.md
    View file
    Original file line number Diff line number Diff line change
    @@ -154,7 +154,7 @@ When the zfs send has completed, you can `svcadm enable manatee-sitter` in the n

    ## step 8. move manatee v2 out of ONWM

    A slight modification of the steps outlined [here](https://github.com/joyent/manatee/blob/master/docs/user-guide.md#one-node-write-mode). It assumes $CN_V2_PRIMARY is the CN on which you [re]provisioned manateein step 4, and $V2_PRIMARY is the manatee v2 primary.
    A slight modification of the steps outlined [here](https://github.com/joyent/manatee/blob/master/docs/user-guide.md#one-node-write-mode). It assumes `$CN_V2_PRIMARY` is the CN on which you [re]provisioned manateein step 4, and `$V2_PRIMARY` is the manatee v2 primary.

    - select a node to reprovision (we'll assume the v1 sync node, which is typical)
    - reprovision it: `sapiadm reprovision $SYNC $MANATEE_IMAGE`
  5. notmatt revised this gist Mar 30, 2015. 1 changed file with 11 additions and 7 deletions.
    18 changes: 11 additions & 7 deletions manatee_brain_transplant.md
    View file
    Original file line number Diff line number Diff line change
    @@ -154,13 +154,17 @@ When the zfs send has completed, you can `svcadm enable manatee-sitter` in the n

    ## step 8. move manatee v2 out of ONWM

    A slight modification of the steps outlined [here](https://github.com/joyent/manatee/blob/master/docs/user-guide.md#one-node-write-mode):

    - reprovision the v1 sync manatee `sapiadm reprovision $SYNC $MANATEE_IMAGE`
    - take ONE_NODE_WRITE_MODE out of sapi: `sapiadm update $MANATEE_V2_PRIMARY metadata.ONE_NODE_WRITE_MODE=false`
    - Stop the v2 primary manatee sitter process: `sdc-oneachnode -n $CN_PRIMARY "svcadm -z $PRIMARY disable manatee-sitter"`
    - Run # manatee-adm unfreeze.
    - Start the v2 primary manatee sitter process: `sdc-oneachnode -n $CN_PRIMARY "svcadm -z $PRIMARY enable manatee-sitter"`
    A slight modification of the steps outlined [here](https://github.com/joyent/manatee/blob/master/docs/user-guide.md#one-node-write-mode). It assumes $CN_V2_PRIMARY is the CN on which you [re]provisioned manateein step 4, and $V2_PRIMARY is the manatee v2 primary.

    - select a node to reprovision (we'll assume the v1 sync node, which is typical)
    - reprovision it: `sapiadm reprovision $SYNC $MANATEE_IMAGE`
    - re-enable moray: log into moray zones and `svcadm enable *moray-202*`
    - remove metadata from sapi:
    - `sapiadm update $V2_PRIMARY metadata.ONE_NODE_WRITE_MODE=false`
    - `sapiadm update $V2_PRIMARY metadata.DISABLE_SITTER=false`
    - halt primary: `sdc-oneachnode $CN_V2_PRIMARY "vamdm stop $V2_PRIMARY"`
    - run manatee-adm unfreeze in the reprovisioned sync: `sdc-oneachnode $CN_SYNC "zlogin $SYNC 'manatee-adm unfreeze' < /dev/null"`
    - start v2 primary: `sdc-oneachnode $CN_V2_PRIMARY "vmadm start $V2_PRIMARY"`

    The modification takes advantage of the fact that manatee-sitter does not immediately pick up new config, so we can use SAPI to write the new config file, but it will not take effect until after manatee is stopped and restarted.

  6. notmatt revised this gist Mar 30, 2015. 1 changed file with 34 additions and 16 deletions.
    50 changes: 34 additions & 16 deletions manatee_brain_transplant.md
    View file
    Original file line number Diff line number Diff line change
    @@ -22,6 +22,8 @@ svcadm disable registrar
    svcadm disable *moray-202*
    ```

    Wait for moray to fall out of DNS. On older stacks, `sdc-healtcheck` is advised, as a number of SDC services may require restarts, as they often do not reconnect correctly when moray goes down.

    In the headnode GZ:
    ```
    sapiadm reprovision $TARGET_MORAY $MORAY_IMAGE
    @@ -72,23 +74,19 @@ zfs destroy zones/$MANATEE_UUID/data/manatee@backup
    zlogin $MANATEE_UUID "svcadm enable manatee-sitter" < /dev/null
    ```

    ## step 4. reprovision a node to manatee v2
    ## step 4. Create the first manatee v2 node

    In this step, we will set up a manatee node with the new code, but configured to *not* start manatee-sitter on setup (preventing the creation of any state). We will then destroy its delegated dataset, and set it up to zfs recv in the next step. Using the environment variables above:
    In this step, we will use one of two options to create the first manatee v2 node; it will be configured to not start manatee-sitter when the zones come up. This will allow us to destroy its delegated dataset, and set it up for zfs recv in the following steps.

    ```
    sapiadm update $MANATEE_SVC params.image_uuid=$MANATEE_IMAGE
    sapiadm update $ASYNC metadata.DISABLE_SITTER=true
    sapiadm update $ASYNC metadata.ONE_NODE_WRITE_MODE=true
    sapiadm reprovision $ASYNC $MANATEE_IMAGE
    ```
    The first option is to create a new manatee on the same CN as the existing primary; this can speed up the zfs send/recv step, as it will operate on the same filesystem.

    ### Option 1: create a new manatee

    Alternately, we can provision a new manatee node on the same CN as the primary, to speed up the zfs send/recv section of the upgrade. In that case, instead of the above reprovision steps, perform the following:
    ```
    sapiadm update $MANATEE_SVC params.image_uuid=$MANATEE_IMAGE
    cat > manatee3.json <<EOF
    {
    "service_uuid": "$MANATEE_SVC",
    "service_uuid": "$MANATEE_SVC",
    "params": {
    "alias": "manatee3",
    "server_uuid: "$CN_PRIMARY"
    @@ -102,7 +100,20 @@ EOF
    sdc-sapi /instances -X POST [email protected]
    ```

    Wait for the reprovision to complete, and log into `$ASYNC`, then:
    ### Option 2: reprovision the `$ASYNC`

    This reprovisions the `$ASYNC` manatee to the new manatee image. zfs send/recv will take place over the admin network, which can take some time for large installations (e.g., JPC can take 30+ minutes), but has the advantage of less cleanup, and no changes to manatee zone UUIDs, IPs, etc. (This should not matter for the SDC stack, but may be inappropriately used by operators)

    ```
    sapiadm update $MANATEE_SVC params.image_uuid=$MANATEE_IMAGE
    sapiadm update $ASYNC metadata.DISABLE_SITTER=true
    sapiadm update $ASYNC metadata.ONE_NODE_WRITE_MODE=true
    sapiadm reprovision $ASYNC $MANATEE_IMAGE
    ```

    ## step 5. prepping the new manatee node

    Wait for the [re]provision from step 4 to complete, log into the new node, then:
    ```
    # ensure that manatee-sitter is indeed disabled
    svcs manatee-sitter
    @@ -116,9 +127,16 @@ zfs destroy -r zones/$ZONE_UUID/data/manatee
    nc -l 1337 | zfs recv zones/$ZONE_UUID/data/manatee
    ```

    ## step 5. migrate
    At this point, we can safely disable the moray services, which prevents any writes to the system and somwhat simplifies rollback. Log into each moray and `svcadm disable *moray-202*`.


    ## step 6. migrate

    On the old primary (now in ONWM), we will disable manatee, take a snapshot, and send it to the v2 manatee node. Depending on your choice in step 4, there are two ways this is possible. If you created a new manatee on the same CN as the v1 primary, you can use option 1. If you reprovisioned a v1 node, you can use option 2 (option 2 also works on new manatee nodes, but is slower than it could be).

    ### Option 1: donor/recipient on the same CN

    On the old primary (now in ONWM), we will disable manatee, take a snapshot, and send it to the v2 manatee node.
    ### Option 2: donor/recipient on different CNs

    ```
    ZONE_UUID=$(sysinfo | json UUID)
    @@ -128,13 +146,13 @@ zfs snapshot zones/$ZONE_UUID/data/manatee@migrate
    zfs send -v zones/$ZONE_UUID/data/manatee@migrate | nc $IP 1337
    ```

    ## step 6 - enable the new manatee!
    ## step 7 - enable the new manatee!

    NB: rolling back after this point is slightly more complicated, see below.

    When the zfs send has completed, you can `svcadm enable manatee-sitter` in the new zone. `manatee-stat` in the new zone will indicate when the primary comes online, and the stack can be checked via the usual `sdc-healthcheck`, provisioning tests, and so on.
    When the zfs send has completed, you can `svcadm enable manatee-sitter` in the new zone. `manatee-stat` in the new zone will indicate when the v2 primary comes online, and the stack can be checked via the usual `sdc-healthcheck`, provisioning tests, and so on.

    ## step 6. move manatee v2 out of ONWM
    ## step 8. move manatee v2 out of ONWM

    A slight modification of the steps outlined [here](https://github.com/joyent/manatee/blob/master/docs/user-guide.md#one-node-write-mode):

  7. notmatt revised this gist Feb 12, 2015. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion manatee_brain_transplant.md
    View file
    Original file line number Diff line number Diff line change
    @@ -94,7 +94,8 @@ cat > manatee3.json <<EOF
    "server_uuid: "$CN_PRIMARY"
    },
    "metadata": {
    "DISABLE_SITTER": true
    "DISABLE_SITTER": true,
    "ONE_NODE_WRITE_MODE": true
    }
    }
    EOF
  8. notmatt revised this gist Feb 12, 2015. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion manatee_brain_transplant.md
    View file
    Original file line number Diff line number Diff line change
    @@ -88,7 +88,8 @@ Alternately, we can provision a new manatee node on the same CN as the primary,
    sapiadm update $MANATEE_SVC params.image_uuid=$MANATEE_IMAGE
    cat > manatee3.json <<EOF
    {
    "service_uuid": "$MANATEE_SVC", "params": {
    "service_uuid": "$MANATEE_SVC",
    "params": {
    "alias": "manatee3",
    "server_uuid: "$CN_PRIMARY"
    },
  9. notmatt revised this gist Feb 12, 2015. 1 changed file with 17 additions and 0 deletions.
    17 changes: 17 additions & 0 deletions manatee_brain_transplant.md
    View file
    Original file line number Diff line number Diff line change
    @@ -83,6 +83,23 @@ sapiadm update $ASYNC metadata.ONE_NODE_WRITE_MODE=true
    sapiadm reprovision $ASYNC $MANATEE_IMAGE
    ```

    Alternately, we can provision a new manatee node on the same CN as the primary, to speed up the zfs send/recv section of the upgrade. In that case, instead of the above reprovision steps, perform the following:
    ```
    sapiadm update $MANATEE_SVC params.image_uuid=$MANATEE_IMAGE
    cat > manatee3.json <<EOF
    {
    "service_uuid": "$MANATEE_SVC", "params": {
    "alias": "manatee3",
    "server_uuid: "$CN_PRIMARY"
    },
    "metadata": {
    "DISABLE_SITTER": true
    }
    }
    EOF
    sdc-sapi /instances -X POST [email protected]
    ```

    Wait for the reprovision to complete, and log into `$ASYNC`, then:
    ```
    # ensure that manatee-sitter is indeed disabled
  10. notmatt revised this gist Feb 9, 2015. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions manatee_brain_transplant.md
    View file
    Original file line number Diff line number Diff line change
    @@ -77,10 +77,10 @@ zlogin $MANATEE_UUID "svcadm enable manatee-sitter" < /dev/null
    In this step, we will set up a manatee node with the new code, but configured to *not* start manatee-sitter on setup (preventing the creation of any state). We will then destroy its delegated dataset, and set it up to zfs recv in the next step. Using the environment variables above:

    ```
    sapiadm update $MANATEE_SVC $MANATEE_IMAGE
    sapiadm update $MANATEE_SVC params.image_uuid=$MANATEE_IMAGE
    sapiadm update $ASYNC metadata.DISABLE_SITTER=true
    sapiadm update $ASYNC metadata.ONE_NODE_WRITE_MODE=true
    sapiadm reprovision $ASYNC
    sapiadm reprovision $ASYNC $MANATEE_IMAGE
    ```

    Wait for the reprovision to complete, and log into `$ASYNC`, then:
  11. Matt Smillie revised this gist Feb 6, 2015. 1 changed file with 32 additions and 17 deletions.
    49 changes: 32 additions & 17 deletions manatee_brain_transplant.md
    View file
    Original file line number Diff line number Diff line change
    @@ -29,12 +29,30 @@ sapiadm reprovision $TARGET_MORAY $MORAY_IMAGE

    ## step 2. reduce existing manatee to ONWM

    It helps to be logged on to the CNs of each manatee node.
    It can help to be logged on to the CNs of each manatee node, but most operations will be done in the HN/GZ via `sdc-oneachnode`. First we'll se up some environment variables:

    1. make note of the current manatee status via `manatee-stat` in one of the zones
    2. in the async node `svcadm disable manatee-sitter`
    3. in the sync node `svcadm disable manatee-sitter`
    4. on the primary node:
    ```
    SDC=$(sdc-sapi /applications?name=sdc | json -Ha uuid)
    MANATEE_SVC=$(sdc-sapi "/services?name=manatee&application_uuid=$SDC" | json -Ha uuid)
    # NB: sdc-manatee-stat isn't present in the GZ on all installs;
    # manatee-stat inside any manatee zone will be. Adjust if required.
    MANATEE_STAT=$(sdc-manatee-stat | json | tee initial_manatee_stat.json)
    PRIMARY=$(echo $MANATEE_STAT | json sdc.primary.zoneId)
    CN_PRIMARY=$(sdc-vmapi /vms/$PRIMARY | json -H server_uuid)
    SYNC=$(echo $MANATEE_STAT | json sdc.sync.zoneId)
    CN_SYNC=$(sdc-vmapi /vms/$SYNC | json -H server_uuid)
    ASYNC=$(echo $MANATEE_STAT | json sdc.async.zoneId)
    CN_ASYNC=$(sdc-vmapi /vms/$ASYNC | json -H server_uuid)
    ```

    Now we will proceed to disable the async and sync nodes:
    ```
    sdc-oneachnode -n $CN_ASYNC "svcadm -z $ASYNC disable manatee-sitter"
    sdc-oneachnode -n $CN_SYNC "svcadm -z $SYNC disable manatee-sitter"
    ```

    Set ONWM on the primary node:
    ```
    svcadm disable config-agent
    vim /opt/smartdc/manatee/etc/sitter.json
    @@ -56,18 +74,16 @@ zlogin $MANATEE_UUID "svcadm enable manatee-sitter" < /dev/null

    ## step 4. reprovision a node to manatee v2

    In this step, we will set up a manatee node with the new code, but configured to *not* start manatee-sitter on setup (preventing the creation of any state). We will then destroy its delegated dataset, and set it up to zfs recv in the next step.
    In this step, we will set up a manatee node with the new code, but configured to *not* start manatee-sitter on setup (preventing the creation of any state). We will then destroy its delegated dataset, and set it up to zfs recv in the next step. Using the environment variables above:

    Where `$MANATEE_ASYNC` is the UUID of the former async node, in the headnode GZ:
    ```
    MANATEE_SVC=$(sdc-sapi /services?name=manatee | json -Ha uuid)
    sapiadm update $MANATEE_SVC $MANATEE_IMAGE
    sapiadm update $MANATEE_ASYNC metadata.DISABLE_SITTER=true
    sapiadm update $MANATEE_ASYNC metadata.ONE_NODE_WRITE_MODE=true
    sapiadm reprovision $MANATEE_ASYNC
    sapiadm update $ASYNC metadata.DISABLE_SITTER=true
    sapiadm update $ASYNC metadata.ONE_NODE_WRITE_MODE=true
    sapiadm reprovision $ASYNC
    ```

    Wait for the reprovision to complete, and log into `$MANATEE_ASYNC`, then:
    Wait for the reprovision to complete, and log into `$ASYNC`, then:
    ```
    # ensure that manatee-sitter is indeed disabled
    svcs manatee-sitter
    @@ -103,12 +119,11 @@ When the zfs send has completed, you can `svcadm enable manatee-sitter` in the n

    A slight modification of the steps outlined [here](https://github.com/joyent/manatee/blob/master/docs/user-guide.md#one-node-write-mode):

    - reprovision the v1 sync manatee
    - take ONE_NODE_WRITE_MODE out of sapi:
    - `sapiadm update $MANATEE_V2_PRIMARY metadata.ONE_NODE_WRITE_MODE=false
    - Stop the v2 primary manatee sitter process.
    - reprovision the v1 sync manatee `sapiadm reprovision $SYNC $MANATEE_IMAGE`
    - take ONE_NODE_WRITE_MODE out of sapi: `sapiadm update $MANATEE_V2_PRIMARY metadata.ONE_NODE_WRITE_MODE=false`
    - Stop the v2 primary manatee sitter process: `sdc-oneachnode -n $CN_PRIMARY "svcadm -z $PRIMARY disable manatee-sitter"`
    - Run # manatee-adm unfreeze.
    - Start the v2 primary manatee sitter process.
    - Start the v2 primary manatee sitter process: `sdc-oneachnode -n $CN_PRIMARY "svcadm -z $PRIMARY enable manatee-sitter"`

    The modification takes advantage of the fact that manatee-sitter does not immediately pick up new config, so we can use SAPI to write the new config file, but it will not take effect until after manatee is stopped and restarted.

  12. notmatt revised this gist Feb 6, 2015. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion manatee_brain_transplant.md
    View file
    Original file line number Diff line number Diff line change
    @@ -90,7 +90,7 @@ ZONE_UUID=$(sysinfo | json UUID)
    IP=XXX # from above.
    svcadm disable manatee-sitter
    zfs snapshot zones/$ZONE_UUID/data/manatee@migrate
    zfs send -v zones/$ZONE_UUID/data/manatee@migrate | cn $IP 1337
    zfs send -v zones/$ZONE_UUID/data/manatee@migrate | nc $IP 1337
    ```

    ## step 6 - enable the new manatee!
  13. Matt Smillie created this gist Feb 5, 2015.
    129 changes: 129 additions & 0 deletions manatee_brain_transplant.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,129 @@
    # Manatee Brain Transplant

    The goal of this process is to upgrade a manatee *of any vintage* to Manatee v2. It relies on ZFS send/recv to replicate the data, but is limited to a migration between nodes in ONWM.

    ## step 1. upgrade moray

    Upgrading moray to a forward/backward compatible version is a prerequisite of the upgrade. The usual process is to disable one moray node, double-check the stack reconnects correctly, reprovision that node, and then repeat for other moray nodes.

    If there is only one moray node deployed, deploying a second using the new image allows you to upgrade the original node as above.


    Pre work - update the moray image in SAPI:
    ```
    SDC=$(sdc-sapi /applications?name=sdc | json -Ha uuid)
    MORAY_SVC=$(sdc-sapi "/services?name=moray&application_uuid=$SDC" | json -Ha uuid)
    sapiadm update $MORAY_SVC params.image_uuid=$MORAY_IMAGE
    ```

    In the target moray zone:
    ```
    svcadm disable registrar
    svcadm disable *moray-202*
    ```

    In the headnode GZ:
    ```
    sapiadm reprovision $TARGET_MORAY $MORAY_IMAGE
    ```

    ## step 2. reduce existing manatee to ONWM

    It helps to be logged on to the CNs of each manatee node.

    1. make note of the current manatee status via `manatee-stat` in one of the zones
    2. in the async node `svcadm disable manatee-sitter`
    3. in the sync node `svcadm disable manatee-sitter`
    4. on the primary node:
    ```
    svcadm disable config-agent
    vim /opt/smartdc/manatee/etc/sitter.json
    # search for oneNodeWriteMode and set it to true.
    svcadm restart manatee-sitter
    ```

    ## step 3. backup manatee.

    In the GZ of the CN of the primary manatee:
    ```
    MANATEE_UUID=$(vmadm lookup alias=~manatee)
    zlogin $MANATEE_UUID "svcadm disable manatee-sitter" < /dev/null
    zfs snapshot -r zones/$MANATEE_UUID/data/manatee@backup
    zfs send zones/$MANATEE_UUID/data/manatee@backup >./manatee-backup.zfs
    zfs destroy zones/$MANATEE_UUID/data/manatee@backup
    zlogin $MANATEE_UUID "svcadm enable manatee-sitter" < /dev/null
    ```

    ## step 4. reprovision a node to manatee v2

    In this step, we will set up a manatee node with the new code, but configured to *not* start manatee-sitter on setup (preventing the creation of any state). We will then destroy its delegated dataset, and set it up to zfs recv in the next step.

    Where `$MANATEE_ASYNC` is the UUID of the former async node, in the headnode GZ:
    ```
    MANATEE_SVC=$(sdc-sapi /services?name=manatee | json -Ha uuid)
    sapiadm update $MANATEE_SVC $MANATEE_IMAGE
    sapiadm update $MANATEE_ASYNC metadata.DISABLE_SITTER=true
    sapiadm update $MANATEE_ASYNC metadata.ONE_NODE_WRITE_MODE=true
    sapiadm reprovision $MANATEE_ASYNC
    ```

    Wait for the reprovision to complete, and log into `$MANATEE_ASYNC`, then:
    ```
    # ensure that manatee-sitter is indeed disabled
    svcs manatee-sitter
    # take note of IP address
    ifconfig | grep inet
    # destroy dataset
    ZONE_UUID=$(sysinfo | json UUID)
    zfs destroy -r zones/$ZONE_UUID/data/manatee
    nc -l 1337 | zfs recv zones/$ZONE_UUID/data/manatee
    ```

    ## step 5. migrate

    On the old primary (now in ONWM), we will disable manatee, take a snapshot, and send it to the v2 manatee node.

    ```
    ZONE_UUID=$(sysinfo | json UUID)
    IP=XXX # from above.
    svcadm disable manatee-sitter
    zfs snapshot zones/$ZONE_UUID/data/manatee@migrate
    zfs send -v zones/$ZONE_UUID/data/manatee@migrate | cn $IP 1337
    ```

    ## step 6 - enable the new manatee!

    NB: rolling back after this point is slightly more complicated, see below.

    When the zfs send has completed, you can `svcadm enable manatee-sitter` in the new zone. `manatee-stat` in the new zone will indicate when the primary comes online, and the stack can be checked via the usual `sdc-healthcheck`, provisioning tests, and so on.

    ## step 6. move manatee v2 out of ONWM

    A slight modification of the steps outlined [here](https://github.com/joyent/manatee/blob/master/docs/user-guide.md#one-node-write-mode):

    - reprovision the v1 sync manatee
    - take ONE_NODE_WRITE_MODE out of sapi:
    - `sapiadm update $MANATEE_V2_PRIMARY metadata.ONE_NODE_WRITE_MODE=false
    - Stop the v2 primary manatee sitter process.
    - Run # manatee-adm unfreeze.
    - Start the v2 primary manatee sitter process.

    The modification takes advantage of the fact that manatee-sitter does not immediately pick up new config, so we can use SAPI to write the new config file, but it will not take effect until after manatee is stopped and restarted.

    We should now have a 2-machine manatee v2 cluster, and can check the stack in the usual way. If that's successful, the final step is to reprovision the old v1 manatee; no other special steps are required for that.


    # Rolling back

    At any time before enabling the v2 manatee (step 6), rollback is straight-forward:
    - `svcadm enable config-agent` on the v1 primary
    - wait & double-check that `oneNodeWriteMode: false` is in `/opt/smartdc/manatee/sitter.json`
    - `svcadm restart manatee-sitter` in the v1 primary, wait for `manatee-stat` to stabilize
    - `svcadm enable manatee-sitter` in the sync
    - `sapiadm reprovision $MANATEE_ASYNC $OLD_MANATEE_IMAGE` to revive the async (it may also require a rebuild)

    After enabling the v2 manatee, rollback requires a little more consideration, as we may have accepted writes that we are not prepared to discard. In this case, the "reverse brain transplant" could be attempted; sending a snapshot of the v2 manatee back to the v1 primary (or perhaps the v1 sync).

    If all else fails, we can restore the v1 manatee from its pre-migration backup.