Skip to content

Instantly share code, notes, and snippets.

@scottsb

scottsb/resetting-csync2-cluster.md

Last active October 25, 2021 20:28
Show Gist options
  • Save scottsb/cc4a10b57314f63febe2 to your computer and use it in GitHub Desktop.
Save scottsb/cc4a10b57314f63febe2 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. scottsb revised this gist Feb 17, 2016. 1 changed file with 4 additions and 4 deletions.
    8 changes: 4 additions & 4 deletions resetting-csync2-cluster.md
    View file
    Original file line number Diff line number Diff line change
    @@ -30,7 +30,7 @@ with a named config, each `csync2` command should include `-C [config-name]`.

    ### Phase 1

    1. Forcibly disable all the automatic syncing (on all servers):
    1. On all servers, forcibly disable all the automatic syncing:

    service lsyncd stop
    killall csync2
    @@ -45,7 +45,7 @@ as needing to be synced out to other servers:

    ### Phase 2

    Run all commands in this phase from the authoritative server.
    Run all commands in this phase from the authoritative server selected in the last phase.

    1. Find all differences between authoritative server and remotes and mark for sync:

    @@ -61,10 +61,10 @@ Run all commands in this phase from the authoritative server.

    ### Phase 3

    1. Confirm that all files are in sync (if successful will list no files):
    1. On all servers, confirm that all files are in sync (if successful will list no files):

    csync2 -T

    2. Restart automatic sync service:
    2. On all servers, restart automatic sync service:

    service lsyncd start
  2. scottsb revised this gist Sep 11, 2015. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion resetting-csync2-cluster.md
    View file
    Original file line number Diff line number Diff line change
    @@ -24,7 +24,7 @@ References to lsyncd assume you're triggering csync2 via lsyncd
    be safely ignored if that doesn't apply to you.

    As written these commands do not specify a csync2 config file. If you are using csync2
    with a named config, each `csync` command should include `-C [config-name]`.
    with a named config, each `csync2` command should include `-C [config-name]`.

    ## Instructions

  3. scottsb revised this gist Sep 11, 2015. 1 changed file with 20 additions and 3 deletions.
    23 changes: 20 additions & 3 deletions resetting-csync2-cluster.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,17 +1,34 @@
    # Guide to Resetting a csync2 Cluster

    ## Introduction

    These are possible steps to reset a csync2 cluster that has been seriously fubared.
    This is an apocalyptic approach and should only be used when more surgical fixes (like
    correcting an individual conflict) aren't workable.

    ### Use Cases

    This will solve errors like:

    ERROR from peer 10.0.0.1: File is also marked dirty here!

    And, depending on the cause, it may fix errors like:

    Database backend is exceedingly busy => Terminating (requesting retry).
    ERROR from peer 10.0.0.1: Connection closed.

    ### Important Notes

    References to lsyncd assume you're triggering csync2 via lsyncd
    (as with [puppet-clustersync](https://github.com/scottsb/puppet-clustersync)) but can
    be safely ignored if that doesn't apply to you.

    As written these commands do not specify a csync2 config file. If you are using csync2
    with a named config, each `csync` command should include `-C [config-name]`.

    ## Phase 1
    ## Instructions

    ### Phase 1

    1. Forcibly disable all the automatic syncing (on all servers):

    @@ -26,7 +43,7 @@ as needing to be synced out to other servers:

    csync2 -cIr /

    ## Phase 2
    ### Phase 2

    Run all commands in this phase from the authoritative server.

    @@ -42,7 +59,7 @@ Run all commands in this phase from the authoritative server.

    csync2 -xr /

    ## Phase 3
    ### Phase 3

    1. Confirm that all files are in sync (if successful will list no files):

  4. scottsb revised this gist Sep 11, 2015. 1 changed file with 8 additions and 8 deletions.
    16 changes: 8 additions & 8 deletions resetting-csync2-cluster.md
    View file
    Original file line number Diff line number Diff line change
    @@ -9,38 +9,38 @@ References to lsyncd assume you're triggering csync2 via lsyncd
    be safely ignored if that doesn't apply to you.

    As written these commands do not specify a csync2 config file. If you are using csync2
    with a named config, each `csync` command should includ `-C [config-name]`.
    with a named config, each `csync` command should include `-C [config-name]`.

    ## Phase 1

    1. Forcibly disable all the automatic syncing (on all servers):

    service lsyncd stop
    killall csync2
    service lsyncd stop
    killall csync2

    2. Choose one server that's closest to the correct state and manually adjust files as
    needed to make it the authoritative master.

    3. On all servers, update csync2 database to match filesystem but without marking anything
    as needing to be synced out to other servers:

    csync2 -cIr /
    csync2 -cIr /

    ## Phase 2

    Run all commands in this phase from the authoritative server.

    1. Find all differences between authoritative server and remotes and mark for sync:

    csync2 -TUXI
    csync2 -TUXI

    2. Reset database to force current server to be winner on any conflicts:

    csync2 -fr /
    csync2 -fr /

    3. Run a sync to all other servers:

    csync2 -xr /
    csync2 -xr /

    ## Phase 3

    @@ -50,4 +50,4 @@ Run all commands in this phase from the authoritative server.

    2. Restart automatic sync service:

    service lsyncd start
    service lsyncd start
  5. scottsb revised this gist Sep 11, 2015. 1 changed file with 4 additions and 3 deletions.
    7 changes: 4 additions & 3 deletions resetting-csync2-cluster.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,7 +1,8 @@
    # Resetting a csync2 Cluster
    # Guide to Resetting a csync2 Cluster

    Steps to reset a csync2 cluster that has been seriously fubared. This is an apocalyptic
    approach and should only be used when more surgical changes aren't workable.
    These are possible steps to reset a csync2 cluster that has been seriously fubared.
    This is an apocalyptic approach and should only be used when more surgical fixes (like
    correcting an individual conflict) aren't workable.

    References to lsyncd assume you're triggering csync2 via lsyncd
    (as with [puppet-clustersync](https://github.com/scottsb/puppet-clustersync)) but can
  6. scottsb revised this gist Sep 11, 2015. 1 changed file with 14 additions and 10 deletions.
    24 changes: 14 additions & 10 deletions resetting-csync2-cluster.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,48 +1,52 @@
    # Resetting a csync2 Cluster

    Steps to reset a csync2 cluster that has been seriously fubared. This is an apocalyptic
    approach and should only be used when more surgical changes aren't workable. References
    to lsyncd assume you're triggering csync2 via lsyncd
    approach and should only be used when more surgical changes aren't workable.

    References to lsyncd assume you're triggering csync2 via lsyncd
    (as with [puppet-clustersync](https://github.com/scottsb/puppet-clustersync)) but can
    be safely ignored if that doesn't apply to you.

    As written these commands do not specify a csync2 config file. If you are using csync2
    with a named config, each `csync` command should includ `-C [config-name]`.

    ## Phase 1

    1. Forcibly disable all the automatic syncing (on all servers):

    service lsyncd stop
    killall csync2
    service lsyncd stop
    killall csync2

    2. Choose one server that's closest to the correct state and manually adjust files as
    needed to make it the authoritative master.

    3. On all servers, update csync2 database to match filesystem but without marking anything
    as needing to be synced out to other servers:

    csync2 -C uploads -cIr /
    csync2 -cIr /

    ## Phase 2

    Run all commands in this phase from the authoritative server.

    1. Find all differences between authoritative server and remotes and mark for sync:

    csync2 -C uploads -TUXI
    csync2 -TUXI

    2. Reset database to force current server to be winner on any conflicts:

    csync2 -C uploads -fr /
    csync2 -fr /

    3. Run a sync to all other servers:

    csync2 -C uploads -xr /
    csync2 -xr /

    ## Phase 3

    1. Confirm that all files are in sync (if successful will list no files):

    csync2 -C uploads -T
    csync2 -T

    2. Restart automatic sync service:

    service lsyncd start
    service lsyncd start
  7. scottsb revised this gist Sep 11, 2015. 1 changed file with 7 additions and 7 deletions.
    14 changes: 7 additions & 7 deletions resetting-csync2-cluster.md
    View file
    Original file line number Diff line number Diff line change
    @@ -10,32 +10,32 @@ be safely ignored if that doesn't apply to you.

    1. Forcibly disable all the automatic syncing (on all servers):

    service lsyncd stop
    killall csync2
    service lsyncd stop
    killall csync2

    2. Choose one server that's closest to the correct state and manually adjust files as
    needed to make it the authoritative master.

    3. On all servers, update csync2 database to match filesystem but without marking anything
    as needing to be synced out to other servers:

    csync2 -C uploads -cIr /
    csync2 -C uploads -cIr /

    ## Phase 2

    Run all commands in this phase from the authoritative server.

    1. Find all differences between authoritative server and remotes and mark for sync:

    csync2 -C uploads -TUXI
    csync2 -C uploads -TUXI

    2. Reset database to force current server to be winner on any conflicts:

    csync2 -C uploads -fr /
    csync2 -C uploads -fr /

    3. Run a sync to all other servers:

    csync2 -C uploads -xr /
    csync2 -C uploads -xr /

    ## Phase 3

    @@ -45,4 +45,4 @@ Run all commands in this phase from the authoritative server.

    2. Restart automatic sync service:

    service lsyncd start
    service lsyncd start
  8. scottsb renamed this gist Sep 11, 2015. 1 changed file with 0 additions and 0 deletions.
    0 gistfile1.txt → resetting-csync2-cluster.md
    View file
    File renamed without changes.
  9. scottsb created this gist Sep 11, 2015.
    48 changes: 48 additions & 0 deletions gistfile1.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,48 @@
    # Resetting a csync2 Cluster

    Steps to reset a csync2 cluster that has been seriously fubared. This is an apocalyptic
    approach and should only be used when more surgical changes aren't workable. References
    to lsyncd assume you're triggering csync2 via lsyncd
    (as with [puppet-clustersync](https://github.com/scottsb/puppet-clustersync)) but can
    be safely ignored if that doesn't apply to you.

    ## Phase 1

    1. Forcibly disable all the automatic syncing (on all servers):

    service lsyncd stop
    killall csync2

    2. Choose one server that's closest to the correct state and manually adjust files as
    needed to make it the authoritative master.

    3. On all servers, update csync2 database to match filesystem but without marking anything
    as needing to be synced out to other servers:

    csync2 -C uploads -cIr /

    ## Phase 2

    Run all commands in this phase from the authoritative server.

    1. Find all differences between authoritative server and remotes and mark for sync:

    csync2 -C uploads -TUXI

    2. Reset database to force current server to be winner on any conflicts:

    csync2 -C uploads -fr /

    3. Run a sync to all other servers:

    csync2 -C uploads -xr /

    ## Phase 3

    1. Confirm that all files are in sync (if successful will list no files):

    csync2 -C uploads -T

    2. Restart automatic sync service:

    service lsyncd start