Skip to content

Instantly share code, notes, and snippets.

@Mbitzg

Mbitzg/gist:cabb6ca6f82ca485a391

Forked from drkarl/gist:739a864b3275e901d317
Last active August 29, 2015 14:20
Show Gist options
  • Save Mbitzg/cabb6ca6f82ca485a391 to your computer and use it in GitHub Desktop.
Save Mbitzg/cabb6ca6f82ca485a391 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. @drkarl drkarl revised this gist Mar 20, 2015. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions gistfile1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -24,6 +24,7 @@ Also I would like that you, the HN community, would add more deal breakers for t

    ## [Snebu:](http://www.snebu.com/)
    - Doesn't do encryption
    - File level, not block level deduplication

    ## [Obnam:](http://obnam.org/)
    - Really slow for large backups (from a benchmark between obnam and attic)
  2. @drkarl drkarl revised this gist Mar 16, 2015. 1 changed file with 17 additions and 1 deletion.
    18 changes: 17 additions & 1 deletion gistfile1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -76,7 +76,13 @@ Also Tarsnap scores really high on encryption and deduplication but it has 3 imp
    Attic has some really good comments on HN and good blog posts, doesn't have any particular deal-breaker (for now, if you have one please share with us), so for now is the most promising.

    ## Roll your own
    Some HN users have posted the simple script they use
    Some HN users have posted the simple script they use.
    The scripts usually use a combination of
    ## [rsync](https://rsync.samba.org/)

    ## [LUKS](https://code.google.com/p/cryptsetup/)

    ## [rdiff-backup](http://www.nongnu.org/rdiff-backup/)

    **mikhailian**'s script
    ```
    @@ -160,6 +166,16 @@ zx2c4@thinkpad ~ $ cat Projects/remote-backup.sh
    trace kill %1
    ```

    **pwenzel** suggests
    ```
    rm -rf backup.3
    mv backup.2 backup.3
    mv backup.1 backup.2
    cp -al backup.0 backup.1
    rsync -a --delete source_directory/ backup.0/
    ```
    and https://gist.github.com/ei-grad/7610406

    # Meta-backup solutions (which use several backup solutions)


  3. @drkarl drkarl revised this gist Mar 16, 2015. 1 changed file with 3 additions and 3 deletions.
    6 changes: 3 additions & 3 deletions gistfile1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -2,7 +2,7 @@

    I've been looking for the best Linux backup system, and also reading lots of HN comments.

    Instead of putting pros and cons of every backup system I'll just list some deal-breakers which would disqualify them.
    Instead of putting pros and cons of every backup system I'll just list some **deal-breakers** which would disqualify them.

    Also I would like that you, the HN community, would add more deal breakers for these or other backup systems if you know some more and at the same time, if you have data to disprove some of the deal-breakers listed here (benchmarks, info about something being true for older releases but is fixed on newer releases), please share it so that I can edit this list accordingly.

    @@ -78,7 +78,7 @@ Attic has some really good comments on HN and good blog posts, doesn't have any
    ## Roll your own
    Some HN users have posted the simple script they use

    *mikhailian*'s script
    **mikhailian**'s script
    ```
    FROM=/etc
    TO=/var/backups
    @@ -100,7 +100,7 @@ done
    rsync $OPTS $LINKTO $FROM/ $TO/`/usr/bin/basename $FROM.0`
    ```

    *zx2c4*'s script
    **zx2c4**'s script
    ```
    zx2c4@thinkpad ~ $ cat Projects/remote-backup.sh
    #!/bin/sh
  4. @drkarl drkarl revised this gist Mar 16, 2015. 1 changed file with 6 additions and 0 deletions.
    6 changes: 6 additions & 0 deletions gistfile1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -160,3 +160,9 @@ zx2c4@thinkpad ~ $ cat Projects/remote-backup.sh
    trace kill %1
    ```

    # Meta-backup solutions (which use several backup solutions)


    ## [Backupninja](https://labs.riseup.net/code/projects/backupninja)

    ## [Deltaic](https://github.com/cmusatyalab/deltaic)
  5. @drkarl drkarl revised this gist Mar 16, 2015. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions gistfile1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -77,6 +77,7 @@ Attic has some really good comments on HN and good blog posts, doesn't have any

    ## Roll your own
    Some HN users have posted the simple script they use

    *mikhailian*'s script
    ```
    FROM=/etc
  6. @drkarl drkarl revised this gist Mar 16, 2015. 1 changed file with 86 additions and 1 deletion.
    87 changes: 86 additions & 1 deletion gistfile1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -73,4 +73,89 @@ Also Tarsnap scores really high on encryption and deduplication but it has 3 imp
    - The so called Colin-Percival-gets-hit-by-a-bus scenario

    ## [Attic](https://attic-backup.org/)
    Attic has some really good comments on HN and good blog posts, doesn't have any particular deal-breaker (for now, if you have one please share with us), so for now is the most promising.
    Attic has some really good comments on HN and good blog posts, doesn't have any particular deal-breaker (for now, if you have one please share with us), so for now is the most promising.

    ## Roll your own
    Some HN users have posted the simple script they use
    *mikhailian*'s script
    ```
    FROM=/etc
    TO=/var/backups
    LINKTO=--link-dest=$TO/`/usr/bin/basename $FROM`.1
    OPTS="-a --delete -delete-excluded"
    NUMBER_OF_BACKUPS=8
    find $TO -maxdepth 1 -type d -name "`basename $FROM`.[0-9]"| sort -rn| while read dir
    do
    this=`expr match "$dir" '.*\([0-9]\)'`;
    let next=($this+1)%$NUMBER_OF_BACKUPS;
    basedirname=${dir%.[0-9]}
    if [ $next -eq 0 ] ; then
    rm -rf $dir
    else
    mv $dir $basedirname.$next
    fi
    done
    rsync $OPTS $LINKTO $FROM/ $TO/`/usr/bin/basename $FROM.0`
    ```

    *zx2c4*'s script
    ```
    zx2c4@thinkpad ~ $ cat Projects/remote-backup.sh
    #!/bin/sh
    cd "$(readlink -f "$(dirname "$0")")"
    if [ $UID -ne 0 ]; then
    echo "You must be root."
    exit 1
    fi
    umount() {
    if ! /bin/umount "$1"; then
    sleep 5
    if ! /bin/umount "$1"; then
    sleep 10
    /bin/umount "$1"
    fi
    fi
    }
    unwind() {
    echo "[-] ERROR: unwinding and quitting."
    sleep 3
    trace sync
    trace umount /mnt/mybackupserver-backup
    trace cryptsetup luksClose mybackupserver-backup || { sleep 5; trace cryptsetup luksClose mybackupserver-backup; }
    trace iscsiadm -m node -U all
    trace kill %1
    exit 1
    }
    trace() {
    echo "[+] $@"
    "$@"
    }
    RSYNC_OPTS="-i -rlptgoXDHxv --delete-excluded --delete --progress $RSYNC_OPTS"
    trap unwind INT TERM
    trace modprobe libiscsi
    trace modprobe scsi_transport_iscsi
    trace modprobe iscsi_tcp
    iscsid -f &
    sleep 1
    trace iscsiadm -m discovery -t st -p mybackupserver.somehost.somewere -P 1 -l
    sleep 5
    trace cryptsetup --key-file /etc/dmcrypt/backup-mybackupserver-key luksOpen /dev/disk/by-uuid/10a126a2-c991-49fc-89bf-8d621a73dd36 mybackupserver-backup || unwind
    trace fsck -a /dev/mapper/mybackupserver-backup || unwind
    trace mount -v /dev/mapper/mybackupserver-backup /mnt/mybackupserver-backup || unwind
    trace rsync $RSYNC_OPTS --exclude=/usr/portage/distfiles --exclude=/home/zx2c4/.cache --exclude=/var/tmp / /mnt/mybackupserver-backup/root || unwind
    trace rsync $RSYNC_OPTS /mnt/storage/Archives/ /mnt/mybackupserver-backup/archives || unwind
    trace sync
    trace umount /mnt/mybackupserver-backup
    trace cryptsetup luksClose mybackupserver-backup
    trace iscsiadm -m node -U all
    trace kill %1
    ```

  7. @drkarl drkarl revised this gist Mar 16, 2015. 1 changed file with 16 additions and 15 deletions.
    31 changes: 16 additions & 15 deletions gistfile1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -6,26 +6,26 @@ Instead of putting pros and cons of every backup system I'll just list some deal

    Also I would like that you, the HN community, would add more deal breakers for these or other backup systems if you know some more and at the same time, if you have data to disprove some of the deal-breakers listed here (benchmarks, info about something being true for older releases but is fixed on newer releases), please share it so that I can edit this list accordingly.

    ## Amanda (comments by sammcj)
    ## [Amanda (comments by sammcj)](http://www.amanda.org/)
    - It has a lot of management overhead and that's a problem if you don't have time for a full time backup administrator.
    - It mainly comprises of using tar for backups which is pretty inflexible by modern standards.
    - The enterprise web interface is OK but it's had so many bugs it's not funny.
    - Backups are very slow.
    - Restores are slow and painful to manage.
    - I haven't found it to be great when trying to integrate with puppet / automation frameworks.

    ## Bacula (from the Why section on Burp):
    ## [Bacula (from the Why section on Burp):](http://www.baculasystems.com/)
    - Too complex to configure
    - Stores catalog separate from backups, need to backup catalog
    - Doesn't deduplicate
    - Relies on clock accuracy
    - Can't resume an interrupted backup
    - Retention policy

    ## Snebu:
    ## [Snebu:](http://www.snebu.com/)
    - Doesn't do encryption

    ## Obnam:
    ## [Obnam:](http://obnam.org/)
    - Really slow for large backups (from a benchmark between obnam and attic)
    - To improve performance:
    ```
    @@ -34,42 +34,43 @@ upload-queue-size=512
    ```
    as per: http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2014-June/003086.html

    ## Burp:
    ## [Burp:](http://burp.grke.org/)
    - Client side encryption turns off delta differencing

    ## Bup:
    ## [Bup:](https://bup.github.io/)
    - Can't purge old backups
    - Doesn't encrypt backups (well, there is encbup)

    ## Tarsnap:
    ## [Tarsnap:](https://www.tarsnap.com/)
    - Slow restore performance on large backups? (Sorry Colin aka cperciva)
    - This was a really strong candidate until I read some comments on HN about the slow performance to restore large backups.
    - If this has changed in a recent version or someone has benchmarks to prove or disprove it, it would be really valuable.

    ## Duplicity:
    ## [Duplicity:](http://duplicity.nongnu.org/)
    - Slow restore performance on large backups?
    - This was also a really strong candidate until I read some comments on HN about the slow performance to restore large backups.
    - If this has changed in a recent version or someone has benchmarks to prove or disprove it, it would be really valuable.

    ## BackupPC:
    ## [BackupPC:](http://backuppc.sourceforge.net/)
    - It doesn't do encrypted backups

    ## backup2l:
    ## [backup2l:](http://backup2l.sourceforge.net/)
    - No support for encryption

    ## Arq:
    ## [Arq:](http://www.haystacksoftware.com/arq/)
    - Just included here because I knew someone would mention it in the comments.
    It's Mac OS X only. This list is for Linux server backup systems.

    ## Other contenders (of which I don't have references or information):
    - HashBackup
    - ZBackup
    - ZPaq
    - BTRFS send/receive
    - [HashBackup](http://www.hashbackup.com/)
    - [ZBackup](http://zbackup.org/)
    - [ZPaq](http://mattmahoney.net/dc/zpaq.html)
    - [BTRFS send/receive](https://btrfs.wiki.kernel.org/index.php/Incremental_Backup)

    Also Tarsnap scores really high on encryption and deduplication but it has 3 important cons:
    - Not having control of the server where your backups are stored
    - Bandwith costs make your costs unpredictable
    - The so called Colin-Percival-gets-hit-by-a-bus scenario

    ## [Attic](https://attic-backup.org/)
    Attic has some really good comments on HN and good blog posts, doesn't have any particular deal-breaker (for now, if you have one please share with us), so for now is the most promising.
  8. @drkarl drkarl revised this gist Mar 16, 2015. 1 changed file with 7 additions and 0 deletions.
    7 changes: 7 additions & 0 deletions gistfile1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -6,6 +6,13 @@ Instead of putting pros and cons of every backup system I'll just list some deal

    Also I would like that you, the HN community, would add more deal breakers for these or other backup systems if you know some more and at the same time, if you have data to disprove some of the deal-breakers listed here (benchmarks, info about something being true for older releases but is fixed on newer releases), please share it so that I can edit this list accordingly.

    ## Amanda (comments by sammcj)
    - It has a lot of management overhead and that's a problem if you don't have time for a full time backup administrator.
    - It mainly comprises of using tar for backups which is pretty inflexible by modern standards.
    - The enterprise web interface is OK but it's had so many bugs it's not funny.
    - Backups are very slow.
    - Restores are slow and painful to manage.
    - I haven't found it to be great when trying to integrate with puppet / automation frameworks.

    ## Bacula (from the Why section on Burp):
    - Too complex to configure
  9. @drkarl drkarl revised this gist Mar 16, 2015. 1 changed file with 4 additions and 2 deletions.
    6 changes: 4 additions & 2 deletions gistfile1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -21,8 +21,10 @@ Also I would like that you, the HN community, would add more deal breakers for t
    ## Obnam:
    - Really slow for large backups (from a benchmark between obnam and attic)
    - To improve performance:
    ```lru-size=1024
    upload-queue-size=512```
    ```
    lru-size=1024
    upload-queue-size=512
    ```
    as per: http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2014-June/003086.html

    ## Burp:
  10. @drkarl drkarl revised this gist Mar 16, 2015. 1 changed file with 3 additions and 0 deletions.
    3 changes: 3 additions & 0 deletions gistfile1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,9 +1,12 @@
    # Linux Backup Solutions

    I've been looking for the best Linux backup system, and also reading lots of HN comments.

    Instead of putting pros and cons of every backup system I'll just list some deal-breakers which would disqualify them.

    Also I would like that you, the HN community, would add more deal breakers for these or other backup systems if you know some more and at the same time, if you have data to disprove some of the deal-breakers listed here (benchmarks, info about something being true for older releases but is fixed on newer releases), please share it so that I can edit this list accordingly.


    ## Bacula (from the Why section on Burp):
    - Too complex to configure
    - Stores catalog separate from backups, need to backup catalog
  11. @drkarl drkarl renamed this gist Mar 16, 2015. 1 changed file with 0 additions and 0 deletions.
    0 Linux-backup.txt → gistfile1.md
    View file
    File renamed without changes.
  12. @drkarl drkarl revised this gist Mar 16, 2015. 1 changed file with 42 additions and 40 deletions.
    82 changes: 42 additions & 40 deletions Linux-backup.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -1,57 +1,59 @@
    # Linux Backup Solutions

    I've been looking for the best Linux backup system, and also reading lots of HN comments.
    Instead of putting pros and cons of every backup system I'll just list some deal-breakers which would disqualify them.
    Also I would like that you, the HN community, would add more deal breakers for these or other backup systems if you know some more and at the same time, if you have data to disprove some of the deal-breakers listed here (benchmarks, info about something being true for older releases but is fixed on newer releases), please share it so that I can edit this list accordingly.

    Bacula (from the Why section on Burp):
    Too complex to configure
    Stores catalog separate from backups, need to backup catalog
    Doesn't deduplicate
    Relies on clock accuracy
    Can't resume an interrupted backup
    Retention policy

    Snebu:
    Doesn't do encryption

    Obnam:
    Really slow for large backups (from a benchmark between obnam and attic)
    To improve performance:
    lru-size=1024
    upload-queue-size=512
    ## Bacula (from the Why section on Burp):
    - Too complex to configure
    - Stores catalog separate from backups, need to backup catalog
    - Doesn't deduplicate
    - Relies on clock accuracy
    - Can't resume an interrupted backup
    - Retention policy

    ## Snebu:
    - Doesn't do encryption

    ## Obnam:
    - Really slow for large backups (from a benchmark between obnam and attic)
    - To improve performance:
    ```lru-size=1024
    upload-queue-size=512```
    as per: http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2014-June/003086.html

    Burp:
    Client side encryption turns off delta differencing
    ## Burp:
    - Client side encryption turns off delta differencing

    Bup:
    Can't purge old backups
    Doesn't encrypt backups (well, there is encbup)
    ## Bup:
    - Can't purge old backups
    - Doesn't encrypt backups (well, there is encbup)

    Tarsnap:
    Slow restore performance on large backups? (Sorry Colin aka cperciva)
    This was a really strong candidate until I read some comments on HN about the slow performance to restore large backups.
    If this has changed in a recent version or someone has benchmarks to prove or disprove it, it would be really valuable.
    ## Tarsnap:
    - Slow restore performance on large backups? (Sorry Colin aka cperciva)
    - This was a really strong candidate until I read some comments on HN about the slow performance to restore large backups.
    - If this has changed in a recent version or someone has benchmarks to prove or disprove it, it would be really valuable.

    Duplicity:
    Slow restore performance on large backups?
    This was also a really strong candidate until I read some comments on HN about the slow performance to restore large backups.
    If this has changed in a recent version or someone has benchmarks to prove or disprove it, it would be really valuable.
    ## Duplicity:
    - Slow restore performance on large backups?
    - This was also a really strong candidate until I read some comments on HN about the slow performance to restore large backups.
    - If this has changed in a recent version or someone has benchmarks to prove or disprove it, it would be really valuable.

    BackupPC:
    It doesn't do encrypted backups
    ## BackupPC:
    - It doesn't do encrypted backups

    backup2l:
    No support for encryption
    ## backup2l:
    - No support for encryption

    Arq:
    Just included here because I knew someone would mention it in the comments.
    ## Arq:
    - Just included here because I knew someone would mention it in the comments.
    It's Mac OS X only. This list is for Linux server backup systems.

    Other contenders (of which I don't have references or information):
    HashBackup
    ZBackup
    ZPaq
    BTRFS send/receive
    ## Other contenders (of which I don't have references or information):
    - HashBackup
    - ZBackup
    - ZPaq
    - BTRFS send/receive

    Also Tarsnap scores really high on encryption and deduplication but it has 3 important cons:
    - Not having control of the server where your backups are stored
  13. @drkarl drkarl revised this gist Mar 16, 2015. 1 changed file with 3 additions and 0 deletions.
    3 changes: 3 additions & 0 deletions Linux-backup.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -37,6 +37,9 @@ Slow restore performance on large backups?
    This was also a really strong candidate until I read some comments on HN about the slow performance to restore large backups.
    If this has changed in a recent version or someone has benchmarks to prove or disprove it, it would be really valuable.

    BackupPC:
    It doesn't do encrypted backups

    backup2l:
    No support for encryption

  14. @drkarl drkarl revised this gist Mar 16, 2015. 1 changed file with 4 additions and 0 deletions.
    4 changes: 4 additions & 0 deletions Linux-backup.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -15,6 +15,10 @@ Doesn't do encryption

    Obnam:
    Really slow for large backups (from a benchmark between obnam and attic)
    To improve performance:
    lru-size=1024
    upload-queue-size=512
    as per: http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2014-June/003086.html

    Burp:
    Client side encryption turns off delta differencing
  15. @drkarl drkarl revised this gist Mar 16, 2015. 1 changed file with 4 additions and 0 deletions.
    4 changes: 4 additions & 0 deletions Linux-backup.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -33,6 +33,9 @@ Slow restore performance on large backups?
    This was also a really strong candidate until I read some comments on HN about the slow performance to restore large backups.
    If this has changed in a recent version or someone has benchmarks to prove or disprove it, it would be really valuable.

    backup2l:
    No support for encryption

    Arq:
    Just included here because I knew someone would mention it in the comments.
    It's Mac OS X only. This list is for Linux server backup systems.
    @@ -41,6 +44,7 @@ Other contenders (of which I don't have references or information):
    HashBackup
    ZBackup
    ZPaq
    BTRFS send/receive

    Also Tarsnap scores really high on encryption and deduplication but it has 3 important cons:
    - Not having control of the server where your backups are stored
  16. @drkarl drkarl created this gist Mar 16, 2015.
    50 changes: 50 additions & 0 deletions Linux-backup.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,50 @@
    I've been looking for the best Linux backup system, and also reading lots of HN comments.
    Instead of putting pros and cons of every backup system I'll just list some deal-breakers which would disqualify them.
    Also I would like that you, the HN community, would add more deal breakers for these or other backup systems if you know some more and at the same time, if you have data to disprove some of the deal-breakers listed here (benchmarks, info about something being true for older releases but is fixed on newer releases), please share it so that I can edit this list accordingly.

    Bacula (from the Why section on Burp):
    Too complex to configure
    Stores catalog separate from backups, need to backup catalog
    Doesn't deduplicate
    Relies on clock accuracy
    Can't resume an interrupted backup
    Retention policy

    Snebu:
    Doesn't do encryption

    Obnam:
    Really slow for large backups (from a benchmark between obnam and attic)

    Burp:
    Client side encryption turns off delta differencing

    Bup:
    Can't purge old backups
    Doesn't encrypt backups (well, there is encbup)

    Tarsnap:
    Slow restore performance on large backups? (Sorry Colin aka cperciva)
    This was a really strong candidate until I read some comments on HN about the slow performance to restore large backups.
    If this has changed in a recent version or someone has benchmarks to prove or disprove it, it would be really valuable.

    Duplicity:
    Slow restore performance on large backups?
    This was also a really strong candidate until I read some comments on HN about the slow performance to restore large backups.
    If this has changed in a recent version or someone has benchmarks to prove or disprove it, it would be really valuable.

    Arq:
    Just included here because I knew someone would mention it in the comments.
    It's Mac OS X only. This list is for Linux server backup systems.

    Other contenders (of which I don't have references or information):
    HashBackup
    ZBackup
    ZPaq

    Also Tarsnap scores really high on encryption and deduplication but it has 3 important cons:
    - Not having control of the server where your backups are stored
    - Bandwith costs make your costs unpredictable
    - The so called Colin-Percival-gets-hit-by-a-bus scenario

    Attic has some really good comments on HN and good blog posts, doesn't have any particular deal-breaker (for now, if you have one please share with us), so for now is the most promising.