Blog - /Tech

I'm now using Amazon S3 to store backups of my data. Duplicity is being used to encrypt, create, and manage the backups. I'm quite happy with the arrangement so far.

I'm using the 0.4.9 release of duplicity. If you're using Debian or Ubuntu, be sure to also install the python-boto package. The s3cmd package is handy for listing the contents of your "buckets".

Here are some scripts that I wrote to facilitate this. Replace the angle-bracketed parameters as appropriate.

s3.common

I chose to store the access key and secret key into files, rather than writing them in.

# -*- Shell-Script -*-

export AWS_ACCESS_KEY_ID=$(cat ~/<path to access key>)
export AWS_SECRET_ACCESS_KEY=$(cat <path to secret key>)
GPG_KEY=<your GPG key ID>
unset http_proxy  # needed because GNOME sets this badly

s3-backup-git

This one uses a rudimentary "plist", or "property list" to do some initial rsyncing of the scattered git repos where I manage /home and /etc. If you care about the file permissions and ownership, then you will want to adjust the parameters used for rsync.

#!/bin/bash
#
# Backup all of my git repos.

. $(dirname $0)/s3.common

GIT_BACKUPS="
/etc/.git/ etc.git/
~/.git/ home.git/
/root/.git/ root.git/
"

BKDIR=/stuff/backups/git
S3_DEST=s3+http://<your site name>-backups/git

# Make local copy with proper directory hierarchy
key= ; val=
for i in $GIT_BACKUPS; do
    if test -z $key; then
        key=$i
        continue
    else
        val=$i
    fi
    echo "Syncing $val ..."
    rsync -r -E -l --delete $key $BKDIR/$val
    key= ; val=
done

# Do backups
echo "Backing up $BKDIR ..."
echo
duplicity --encrypt-key $GPG_KEY $BKDIR $S3_DEST

s3-restore-git

#!/bin/bash
#
# Restore git repos from Amazon S3.

. $(dirname $0)/s3.common

RST_DIR=/stuff/backups/restored/git
S3_DEST=s3+http://<your site name>-backups/git

# Make sure destination exists
mkdir -p $RST_DIR

# Restore backups
duplicity --encrypt-key $GPG_KEY $S3_DEST $RST_DIR