Managing Object Storage Duplicity

Duplicity is a Linux backup client to backup files from your Linux desktop or server.


Supported OS


Installing Duplicity

apt based linux distributions
sudo apt-get install duplicity duply python-boto
rpm based linux distributions
sudo yum install duplicity duply python-boto

Set up your configuration with Duplicity using CDN Object Storage

Set up the authentication

  1. Create a profile. Let’s name this profile “test”.

    duply test create

    This will create a configuration file in $HOME/.duply/test/conf.

  2. Open it in your editor.
    You will be presented with a lot of configuration options, but only a few are really important.
    One of them is GPG_KEY and GPG_PW.
    Duplicity supports asymmetric public-key encryption, or symmetric password-only encryption.
    For the purposes of this tutorial we’re going to use symmetric password-only encryption because it’s the easiest.
  3. Generate a random, secure password:

    openssl rand -base64 20
  4. While editing  $HOME/.duply/test/conf comment out GPG_KEY and set a password in GPG_PW:

    GPG_PW='<the password you just got from openssl>'
  5. Scroll down and set the TARGET options.

    TARGET='s3://<bucket name>/<folder name>'
    TARGET_USER='<your cloudstorage access key>'
    TARGET_PASS='<your cloudstorage secret key>'
  6. Set the base directory of the backup. We want to backup the entire filesystem:

  7. It is also possible to set a maximum time for keeping old backups. In this tutorial, let’s set it to 6 months.

  8. Save and close the configuration file.

Exclude files not to backed up

There are also some things that we never want to backup, such as /tmp, /dev and log files.

So we create an exclusion file $HOME/.duply/test/exclude with the following contents:

- /dev
- /home/*/.cache
- /home/*/.ccache
- /lost+found
- /media
- /mnt
- /proc
- /root/.cache
- /root/.ccache
- /run
- /selinux
- /sys
- /tmp
- /var/cache/*/*
- /var/log
- /var/run
- /var/tmp

This file follows the Duplicity file list syntax. The - sign here means “exclude this directory”. For more information, please refer to the Duplicity man page.

Create Bucket

Finally, use s3cmd (see this page: Managing Object Storage Duplicity) to create your storage bucket.

Initiating the backup with Duplicity using CDN Object Storage

We’re now ready to initiate the backup. This can take a while, so let’s open a screen session so that we can terminate the SSH session and check back later.

  1. Open a screen session

    sudo apt-get install screen
  2. Initiate the backup:

    sudo duply test backup

    Press Esc-D to detach the screen session.

  3. Check back a few hours later. Login to your server and reattach your screen session:

    screen -R
  4. You should see something like this, which means that the backup succeeded. Congratulations!

    --------------[ Backup Statistics ]--------------


    Errors 0


    --- Finished state OK at 16:48:16.192 - Runtime 01:17:08.540 ---

    --- Start running command POST at 16:48:16.213 ---

    Skipping n/a script '/home/admin/.duply/main/post'.

    --- Finished state OK at 16:48:16.244 - Runtime 00:00:00.031 ---

Setting up periodic incremental backups with cron

We can use cron, the system’s periodic task scheduler, to setup periodic incremental backups.

  1. Edit root’s crontab:

    sudo crontab -e
  2. Insert the following:

    0 2 * * 7 env HOME=/root duply test backup

    This line runs the duply test backup command every Sunday at 2:00 AM.
    If you want to setup daily backups, replace “0 2 * * 7” with “0 2 * * *”.

Restoring a backup with Duplicity using CDN Object Storage

Simple restores

You can restore the latest backup with the Duply restore command.

It is important to use sudo because this allows Duplicity to restore the original filesystem metadata.

The following will restore the latest backup to a specific directory.
The target directory does not need to exist, Duplicity will automatically create it.
After restoration, you can move its contents to the root filesystem using mv.

  1. Restore files

    sudo duply test restore /restored_files

    You can’t just do sudo duply test restore / here because your system files (e.g. bash, libc, etc) are in use.

    Moving the files from /restored_files to / using mv might still not work for you.
    In that case, consider booting your server from a rescue system and restoring from there.

Restoring a specific file or directory

  1. Use the fetch command to restore a specific file.

    sudo duply test fetch etc/passwd /home/admin/passwd

    This restores the /etc/passwd file in the backup and saves it to /home/admin/passwd.

    Notice the lack of leading slash in the etc/passwd argument.

    The fetch command also works on directories:

    sudo duply test fetch etc /home/admin/etc

Restoring from a specific date

Every restoration command accepts a date, allowing you to restore from that specific date.

  1. Use the status command to get an overview of backup dates:

    duply test status
    Number of contained backup sets: 2
    Total number of contained volumes: 2
     Type of backup set:                            Time:      Num volumes:
                    Full         Sat Nov  8 07:38:30 2017                 1
             Incremental         Sat Nov  9 07:43:17 2017                 1

    In this example, you are restoring the November 8 backup. 
    Unfortunately you can’t just copy and paste the time string.
    Instead, you'll have to write the time in the w3 format.

    See also the Time Formats section in the Duplicity man page.

    sudo duply test restore /restored_files '2017-11-08T07:38:30'

    Safely store your keys or passwords!
    Whether you used asymmetric public-key encryption or symmetric password-only encryption, you must store them safely!
    If you ever lose them, you will lose your data.
    There is no way to recover encrypted data for which the key or password is lost.

Get Support

Need Technical Support?

Have a specific challenge with your setup?

Create a Ticket