Object Storage with Duplicity and Rclone

At this point I prefer using restic for my object storage backup needs but since I did a POC for duplicity and specifically using rclone with duplicity I am writing down my notes. A good description of duplicity and restic here:

Backing Up Linux to Backblaze B2 with Duplicity and Restic

We"re highlighting Duplicity and Restic because they exemplify two different philosophical approaches to data backup: "Old School" (Duplicity) vs "New School" (Restic).

Since I am doing my tests with Oracle Cloud Infrastructure (OCI) Object Storage and so far it”s Amazon S3 Compatibility Interface does not work out of the box with most tools except with rclone, I am using rclone as a backend. With restic using rclone as a back-end worked pretty smooth but duplicity does not have good rclone support so I used a python back-end written by Francesco Magno and hosted here: https://github.com/GilGalaad/duplicity-rclone/blob/master/README.md

I had a couple issues with getting duplicity to work with this back-end so I will show how to get around it.

First:
1. Make sure rclone is working with your rclone config and can at least “ls” your bucket.
2. Setup a gpg key.
3. Copy rclonebackend.py to duplicity backends folder. In my case /usr/lib64/python2.7/site-packages/duplicity/backends

1
# PASSPHRASE="mypassphrase" duplicity --encrypt-key 094CA414 /tmp rclone://mycompany-POC-phoenix:dr01-duplicity
2
InvalidBackendURL: Syntax error (port) in: rclone://mycompany-POC-phoenix:dr01-duplicity AFalse BNone Cmycompany-POC-phoenix:dr01-duplicity

## Hack backends.py

1
# diff /usr/lib64/python2.7/site-packages/duplicity/backend.py /tmp/backend.py
2
303c303
3
< if not (self.scheme in ['rsync'] and re.search('::[^:]*$', self.url_string) or (self.scheme in ['rclone']) ): --- >             if not (self.scheme in ['rsync'] and re.search('::[^:]*$', self.url_string)):

1
# PASSPHRASE="mypassphrase" duplicity --encrypt-key 094CA414 /tmp rclone://mycompany-POC-phoenix:dr01-duplicity
2
Local and Remote metadata are synchronized, no sync needed.
3
Last full backup date: none
4
No signatures found, switching to full backup.
5
--------------[ Backup Statistics ]--------------
6
StartTime 1533652997.49 (Tue Aug  7 14:43:17 2018)
7
EndTime 1533653022.35 (Tue Aug  7 14:43:42 2018)
8
ElapsedTime 24.86 (24.86 seconds)
9
SourceFiles 50
10
SourceFileSize 293736179 (280 MB)
11
NewFiles 50
12
NewFileSize 136467418 (130 MB)
13
DeletedFiles 0
14
ChangedFiles 0
15
ChangedFileSize 0 (0 bytes)
16
ChangedDeltaSize 0 (0 bytes)
17
DeltaEntries 50
18
RawDeltaSize 293723433 (280 MB)
19
TotalDestinationSizeChange 279406571 (266 MB)
20
Errors 0
21
-------------------------------------------------
22

23
# rclone ls mycompany-POC-phoenix:dr01-duplicity
24
  1773668 duplicity-full-signatures.20180807T144317Z.sigtar.gpg
25
      485 duplicity-full.20180807T144317Z.manifest.gpg
26
209763240 duplicity-full.20180807T144317Z.vol1.difftar.gpg
27
 69643331 duplicity-full.20180807T144317Z.vol2.difftar.gpg
28

29
# PASSPHRASE="mypassphrase" duplicity --encrypt-key 094CA414 collection-status rclone://mycompany-POC-phoenix:dr01-duplicity
30
Last full backup date: Tue Aug  7 14:43:17 2018
31
Collection Status
32
-----------------
33
Connecting with backend: BackendWrapper
34
Archive dir: /root/.cache/duplicity/df529824ba5d10f9e31329e440c5efa6
35

36
Found 0 secondary backup chains.
37

38
Found primary backup chain with matching signature chain:
39
-------------------------
40
Chain start time: Tue Aug  7 14:43:17 2018
41
Chain end time: Tue Aug  7 14:50:12 2018
42
Number of contained backup sets: 2
43
Total number of contained volumes: 3
44
 Type of backup set:                            Time:      Num volumes:
45
                Full         Tue Aug  7 14:43:17 2018                 2
46
         Incremental         Tue Aug  7 14:50:12 2018                 1
47
-------------------------
48
No orphaned or incomplete backup sets found.