Object Storage with Restic and Rclone

I have been playing around with some options to utilize Object Storage for backups. Since I am working on Oracle Cloud Infrastructure (OCI) I am doing my POC using the OCI Object Storage. OCI object storage does have Swift and S3 Compatibility API”s to interface with. Of course if you want commercial backups many of them can use object storage as back-ends now so that would be the correct answer. If your needs does not warrant commercial backups solutions you can try several things. A few options I played with.

1. Bareos server/client with the object storage droplet. Not working reliably. Too experimental with droplet?
2. Rclone and using tar to pipe with rclone”s rcat feature. This works well but is not a backup solution as in incrementals etc.
3. Duplicati. In my case using rclone as connection since S3 interface on OCI did not work.
4. Dupliciti. Could not get this one to work to S3 interface on OCI.
5. Restic. In my case using rclone as connection since S3 interface on OCI did not work.

So far duplicati was not bad but had some bugs. It is beta software so probably should expect problems. Restic is doing a good job so far and I show a recipe of my POC below:

Out of scope is setting up rclone, rclone.conf. Make sure you test that rclone is accessing your bucket first.

Restic binary

1
# wget https://github.com/restic/restic/releases/download/v0.9.1/restic_0.9.1_linux_amd64.bz2
2
2018-08-03 10:25:10 (3.22 MB/s) - "restic_0.9.1_linux_amd64.bz2" saved [3786622/3786622]
3
# bunzip2 restic_0.9.1_linux_amd64.bz2
4
# mv restic_0.9.1_linux_amd64 /usr/local/bin/
5
# chmod +x /usr/local/bin/restic_0.9.1_linux_amd64
6
# mv /usr/local/bin/restic_0.9.1_linux_amd64 /usr/local/bin/restic
7
# /usr/local/bin/restic version
8
restic 0.9.1 compiled with go1.10.3 on linux/amd64

Initialize repo

1
# rclone ls s3_servers_phoenix:oci02a
2
# export RESTIC_PASSWORD="WRHYEjblahblah0VWq5qM"
3
# /usr/local/bin/restic -r rclone:s3_servers_phoenix:oci02a init
4
created restic repository 2bcf4f5864 at rclone:s3_servers_phoenix:oci02a
5

6
Please note that knowledge of your password is required to access
7
the repository. Losing your password means that your data is
8
irrecoverably lost.
9

10
# rclone ls s3_servers_phoenix:oci02a
11
      155 config
12
      458 keys/530a67c4674b9abf6dcc9e7b75c6b319187cb8c3ed91e6db992a3e2cb862af63

Run a backup

1
# time /usr/local/bin/restic -r rclone:s3_servers_phoenix:oci02a backup /opt/applmgr/12.2
2
repository 2bcf4f58 opened successfully, password is correct
3

4
Files:       1200934 new,     0 changed,     0 unmodified
5
Dirs:            2 new,     0 changed,     0 unmodified
6
Added:      37.334 GiB
7

8
processed 1200934 files, 86.311 GiB in 1:31:40
9
snapshot af4d5598 saved
10

11
real  91m40.824s
12
user  23m4.072s
13
sys  7m23.715s
14

15
# /usr/local/bin/restic -r rclone:s3_servers_phoenix:oci02a snapshots
16
repository 2bcf4f58 opened successfully, password is correct
17
ID        Date                 Host              Tags        Directory
18
----------------------------------------------------------------------
19
af4d5598  2018-08-03 10:35:45  oci02a              /opt/applmgr/12.2
20
----------------------------------------------------------------------
21
1 snapshots

Run second backup

1
# /usr/local/bin/restic -r rclone:s3_servers_phoenix:oci02a backup /opt/applmgr/12.2
2
repository 2bcf4f58 opened successfully, password is correct
3

4
Files:           0 new,     0 changed, 1200934 unmodified
5
Dirs:            0 new,     0 changed,     2 unmodified
6
Added:      0 B
7

8
processed 1200934 files, 86.311 GiB in 47:46
9
snapshot a158688a saved

Example cron entry

1
# crontab -l
2
05 * * * * /usr/local/bin/restic -r servers_phoenix:oci02a backup -q /usr; /usr/local/bin/restic -r servers_phoenix:oci02a forget -q --prune --keep-hourly 2 --keep-daily 7