Since btrfs has send and receive capabilities I took a look at it. The title is replication but if you are interested in enterprise level sophisticated storage level replication for disaster recover or better yet mature data set cloning for non production instances you will need to look further. For example the Oracle ZFS appliance has a mature replication engine built on send and receive but handles all the replication magic for you. I am not aware of commercial solutions built on btrfs that has the mature functionality the ZFS appliance can offer yet. Note we are not just talking replication but also snapshot cloning, sharing and protection of snapshots on the target end. So for now here is what I have tested for pure btrfs send and receive.
Some details on machine 1:
root@u1604b1-m1:~# more /etc/issueUbuntu Xenial Xerus (development branch) \n \l
root@u1604b1-m1:~# df -hFilesystem Size Used Avail Use% Mounted on[..]/dev/sda1 7.5G 3.9G 3.1G 56% //dev/sda1 7.5G 3.9G 3.1G 56% /home[..]
root@u1604b1-m1:~# mount[..]/dev/sda1 on / type btrfs (rw,relatime,space_cache,subvolid=257,subvol=/@)/dev/sda1 on /home type btrfs (rw,relatime,space_cache,subvolid=258,subvol=/@home)[..]
root@u1604b1-m1:~# btrfs --versionbtrfs-progs v4.4
root@u1604b1-m1:~# btrfs subvolume list /ID 257 gen 47 top level 5 path @ID 258 gen 47 top level 5 path @homeTest ssh to machine 2:
root@u1604b1-m1:~# ssh root@192.168.2.29 uptimeroot@192.168.2.29's password: 10:33:23 up 5 min, 1 user, load average: 0.22, 0.37, 0.19Machine 2 subvolumes before we receive:
root@u1604b1-m2:~# btrfs subvolume list /ID 257 gen 40 top level 5 path @ID 258 gen 40 top level 5 path @homeCreate a subvolume, add a file and take a snapshot:
root@u1604b1-m1:~# btrfs subvolume create /tank1Create subvolume '//tank1'
root@u1604b1-m1:~# btrfs subvolume list /ID 257 gen 53 top level 5 path @ID 258 gen 50 top level 5 path @homeID 264 gen 53 top level 257 path tank1
root@u1604b1-m1:~# ls /tank1
root@u1604b1-m1:~# touch /tank1/rr_test1root@u1604b1-m1:~# ls -l /tank1/total 0-rw-r--r-- 1 root root 0 Mar 10 10:38 rr_test1
root@u1604b1-m1:~# btrfs subvolume snapshot /tank1 /tank1_snapshotCreate a snapshot of '/tank1' in '//tank1_snapshot'
root@u1604b1-m1:~# ls -l /tank1_snapshot/total 0-rw-r--r-- 1 root root 0 Mar 10 10:38 rr_test1
root@u1604b1-m1:~# btrfs subvolume list /ID 257 gen 63 top level 5 path @ID 258 gen 58 top level 5 path @homeID 264 gen 59 top level 257 path tank1ID 265 gen 59 top level 257 path tank1_snapshotDelete a snapshot:
root@u1604b1-m1:~# btrfs subvolume delete /tank1_snapshotDelete subvolume (no-commit): '//tank1_snapshot'
root@u1604b1-m1:~# btrfs subvolume list /ID 257 gen 64 top level 5 path @ID 258 gen 58 top level 5 path @homeID 264 gen 59 top level 257 path tank1Take a read-only snapshot and send to machine 2:
root@u1604b1-m1:~# btrfs subvolume snapshot -r /tank1 /tank1_snapshot
Create a readonly snapshot of '/tank1' in '//tank1_snapshot'
root@u1604b1-m1:~# btrfs send /tank1_snapshot | ssh root@192.168.2.29 "btrfs receive /"At subvol /tank1_snapshotroot@192.168.2.29's password:At subvol tank1_snapshot
Machine 2 after receiving snapshot1:
root@u1604b1-m2:~# btrfs subvolume list /ID 257 gen 61 top level 5 path @ID 258 gen 60 top level 5 path @homeID 264 gen 62 top level 257 path tank1_snapshot
root@u1604b1-m2:~# ls -l /tank1_snapshot/total 0-rw-r--r-- 1 root root 0 Mar 10 10:38 rr_test1Create one more file:
root@u1604b1-m1:~# touch /tank1/rr_test2
root@u1604b1-m1:~# btrfs subvolume snapshot -r /tank1 /tank1_snapshot2Create a readonly snapshot of '/tank1' in '//tank1_snapshot2'
root@u1604b1-m1:~# btrfs send /tank1_snapshot2 | ssh root@192.168.2.29 "btrfs receive /"At subvol /tank1_snapshot2root@192.168.2.29's password:At subvol tank1_snapshot2Machine 2 after receiving snapshot 2:
root@u1604b1-m2:~# btrfs subvolume list /ID 257 gen 65 top level 5 path @ID 258 gen 60 top level 5 path @homeID 264 gen 62 top level 257 path tank1_snapshotID 265 gen 66 top level 257 path tank1_snapshot2
root@u1604b1-m2:~# ls -l /tank1_snapshot2/total 0-rw-r--r-- 1 root root 0 Mar 10 10:38 rr_test1-rw-r--r-- 1 root root 0 Mar 10 10:53 rr_test2Incremental send(adding -v for now to see more detail):
root@u1604b1-m1:~# btrfs subvolume snapshot -r /tank1 /tank1_snapshot3Create a readonly snapshot of '/tank1' in '//tank1_snapshot3'
root@u1604b1-m1:~# btrfs send -vp /tank1_snapshot2 /tank1_snapshot3 | ssh root@192.168.2.29 "btrfs receive /"At subvol /tank1_snapshot3BTRFS_IOC_SEND returned 0joining genl threadroot@192.168.2.29's password:At snapshot tank1_snapshot3Using larger files to see effect of incremental better:
root@u1604b1-m1:~# cp /media/sf_E_DRIVE/ISO/ubuntu-gnome-15.10-desktop-amd64.iso /tank1/root@u1604b1-m1:~# du -sh /tank11.1G /tank1
root@u1604b1-m1:~# btrfs subvolume snapshot -r /tank1 /tank1_snapshot6Create a readonly snapshot of '/tank1' in '//tank1_snapshot6'
root@u1604b1-m1:~# time btrfs send -vp /tank1_snapshot5 /tank1_snapshot6 | ssh root@192.168.2.29 "btrfs receive -v /"At subvol /tank1_snapshot6root@192.168.2.29's password:receiving snapshot tank1_snapshot6 uuid=d38490b3-e6ee-3f41-b63d-460d11f8e757, ctransid=272 parent_uuid=ec3f1fb5-9bed-3e4c-9c5b-a6c586b10531, parent_ctransid=201BTRFS_IOC_SEND returned 0joining genl threadBTRFS_IOC_SET_RECEIVED_SUBVOL uuid=d38490b3-e6ee-3f41-b63d-460d11f8e757, stransid=272At snapshot tank1_snapshot6
real 1m10.578suser 0m0.696ssys 0m16.064sMachine 2 after snapshot6:
total 1.1G-rw-r--r-- 1 root root 0 Mar 10 10:38 rr_test1-rw-r--r-- 1 root root 22 Mar 10 12:19 rr_test2-rwxr-x--- 1 root root 1.1G Mar 10 13:04 ubuntu-gnome-15.10-desktop-amd64.isoroot@u1604b1-m1:~# cp /media/sf_E_DRIVE/ISO/ubuntu-gnome-15.10-desktop-i386.iso /tank1/
root@u1604b1-m1:~# btrfs subvolume snapshot -r /tank1 /tank1_snapshot7Create a readonly snapshot of '/tank1' in '//tank1_snapshot7'
root@u1604b1-m1:~# time btrfs send -vp /tank1_snapshot6 /tank1_snapshot7 | ssh root@192.168.2.29 "btrfs receive -v /"At subvol /tank1_snapshot7root@192.168.2.29's password:receiving snapshot tank1_snapshot7 uuid=5c255311-0f60-4149-91f7-99d9d5acf64c, ctransid=276 parent_uuid=d38490b3-e6ee-3f41-b63d-460d11f8e757, parent_ctransid=272
BTRFS_IOC_SEND returned 0joining genl threadBTRFS_IOC_SET_RECEIVED_SUBVOL uuid=5c255311-0f60-4149-91f7-99d9d5acf64c, stransid=276At snapshot tank1_snapshot7
real 1m17.393suser 0m0.640ssys 0m16.716s
Machine 2 after snapshot7:
root@u1604b1-m2:~# ls -lh /tank1_snapshot7total 2.0G-rw-r--r-- 1 root root 0 Mar 10 10:38 rr_test1-rw-r--r-- 1 root root 22 Mar 10 12:19 rr_test2-rwxr-x--- 1 root root 1.1G Mar 10 13:04 ubuntu-gnome-15.10-desktop-amd64.iso-rwxr-x--- 1 root root 1.1G Mar 10 13:07 ubuntu-gnome-15.10-desktop-i386.isoExperiment: On the target I sent a snapshot to a new btrfs subvolume. So in effect become independent. This does not really help us with cloning since with large datasets it takes too long as well as duplicate the space which nullifies why we like COW.
root@u1604b1-m2:~# btrfs subvolume create /tank1_cloneCreate subvolume '//tank1_clone'
root@u1604b1-m2:~# btrfs send /tank1_snapshot3 | btrfs receive /tank1_cloneAt subvol /tank1_snapshot3At subvol tank1_snapshot3This was just my initial look see on what btrfs is capable of and how similar it is to ZFS and ZFS appliance functionality.
So far at least it seems promising that send and receive is being addressed in btrfs but I don”t think you can easily roll your own solution for A) replication and B) writable snapshots(clones) with btrfs yet. There will be too much work around building the replication discipline and framework.
A few links that I came across that are useful while looking at btrfs and the topics of replication and database cloning.
1. http://rockstor.com/blog/snapshots/data-replication-with-rockstor/
2. http://blog.contractoracle.com/2013/02/oracle-database-on-btrfs-reduce-costs.html
3. http://www.cybertec.at/2015/01/forking-databases-the-art-of-copying-without-copying/
4. http://blog.contractoracle.com/2013/02/oracle-database-on-btrfs-reduce-costs.html
5. https://bdrouvot.wordpress.com/2014/04/25/reduce-resource-consumption-and-clone-in-seconds-your-oracle-virtual-environment-on-your-laptop-using-linux-containers-and-btrfs/
6. https://docs.opensvc.com/storage.btrfs.html
7. https://ilmarkerm.eu/blog/2014/08/cloning-pluggable-database-with-custom-snapshot/
8. http://blog.ronnyegner-consulting.de/2010/02/17/creating-database-clones-with-zfs-really-fast/
9. http://www.seedsofgenius.net/solaris/zfs-vs-btrfs-a-reference