Does encfs encryption break rsync byte-difference algorithm?

Rsync is using an optimized algorithm in order to minimize the overhead when one size is synchronized up several times but it has small changes. To keep simple, the similar parts are skipped while only the differences are synchronized again (see more details).

One can wonder is using a secured remote backup method like I have previously talked out setup does not defeat the purpose of this beautiful rsync algorithm.

Cases where it works

If the file is appended to, or some internal part change without affecting the rest, then there should not be any problem as encfs is using block-level encryption. In other words, if a byte changes in a file, only the corresponding block in the encrypted version is altered.

The small test below confirms that practically this works.

First, let’s create a random bunch of files which are 1000000 bytes long

jc@workstation:~$ for i in {0..5}; do dd bs=1000000 count=1 skip=0 if=/dev/urandom of=r$i; done

And we combine them into a 5000000 bytes file

jc@workstation:~$ cat r0 r1 r2 r3 r4 > data/file

Let’s now rsync this over to the remote backup server

jc@workstation:~$ rsync --archive --progress data/file rsync://server/test
sending incremental file list
file
     5000000 100%
sent 5000524 bytes  received 13473 bytes
total size is 5000000

As you can see the full file has been copied : a bit more than 5000000 bytes (5000524 bytes sent).
If we copy over the encrypted version of the file, here again the full file is sent across, of course :

jc@workstation:~$ rsync --archive --progress data-encrypted/ywSJwhDVfbKogK6MKSLCGjvt rsync://server/test
sending incremental file list
ywSJwhDVfbKogK6MKSLCGjvt
     5000000 100%
sent 5000968 bytes  received 13424 bytes
total size is 5000000

Now, let’s modify our original file a bit

jc@workstation:~$ cat r0 r5 r2 r3 r4 > data/file

And synchronize them over again

jc@workstation:~$ rsync -a --progress data/file rsync://server/test
sending incremental file list
file
     5000000 100%
sent 1009524 bytes  received 13473 bytes
total size is 5000000
jc@workstation:~$ rsync -a --progress data-encrypted/ywSJwhDVfbKogK6MKSLCGjvt rsync://server/test
sending incremental file list
ywSJwhDVfbKogK6MKSLCGjvt
     5000000 100%
sent 1009544 bytes  received 13473 bytes
total size is 5000000

You can verify that both for the original and encrypted version, only the difference has been synchronized (about 1000000 bytes).

Cases where it breaks

If we take some extreme examples where a single byte is appended to the beginning of a file, rsync is able to handle this well on the original file, while there are too many changes on the encrypted version, resulting in a full copy

jc@workstation:~$ dd bs=500000 count=1 skip=0 if=/dev/urandom of=mainfile
1+0 records in
1+0 records out
500000 bytes (500 kB) copied, 0,18426 s, 2,7 MB/s
jc@workstation:~$ dd bs=1 count=1 skip=0 if=/dev/urandom of=extrabyte
1+0 records in
1+0 records out
1 byte (1 B) copied, 9,4875e-05 s, 10,5 kB/s
jc@workstation:~$ cat mainfile >data/file
jc@workstation:~$ rsync --archive --progress data/file rsync://server/backup
sending incremental file list
file
      500000 100%   44.34kB/s    0:00:10 (xfer#1, to-check=0/1)

sent 500128 bytes  received 6033 bytes  27360.05 bytes/sec
total size is 500000  speedup is 0.99
jc@workstation:~$ rsync --archive --progress encrypted/ywSJwhDVfbKogK6MKSLCGjvt  rsync://server/backup
sending incremental file list
ywSJwhDVfbKogK6MKSLCGjvt
      500000 100%   55.35kB/s    0:00:08 (xfer#1, to-check=0/1)

sent 500148 bytes  received 27 bytes  32269.35 bytes/sec
total size is 500000  speedup is 1.00
jc@workstation:~$ cat extrabyte mainfile >data/file
jc@workstation:~$ rsync --archive --progress data/file rsync://server/backup
sending incremental file list
file
      500001 100%   52.91MB/s    0:00:00 (xfer#1, to-check=0/1)

sent 2913 bytes  received 4293 bytes  1108.62 bytes/sec
total size is 500001  speedup is 69.39
jc@workstation:~$ rsync --archive --progress encrypted/ywSJwhDVfbKogK6MKSLCGjvt  rsync://server/backup
sending incremental file list
ywSJwhDVfbKogK6MKSLCGjvt
      500001 100%   42.94kB/s    0:00:10 (xfer#1, to-check=0/1)

sent 500149 bytes  received 4293 bytes  28825.26 bytes/sec
total size is 500001  speedup is 0.99
· · Posted in · Tagged ,