rsync is a file transfer and file synchronization program for Unix and Unix-like computers that provides efficient transfers and synchronization "by sending just the differences in the files across the link, without requiring that both sets of files are present at one of the ends of the link beforehand," according to the rsync website. Because of this efficiency, it is, in some cases, a good alternative to backup and transfer schemes that use tar and scp.
This article will provide some examples of using rsync in both capacities and addresses a problem encountered when using it to transfer very large files.
Among its other features, as listed on the rsync Feature Page, are that it:
- can update whole directory trees and filesystems
- optionally preserves symbolic links, hard links, file ownership, permissions, devices and times
- requires no special privileges to install
- [uses] internal pipelining [to reduce] latency for multiple files
- can use rsh, ssh or direct sockets as the transport
- supports anonymous rsync which is ideal for mirroring
The basic command is similar to cp and scp:
rsync [options] /source/path/ /target/pathThe training slash in /source/path/ ensures the contents of /source/path/ are copied into /target/path. If rsync is used to transfer to a remote host, the command is:
rsync [options] /source/path/ email@example.com:/target/pathAlso like scp, for transfers to remote hosts, it can use ssh to make the connection.
Common options include:
rsync --helpto use rsync effectively.
I use rsync primarily to backup a data partition and a partition that contains a virtual machine disk to an external hard drive. The partitions on the external hard drive are of equal size to the partitions to be backed up. Also, I've assigned volume labels to the partitions on the external hard drive -- this allows me to mount them easily with the labels without having to use the kernel assigned block device names, the UUIDs of the partitions or any of the other available identifiers after first having identified these identifiers.
I have also created directories for mount points -- with names that obviously correspond to the appropriate partition on the hard drive -- in my user directory to make mounting the external hard drives easy. This also avoids ownership problems compared to using the system supplied automatic mount points in /run/media.
After plugging in the external hard drive, the available disks or partitions for backing up can be determined with:
The external hard drive is /dev/sdc. It actually has several partitions, one for each data partition on my laptop's disks. For this example I want to use /dev/sdc1 with the LABEL CDEXT4-backup. (I actually renamed the partition in the course of writing this, so the examples below use DataEXT4-backup and not CDEXT4-backup.)
As mentioned above making a directory to mount the external hard drive backup partitions is desirable as opposed to using the automatic system mounting point. In my case I created a mount point in my user directory and mounted the external hard drive with:
mkdir -p /home/brook/mnt-backup-media/DataEXT4-backup
sudo mount LABEL=DataEXT4-backup /home/brook/mnt-backup-media/DataEXT4-backupor
sudo mount /dev/sdc1 /home/brook/mnt-backup-media/DataEXT4-backup
I chose the first form because it is easier to use -- anytime I plug in the external hard drive that contains this partition, I can just use the LABEL of the volume (not the PARTITITON LABEL) instead of having to look up the /dev/sdX assigned by the system. Also if the USB connection is loose and the drive gets momentarily disconnected, the transfer will fail because a new /dev/sdX is assigned.
At this point the permissions of the mount point directory should be verified. Even though the directory was created by a regular user, after mounting the system may change the owner and group to root, causing the subsequent transfer to fail. This was the case in Fedora 27. So I changed the owner and group back to my user using:
sudo chown -R brook:brook /home/brook/mnt-backup-media/DataEXT4-backupNote that the group name may be different from the user name on some distributions.
rsync -aP --exclude="LinuxLook" --exclude="SoftwareDownloads" --exclude="Entertainment" --delete /home/brook/DataEXT4/ /home/brook/mnt-backup-media/DataEXT4-backup
Transferring files to a remote host using rsync is similar to using scp. In the screenshot below, I use it to transfer some files to a remote host, in this case a virtual machine accessible from the host at IP 192.168.56.101, using:
rsync -rP ./images/optimized/ firstname.lastname@example.org:/var/www/ordinatechnic.com/public_html/sites/all/general-guides/images/optimizedIn this case the -r is not necessary since there aren't any directories in ./images/optimized. The effect of using the -P option is evident.
The error I referred to at the top of the article involves transferring large files, which can cause transfers to fail due memory depletion if the current desktop session has been active for a while. The output from the failed execution
rsync -aP ~/UbuntuWDVM/ /home/brook/mnt-backup-media/UbuntuWDVM-backuis shown in the following screenshot, where the relevant lines are:
This happens to be misleading because there was adequate space on the external drive. The problem in fact happened to be low memory.
rsync: write failed on "/home/brook/mnt-backup-media/UbuntuWDVM-backup/UbuntuWDVM-flat.vmdk": No space left on device (28) rsync error: error in file IO (code 11) at receiver.c(393) receiver=3.1.2]
sync; echo 3 > /proc/sys/vm/drop_cachesand rerunning the command.
While scp and tar are essential programs for file transfer and backup, rsync is more beneficial in certain file transfer and backup scenarios.