10 Linux rsync Examples to Exclude Files/Directories -

10 Linux rsync Examples to Exclude Files/Directories

Rsync (Remote Sync) is a most commonly used command for copying and synchronizing files and directories remotely as well as locally in Linux/Unix systems. With the help of rsync command you can copy and synchronize your data remotely and locally across directories, across disks and networks, perform data backups and mirroring between two Linux machines.

Rsync is one of the most useful utilities for a server administrator, but it syncs everything by default, which can be annoying if your application creates a lot of temporary files. Here’s how to exclude files when using rsync.

Important features of rsync command

Speed : First time, rsync replicates the whole content between the source and destination directories. Next time, rsync transfers only the changed blocks or bytes to the destination location, which makes the transfer really fast.
Security : rsync allows encryption of data using ssh protocol during transfer.
Low Bandwidth: rsync uses compression and decompression of data block by block at the sending and receiving end respectively. So the bandwidth used by rsync will be always less compared to other file transfer protocols like FTP.
Privileges : No special privileges are required to install and execute rsync.

The basic syntax of rsync is very straight forward, and operates in a way that is similar to ssh, scp, and cp.

$ sudo rsync options source destination

-v : verbose.
-r : copies data recursively (but don’t preserve timestamps and permission while transferring data).
-a : archive mode, archive mode allows copying files recursively and it also preserves symbolic links, file permissions, user & group ownerships and timestamps.
-z : compress file data.
-h : human-readable, output numbers in a human-readable format.
-d : transfer directories without recursing -e : specify the ssh as remote shell.

First, create a sample directory structure as shown below (with some empty files) that can be used for testing purpose.

  $ cd ~
  $ sudo mkdir -p source/dir1/dir2
  $ sudo mkdir -p source/dir3
  $ sudo touch source/file1.txt
  $ sudo touch source/file2.txt
  $ sudo touch source/dir1/dir2/file3.txt
  $ sudo touch source/dir3/file4.txt

The above commands will create a source directory (under home directory) with the following structure,To see the structure of source directory type below command.

$ sudo find source/

  source
  - file1.txt
  - file2.txt
  - dir1
    - dir2
      - file3.txt
  - dir3
    - file4.txt

1) Exclude a specific directory

Frist we remove destination directory if it exist by typing below command.

$ sudo rm -rf destination

If we don’t want to sync the dir3 and its subdirectories from the source to the destination folder, we use the rsync --exclude option as shown below.

$ sudo rsync -avz --exclude 'dir3' source/ destination/

For remote sync, we can type command below.

$ sudo rsync -avz --exclude 'dir3' source/ bobby@192.168.122.105:destination/

output
sending incremental file list
created directory destination
./
file1.txt
file2.txt
dir1/
dir1/dir2/
dir1/dir2/file3.txt

sent 307 bytes  received 126 bytes  866.00 bytes/sec
total size is 0  speedup is 0.00

2) Exclude a specific file

If we don’t want to sync the file3.txt from the source to the destination folder, we type the command below.

$ sudo rm -rf destination

$ sudo rsync -avz --exclude 'dir1/dir2/file3.txt' source/ destination/

For remote sync, we can type command below.

$ sudo rsync -avz --exclude 'dir1/dir2/file3.txt' source/ bobby@192.168.122.105:destination/

output
sending incremental file list
created directory destination
./
file1.txt
file2.txt
dir1/
dir1/dir2/
dir3/
dir3/file4.txt

sent 338 bytes  received 130 bytes  936.00 bytes/sec
total size is 0  speedup is 0.00

For example in real time where you wish to rsync /var to a backup directory but you don't wish to include cache and tmp folder that usually don't hold important content between restarts you can use the following command:

$ sudo rsync -aAXhv --exclude={"/var/cache","/var/tmp"} /var remoteuser@192.168.122.105:destination/

3) Exclude a specific file type

To exclude a specific with specific extension, we use the appropriate pattern. For example, to exclude all the files that contains .txt as extension, we type the command below.

$ sudo rm -rf destination

$ sudo rsync -avz --exclude '*.txt' source/ destination/

For remote sync, we can type command below.

$ sudo rsync -avz --exclude '*.txt' source/ bobby@192.168.122.105:destination/

output
sending incremental file list
created directory destination
./
dir1/
dir1/dir2/
dir3/

sent 136 bytes  received 65 bytes  402.00 bytes/sec
total size is 0  speedup is 0.00

4) Exclude multiple directories that matches a pattern

We will exclude any directory (or subdirectories) under source/ that starts with "dir"

$ sudo rm -rf destination

$ sudo rsync -avz --exclude 'dir*' source/ destination/

For remote sync, we can type command below.

$ $ sudo rsync -avz --exclude 'dir*' source/ bobby@192.168.122.105:destination/

output
sending incremental file list
created directory destination
./
file1.txt
file2.txt

sent 168 bytes  received 91 bytes  518.00 bytes/sec
total size is 0  speedup is 0.00

5) Exclude multiple files and directories at the same time

First, create a text file with a list of all the files and directories you don’t want to backup. This is the list of files and directories you want to exclude from the rsync.

$ sudo nano exclude-list.txt

  file2.txt
  dir3/file4.txt

Next, execute the rsync using --exclude-from option with the exclude-list.txt as shown below.

$ sudo rm -rf destination

$ sudo rsync -avz --exclude-from 'exclude-list.txt' source/ destination/

For remote sync, we can type command below.

$ sudo rsync -avz --exclude-from 'exclude-list.txt' source/ bobby@192.168.122.105:destination/

output
sending incremental file list
created directory destination
./
file1.txt
dir1/
dir1/dir2/
dir1/dir2/file3.txt
dir3/

sent 277 bytes  received 111 bytes  776.00 bytes/sec
total size is 0  speedup is 0.00

6) Exclude path is always relative

The exclude option seems to have a full path (i.e /dir3/file4.txt). But, from rsync point of view, exclude path is always relative, and it will be treated as dir3/file4.txt. In the example below, rsync will look for dir3 under source directory (and not under / root directory).

$ sudo rm -rf destination

$ sudo rsync -avz --exclude '/dir3/file4.txt' source/ destination/

For remote sync, we can type command below.

$ sudo rsync -avz --exclude '/dir3/file4.txt' source/ bobby@192.168.122.105:destination/

output
sending incremental file list
created directory destination
./
file1.txt
file2.txt
dir1/
dir1/dir2/
dir1/dir2/file3.txt
dir3/

sent 334 bytes  received 130 bytes  928.00 bytes/sec
total size is 0  speedup is 0.00

It is similar to command below.

$ sudo rsync -avz --exclude 'dir3/file4.txt' source/ destination/

7) Exclude files from the command line

can also exclude files directly from the command line, this is useful when you have a smaller number of files to exclude. For example if we wish to sync /source to a destination directory but you don't wish to include file3.txt and file4.txt files, we can use the following command.

$ sudo rm -rf destination

$ sudo rsync -avz --exclude={"/dir1/dir2/file3.txt","/dir3/file4.txt"} source/ destinaton/

For remote sync, we can type command below.

$ sudo rsync -avz --exclude={"/dir1/dir2/file3.txt","/dir3/file4.txt"} source/ bobby@192.168.122.105:destination/

output
sending incremental file list
./
file1.txt
file2.txt
dir1/
dir1/dir2/
dir3/

sent 268 bytes  received 73 bytes  682.00 bytes/sec
total size is 0  speedup is 0.00

8) Exclude multiple files or folders

When we want to exclude type of files *.txt and specific folder dir3, we can type command below.

$ sudo rm -rf destination

$ sudo rsync -avz --exclude '*.txt' --exclude 'dir3' source/ destination/

For remote sync, we can type command below.

$ sudo rsync -avz --exclude '*.txt' --exclude 'dir3' source/ bobby@192.168.122.105:destination/

output
sending incremental file list
created directory destination
./
dir1/
dir1/dir2/

sent 109 bytes  received 61 bytes  340.00 bytes/sec
total size is 0  speedup is 0.00

9) Exclude Multiple Directories with Specific Pattern

We can also provide multiple directories with specific pattern for exclude. We will use wildcard * to complete dir directory name.

$ sudo rm -rf destination

$ sudo rsync -avz --exclude 'dir*' source/ destination/

For remote sync, we can type command below.

$ sudo rsync -avz --exclude 'dir*' source/ bobby@192.168.122.105:destination/

output
sending incremental file list
created directory destination
./
file1.txt
file2.txt

sent 168 bytes  received 91 bytes  518.00 bytes/sec
total size is 0  speedup is 0.00

10) Exclude Certain Size Files

We encounter during the backup process some large size files took really long time to copy. we have to copy some movies which are over 1GB in size. So, We think that it would be better if we could simply exclude the big files or any unimportant files in order to complete the backup as soon as possible.

For example, we want to exclude files bigger than 3 MB in size. All we have to do is use --max-size=SIZE option with rsync command. This option will not transfer any file larger than the specified size.

$ sudo rm -rf destination

$ sudo rsync -avz --max-size=3m source/ destination/

For remote sync, we can type command below.

$ sudo rsync -avz --max-size=3m source/ bobby@192.168.122.105:destination/

output
sending incremental file list
created directory destination
./
file1.txt
file2.txt
dir1/
dir1/dir2/
dir1/dir2/Bootstrap-tutorial.pdf
dir1/dir2/file3.txt
dir3/
dir3/file4.txt

sent 3,080,297 bytes  received 168 bytes  6,160,930.00 bytes/sec
total size is 8,101,480  speedup is 2.63

We can also use --min-size=SIZE to exclude any file smaller than the specified size.

for example, to exclude files which are smaller than 7 MB, run the following command.

$ sudo rsync -avz --min-size=7m source/ destination/

output
sending incremental file list
created directory destination
./
dir1/
dir1/dir2/
dir3/

sent 313 bytes  received 65 bytes  756.00 bytes/sec
total size is 8,101,480  speedup is 21,432.49

The suffixes of sizes are as follows:

"K" (or "KiB") is a kilobyte (1024)
"M" (or "MiB") is a megabyte (1024*1024)
"G" (or "GiB") is a gigabyte (1024*1024*1024)

If you want the multiplier to be 1000 instead of 1024, use "KB", "MB", or "GB" (Note: lower-case is also accepted for all values).