10 Linux rsync Examples to Exclude Files/Directories

Rsync (Remote Sync) is a most commonly used command for copying and synchronizing files and directories remotely as well as locally in Linux/Unix systems. With the help of rsync command you can copy and synchronize your data remotely and locally across directories, across disks and networks, perform data backups and mirroring between two Linux machines.

Rsync is one of the most useful utilities for a server administrator, but it syncs everything by default, which can be annoying if your application creates a lot of temporary files. Here’s how to exclude files when using rsync.

Important features of rsync command

  • Speed : First time, rsync replicates the whole content between the source and destination directories. Next time, rsync transfers only the changed blocks or bytes to the destination location, which makes the transfer really fast.
  • Security : rsync allows encryption of data using ssh protocol during transfer.
  • Low Bandwidth: rsync uses compression and decompression of data block by block at the sending and receiving end respectively. So the bandwidth used by rsync will be always less compared to other file transfer protocols like FTP.
  • Privileges : No special privileges are required to install and execute rsync.

The basic syntax of rsync is very straight forward, and operates in a way that is similar to ssh, scp, and cp.

$ sudo rsync options source destination
  • -v : verbose.
  • -r : copies data recursively (but don’t preserve timestamps and permission while transferring data).
  • -a : archive mode, archive mode allows copying files recursively and it also preserves symbolic links, file permissions, user & group ownerships and timestamps.
  • -z : compress file data.
  • -h : human-readable, output numbers in a human-readable format.
  • -d : transfer directories without recursing -e : specify the ssh as remote shell.

First, create a sample directory structure as shown below (with some empty files) that can be used for testing purpose.

  $ cd ~
  $ sudo mkdir -p source/dir1/dir2
  $ sudo mkdir -p source/dir3
  $ sudo touch source/file1.txt
  $ sudo touch source/file2.txt
  $ sudo touch source/dir1/dir2/file3.txt
  $ sudo touch source/dir3/file4.txt

The above commands will create a source directory (under home directory) with the following structure,To see the structure of source directory type below command.

 
$ sudo find source/
  source
  - file1.txt
  - file2.txt
  - dir1
    - dir2
      - file3.txt
  - dir3
    - file4.txt

1) Exclude a specific directory

Frist we remove destination directory if it exist by typing below command.

$ sudo rm -rf destination

If we don’t want to sync the dir3 and its subdirectories from the source to the destination folder, we use the rsync --exclude option as shown below.

$ sudo rsync -avz --exclude 'dir3' source/ destination/

For remote sync, we can type command below.

$ sudo rsync -avz --exclude 'dir3' source/ bobby@192.168.122.105:destination/
output
sending incremental file list
created directory destination
./
file1.txt
file2.txt
dir1/
dir1/dir2/
dir1/dir2/file3.txt

sent 307 bytes  received 126 bytes  866.00 bytes/sec
total size is 0  speedup is 0.00

2) Exclude a specific file

If we don’t want to sync the file3.txt from the source to the destination folder, we type the command below.

$ sudo rm -rf destination
$ sudo rsync -avz --exclude 'dir1/dir2/file3.txt' source/ destination/

For remote sync, we can type command below.

$ sudo rsync -avz --exclude 'dir1/dir2/file3.txt' source/ bobby@192.168.122.105:destination/
output
sending incremental file list
created directory destination
./
file1.txt
file2.txt
dir1/
dir1/dir2/
dir3/
dir3/file4.txt

sent 338 bytes  received 130 bytes  936.00 bytes/sec
total size is 0  speedup is 0.00

For example in real time where you wish to rsync /var to a backup directory but you don't wish to include cache and tmp folder that usually don't hold important content between restarts you can use the following command:

$ sudo rsync -aAXhv --exclude={"/var/cache","/var/tmp"} /var remoteuser@192.168.122.105:destination/

3) Exclude a specific file type

To exclude a specific with specific extension, we use the appropriate pattern. For example, to exclude all the files that contains .txt as extension, we type the command below.

$ sudo rm -rf destination
$ sudo rsync -avz --exclude '*.txt' source/ destination/

For remote sync, we can type command below.

$ sudo rsync -avz --exclude '*.txt' source/ bobby@192.168.122.105:destination/
output
sending incremental file list
created directory destination
./
dir1/
dir1/dir2/
dir3/

sent 136 bytes  received 65 bytes  402.00 bytes/sec
total size is 0  speedup is 0.00

4) Exclude multiple directories that matches a pattern

We will exclude any directory (or subdirectories) under source/ that starts with "dir"

$ sudo rm -rf destination
$ sudo rsync -avz --exclude 'dir*' source/ destination/

For remote sync, we can type command below.

$ $ sudo rsync -avz --exclude 'dir*' source/ bobby@192.168.122.105:destination/
output
sending incremental file list
created directory destination
./
file1.txt
file2.txt

sent 168 bytes  received 91 bytes  518.00 bytes/sec
total size is 0  speedup is 0.00

5) Exclude multiple files and directories at the same time

First, create a text file with a list of all the files and directories you don’t want to backup. This is the list of files and directories you want to exclude from the rsync.

$ sudo nano exclude-list.txt
  file2.txt
  dir3/file4.txt

Next, execute the rsync using --exclude-from option with the exclude-list.txt as shown below.

$ sudo rm -rf destination
$ sudo rsync -avz --exclude-from 'exclude-list.txt' source/ destination/

For remote sync, we can type command below.

$ sudo rsync -avz --exclude-from 'exclude-list.txt' source/ bobby@192.168.122.105:destination/
output
sending incremental file list
created directory destination
./
file1.txt
dir1/
dir1/dir2/
dir1/dir2/file3.txt
dir3/

sent 277 bytes  received 111 bytes  776.00 bytes/sec
total size is 0  speedup is 0.00

6) Exclude path is always relative

The exclude option seems to have a full path (i.e /dir3/file4.txt). But, from rsync point of view, exclude path is always relative, and it will be treated as dir3/file4.txt. In the example below, rsync will look for dir3 under source directory (and not under / root directory).

$ sudo rm -rf destination
$ sudo rsync -avz --exclude '/dir3/file4.txt' source/ destination/

For remote sync, we can type command below.

$ sudo rsync -avz --exclude '/dir3/file4.txt' source/ bobby@192.168.122.105:destination/
output
sending incremental file list
created directory destination
./
file1.txt
file2.txt
dir1/
dir1/dir2/
dir1/dir2/file3.txt
dir3/

sent 334 bytes  received 130 bytes  928.00 bytes/sec
total size is 0  speedup is 0.00

It is similar to command below.

$ sudo rsync -avz --exclude 'dir3/file4.txt' source/ destination/

7) Exclude files from the command line

can also exclude files directly from the command line, this is useful when you have a smaller number of files to exclude. For example if we wish to sync /source to a destination directory but you don't wish to include file3.txt and file4.txt files, we can use the following command.

$ sudo rm -rf destination
$ sudo rsync -avz --exclude={"/dir1/dir2/file3.txt","/dir3/file4.txt"} source/ destinaton/

For remote sync, we can type command below.

$ sudo rsync -avz --exclude={"/dir1/dir2/file3.txt","/dir3/file4.txt"} source/ bobby@192.168.122.105:destination/
output
sending incremental file list
./
file1.txt
file2.txt
dir1/
dir1/dir2/
dir3/

sent 268 bytes  received 73 bytes  682.00 bytes/sec
total size is 0  speedup is 0.00

8) Exclude multiple files or folders

When we want to exclude type of files *.txt and specific folder dir3, we can type command below.

$ sudo rm -rf destination
$ sudo rsync -avz --exclude '*.txt' --exclude 'dir3' source/ destination/

For remote sync, we can type command below.

$ sudo rsync -avz --exclude '*.txt' --exclude 'dir3' source/ bobby@192.168.122.105:destination/
output
sending incremental file list
created directory destination
./
dir1/
dir1/dir2/

sent 109 bytes  received 61 bytes  340.00 bytes/sec
total size is 0  speedup is 0.00

9) Exclude Multiple Directories with Specific Pattern

We can also provide multiple directories with specific pattern for exclude. We will use wildcard * to complete dir directory name.

$ sudo rm -rf destination
$ sudo rsync -avz --exclude 'dir*' source/ destination/

For remote sync, we can type command below.

$ sudo rsync -avz --exclude 'dir*' source/ bobby@192.168.122.105:destination/
output
sending incremental file list
created directory destination
./
file1.txt
file2.txt

sent 168 bytes  received 91 bytes  518.00 bytes/sec
total size is 0  speedup is 0.00

10) Exclude Certain Size Files

We encounter during the backup process some large size files took really long time to copy. we have to copy some movies which are over 1GB in size. So, We think that it would be better if we could simply exclude the big files or any unimportant files in order to complete the backup as soon as possible.

For example, we want to exclude files bigger than 3 MB in size. All we have to do is use --max-size=SIZE option with rsync command. This option will not transfer any file larger than the specified size.

$ sudo rm -rf destination
$ sudo rsync -avz --max-size=3m source/ destination/

For remote sync, we can type command below.

$ sudo rsync -avz --max-size=3m source/ bobby@192.168.122.105:destination/
output
sending incremental file list
created directory destination
./
file1.txt
file2.txt
dir1/
dir1/dir2/
dir1/dir2/Bootstrap-tutorial.pdf
dir1/dir2/file3.txt
dir3/
dir3/file4.txt

sent 3,080,297 bytes  received 168 bytes  6,160,930.00 bytes/sec
total size is 8,101,480  speedup is 2.63

We can also use --min-size=SIZE to exclude any file smaller than the specified size.

for example, to exclude files which are smaller than 7 MB, run the following command.

$ sudo rsync -avz --min-size=7m source/ destination/
output
sending incremental file list
created directory destination
./
dir1/
dir1/dir2/
dir3/

sent 313 bytes  received 65 bytes  756.00 bytes/sec
total size is 8,101,480  speedup is 21,432.49

The suffixes of sizes are as follows:

  • "K" (or "KiB") is a kilobyte (1024)
  • "M" (or "MiB") is a megabyte (1024*1024)
  • "G" (or "GiB") is a gigabyte (1024*1024*1024)

If you want the multiplier to be 1000 instead of 1024, use "KB", "MB", or "GB" (Note: lower-case is also accepted for all values).

Read Also:

0 (0)
Article Rating (No Votes)
Rate this article
Attachments
There are no attachments for this article.
Comments
There are no comments for this article. Be the first to post a comment.
Full Name
Email Address
Security Code Security Code
Related Articles RSS Feed
Installing and Configuring stunnel on CentOS 6
Viewed 1925 times since Fri, Sep 28, 2018
10 Linux cryptsetup Examples for LUKS Key Management (How to Add, Remove, Change, Reset LUKS encryption Key)
Viewed 2557 times since Tue, Jul 31, 2018
Using grep to find string in files
Viewed 741 times since Fri, May 15, 2020
RHEL: Back-up/Replicate a partition table
Viewed 1793 times since Sun, May 27, 2018
SSH ProxyCommand example: Going through one host to reach another server
Viewed 10015 times since Tue, Aug 6, 2019
Improve security with polyinstantiation
Viewed 10665 times since Fri, May 15, 2020
O’Reilly’s CD bookshelf
Viewed 7606 times since Wed, Jun 27, 2018
HowTo: Kill TCP Connections in CLOSE_WAIT State
Viewed 6580 times since Thu, Feb 14, 2019
KONTO SFTP Z CHROOTEM Z UŻYCIEM OPENSSH-SERVER NA CENTOS/RHEL6
Viewed 804 times since Fri, Nov 30, 2018
An easier way to manage disk decryption at boot with Red Hat Enterprise Linux 7.5 using NBDE
Viewed 4072 times since Mon, Aug 6, 2018