6 rsync Examples to Exclude Multiple Files and Directories using exclude-from
Rsync is very powerful tool to take backups, or sync files and directories between two different locations (or servers).
You know this already, as we presented you with practical examples on rsync earlier.
In a typical backup situation, you might want to exclude one or more files (or directories) from the backup. You might also want to exclude a specific file type from rsync.
This article explains how to ignore multiple files and/or directories during rsync with examples.
First, create a sample directory structure as shown below (with some empty files) that can be used for testing purpose.
$ cd ~ $ mkdir -p source/dir1/dir2 $ mkdir -p source/dir3 $ touch source/file1.txt $ touch source/file2.txt $ touch source/dir1/dir2/file3.txt $ touch source/dir3/file4.txt
The above command will create a source directory (under your home directory) with the following structure.
source - file1.txt - file2.txt - dir1 - dir2 - file3.txt - dir3 - file4.txt
1. Exclude a specific directory
If you don’t want to sync the dir1 (including all it’s subdirectories) from the source to the destination folder, use the rsync –exclude option as shown below.
$ rm -rf destination $ rsync -avz --exclude 'dir1' source/ destination/ building file list ... done created directory dest ./ file1.txt file2.txt dir3/ dir3/file4.txt
Verify to make sure dir1 is not copied from source directory to destination directory.
$ find destination destination destination/file2.txt destination/file1.txt destination/dir3 destination/dir3/file4.txt
2. Exclude multiple directories that matches a pattern
The following example will exclude any directory (or subdirectories) under source/ that matches the pattern “dir*”
$ rm -rf destination $ rsync -avz --exclude 'dir*' source/ destination/ building file list ... done created directory destination ./ file1.txt file2.txt
Verify the destination directory to make sure it didn’t copy any directories that has the keyword “dir” in it.
$ find destination destination destination/file2.txt destination/file1.txt
3. Exclude a specific file
To exclude a specific file, use the relative path of the file in the exclude option as shown below.
$ rm -rf destination $ rsync -avz --exclude 'dir1/dir2/file3.txt' source/ destination/ building file list ... done created directory destination ./ file1.txt file2.txt dir1/ dir1/dir2/ dir3/ dir3/file4.txt
Verify the destination directory to make sure it didn’t copy the specific file ( dir1/dir2/file3.txt in this example).
$ find destination destination destination/file2.txt destination/file1.txt destination/dir1 destination/dir1/dir2 destination/dir3 destination/dir3/file4.txt
4. Exclude path is always relative
If you are not careful, you might make this mistake.
In the following example, the exclude option seems to have a full path (i.e /dir1/dir2/file3.txt). But, from rsync point of view, exclude path is always relative, and it will be treated as dir1/dir2/file3.txt. In the example below, rsync will look for dir1 under source directory (and not under / root directory).
$ rsync -avz --exclude '/dir1/dir2/file3.txt' source/ destination/
So, the above command is exactly same as the following. Just to avoid confusion (and to make it easy to read), don’t give / in front of the exclude path.
$ rsync -avz --exclude 'dir1/dir2/file3.txt' source/ destination/
5. Exclude a specific file type
To exclude a specific file type that has a specific extension, use the appropriate pattern. For example, to exclude all the files that contains .txt as extension, do the following.
$ rsync -avz --exclude '*.txt' source/ destination/ building file list ... done created directory destination ./ dir1/ dir1/dir2/ dir3/
Verify the destination directory to make sure it didn’t copy the *.txt files.
$ find destination destination destination/dir1 destination/dir1/dir2 destination/dir3
Note: The above is very helpful, when you want to backup your home directory, but exclude all those huge image and video files that has a specific file extension.
6. Exclude multiple files and directories at the same time
When you want to exclude multiple files and directories, you can always specify multiple rsync exclude options in the command line as shown below.
$ rsync -avz --exclude file1.txt --exclude dir3/file4.txt source/ destination/
Wait. What if I had tons of files that I want to exclude from rsync?
I can’t keep adding them in the command line using multiple –exclude, which is hard to read, and hard to re-use the rsync command for later.
So, the better way is to use rsync –exclude-from option as shown below, where you can list all the files (and directories) you want to exclude in a file.
First, create a text file with a list of all the files and directories you don’t want to backup. This is the list of files and directories you want to exclude from the rsync.
$ vim exclude-list.txt file1.txt dir3/file4.txt
Next, execute the rsync using –exclude-from option with the exclude-list.txt as shown below.
$ rm -rf destination $ rsync -avz --exclude-from 'exclude-list.txt' source/ destination/ building file list ... done created directory destination ./ file2.txt dir1/ dir1/dir2/ dir1/dir2/file3.txt dir3/
Verify the desitination directory to make sure the files and directories listed in the exclude-list.txt file is not backed-up.
$ find destination destination destination/file2.txt destination/dir1 destination/dir1/dir2 destination/dir1/dir2/file3.txt destination/dir3
OK I feel really dumb. Before I even posted this question my friend showed me how to do it and it is really simple. To exclude multiple directories you just use multiple --exclude=path
switches. So my command above properly written is as follows:
rsync -arv --exclude=.ccache --exclude=build /home/ben /media/ben/thumbdrive/
When having multiple directories and/or files to exclude, make a text file and use the --exclude-from
switch. Make a file called exclude_me.txt
and in it list your exclusions. Example (/home/ben/exclude_me.txt
):
.ccache
build
.java
.gvfs
.xsession-errors
Then your rsync would look something like:
rsync -arv --exclude-from='/home/ben/exclude_me.txt' /home/ben /media/ben/thumbdrive/
This is some information on filter rules that may help:
/dir/
means exclude the root folder/dir
/dir/*
means get the root folder/dir
but not the contents-
dir/
means exclude any folder anywhere where the name containsdir/
-
Examples excluded:
/dir/, /usr/share/directory/, /var/spool/dir/ /var/spool/lpd/cf
means skip files that start withcf
within any folder within/var/spool/lpd
-
You can also exclude multiple paths within curly braces:
rsync -arv --exclude={.ccache,build} /home/ben /media/ben/thumbdrive/