How to Synchronize Directories Using Lsyncd in Linux
In this article, I'll explain how to install Lsyncd(Live Syncing Mirror Daemon) on Linux systems and how to synchronize remote and local directories in Linux. Lsyncd is a light-weight live mirror solution that is comparatively easy to install without hampering existing local filesystem performance.
It is very useful to keep track of any data modification and sync those data between the directories which are frequently updated with new contents. By default, it is rsync only.
All custom configuration files are written in Lua language, this way powerful, flexible and simple configuration can be obtained. Lsyncd 2.2.1 requires rsync 3.1 on all source and target machines.
Install Lsyncd on RHEL/CentOS 7
In order to enable lsyncd on a CentOS 7.5 system, we will need to enable EPEL repository. You can simply run this command to install it.
#yum install epel-release #yum install lsyncd
You can confirm the installed version by running this command:
# lsyncd -version Version: 2.2.2
Lsyncd Configuration
Lsyncd configuration file is created automatically at /etc/lsyncd.conf on a RHEL/CentOS 7.5 system. By default, its contents look as below:
# cat /etc/lsyncd.conf ---- -- User configuration file for lsyncd. -- -- Simple example for default rsync, but executing moves through on the target. -- -- For more examples, see /usr/share/doc/lsyncd*/examples/
We need to modify this configuration file as per our purpose. As mentioned in this configuration file, all example scripts are available in the following location: /usr/share/doc/lsyncd-2.2.2/examples/
# cd /usr/share/doc/lsyncd-2.2.2/examples/ [root@li1050-94 examples]# ll total 40 -rw-r--r--. 1 root root 715 Feb 16 2017 lalarm.lua -rw-r--r--. 1 root root 1055 Feb 16 2017 lbash.lua -rw-r--r--. 1 root root 534 Feb 16 2017 lecho.lua -rw-r--r--. 1 root root 3376 Feb 16 2017 lftp.lua -rw-r--r--. 1 root root 2278 Feb 16 2017 lgforce.lua -rw-r--r--. 1 root root 2737 Feb 16 2017 limagemagic.lua -rw-r--r--. 1 root root 2770 Feb 16 2017 lpostcmd.lua -rw-r--r--. 1 root root 211 Feb 16 2017 lrsync.lua -rw-r--r--. 1 root root 204 Feb 16 2017 lrsyncssh.lua -rw-r--r--. 1 root root 4047 Feb 16 2017 lsayirc.lua
All these files are lsyncd example configuration files. Out of these files, we are explaining more on the usage of these files namely lrsync.lua and lrsyncssh.lua here. Let's see those example configuration files below:
Sample configuration for local sync:
# cat /usr/share/doc/lsyncd-2.2.2/examples/lrsync.lua
----
-- User configuration file for lsyncd.
--
-- Simple example for default rsync.
--
settings {
statusFile = "/tmp/lsyncd.stat",
statusInterval = 1,
}
sync{
default.rsync,
source="src",
target="trg",
}
Sample configuration for remote sync:
# cat /usr/share/doc/lsyncd-2.2.2/examples/lrsyncssh.lua
----
-- User configuration file for lsyncd.
--
-- Simple example for default rsync, but executing moves through on the target.
--
sync{default.rsyncssh, source="src", host="localhost", targetdir="dst/"}
To retain this sync process instead of running a cron job every X minutes, lsyncd uses linux kernel hooks to get notifications when any file within a directory has been changed. By default, it queues up any sync commands in 20 seconds. we can even modify this time interval as required with the --delay option along with your sync command.
sync {
default.rsyncssh,
source = "SRC",
target = "DEST",
delete = "running", -- prevents deletion of files on startup (ie when a server comes back online, don't delete files that are new on the backup)
delay = 5, -- run every 5 seconds instead of default 20
}
During the use of default.rsyncssh sync command it's always recommended to use --delete option to prevent missing of files from the destination folders. After creating or modifying the lsyncd config file, you must restart the lsyncd process.
Since Lsyncd uses rsync tool to copy, move and delete files from the source to the destination. We can make use of the rsync switches to prevent making unnecessary duplicates on the destination and smoothen this process. Some of the important rsync options are explained below:
--delete: This option ensures that any files in the remote directory that aren't in the source directory are deleted.
--times: This option is fairly important if you are going to be running this script over and over again, as it will keep the times between the two files in sync.
--force: This option allows the deletion of a non-empty directory in order to be replaced by an empty directory.
--links: This option is used to copy symlinks as symlinks.
--progress2: It results in showing the overall progress of the entire transfer, not just of the single file that is being copied.
--dry-run: This option do a trial run without actually performing any deletions or transfers but telling you what it will do. I highly recommend using this option the first time you run any rsync command after writing it.
--owner: ensures that owner user of the file is kept (not the permission level for the owner).
--group: ensures that group user of the file is kept (not the permission level for the group).
--perms: preserve permissions.
--sparse: ensures that sparse image files are transferred efficiently.
Synchronize local directories
Now let's see how to synchronize two local folders using lsyncd. We can create a source folder namely SRC_DIR and a target folder namely DEST_DIR to explain this process more vividly. Let's create the folders and add some files to the source directory to perform the synchronization.
# mkdir SRC_DIR *// Create source directory //*
# mkdir DEST_DIR *// Create target directory //*
# cd SRC_DIR/ *// Move to the source folder and create some random files //*
# touch file{1..10}
~/SRC_DIR# ll *// List out the Source folder contents //*
total 8
drwxr-xr-x 2 root root 4096 Aug 2 07:45 ./
drwx------ 7 root root 4096 Aug 2 07:46 ../
-rw-r--r-- 1 root root 0 Aug 2 07:45 file1
-rw-r--r-- 1 root root 0 Aug 2 07:45 file10
-rw-r--r-- 1 root root 0 Aug 2 07:45 file2
-rw-r--r-- 1 root root 0 Aug 2 07:45 file3
-rw-r--r-- 1 root root 0 Aug 2 07:45 file4
-rw-r--r-- 1 root root 0 Aug 2 07:45 file5
-rw-r--r-- 1 root root 0 Aug 2 07:45 file6
-rw-r--r-- 1 root root 0 Aug 2 07:45 file7
-rw-r--r-- 1 root root 0 Aug 2 07:45 file8
-rw-r--r-- 1 root root 0 Aug 2 07:45 file9
Next, you can create the lsyncd log files and status files to keep track of the process. These steps are optional. But I would recommend maintaining log files for all our tasks.
# mkdir /var/log/lsyncd
# touch /var/log/lsyncd/lsyncd.{log,status}
Now we need to modify our lsyncd configuration file to perform this local rsync. As we discussed before, the default lsyncd configuration file for a CentOS system is/etc/lsyncd.conf. We need to update these configuration files with our source and target directories and our log files.
# cat /etc/lsyncd.conf
----
-- User configuration file for lsyncd.
--
-- Simple example for default rsync, but executing moves through on the target.
--
-- For more examples, see /usr/share/doc/lsyncd*/examples/
--
settings {
logfile = "/var/log/lsyncd/lsyncd.log",
statusFile = "/var/log/lsyncd/lsyncd.status"
}
sync {
default.rsync,
source = "/root/SRC_DIR",
target = "/root/DEST_DIR",
}
You can replace the source and target directories path with your own values. Save and close the configuration file. Once it's done, restart and enable lsyncd service.
# systemctl enable lsyncd lsyncd.service is not a native service, redirecting to systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable lsyncd # systemctl start lsyncd
Now compare both source and target directories contents to confirm its working.
SRC_DIR]# ll total 0 -rw-r--r--. 1 root root 0 Aug 2 13:51 file1 -rw-r--r--. 1 root root 0 Aug 2 13:51 file10 -rw-r--r--. 1 root root 0 Aug 2 13:51 file2 -rw-r--r--. 1 root root 0 Aug 2 13:51 file3 -rw-r--r--. 1 root root 0 Aug 2 13:51 file4 -rw-r--r--. 1 root root 0 Aug 2 13:51 file5 -rw-r--r--. 1 root root 0 Aug 2 13:51 file6 -rw-r--r--. 1 root root 0 Aug 2 13:51 file7 -rw-r--r--. 1 root root 0 Aug 2 13:51 file8 -rw-r--r--. 1 root root 0 Aug 2 13:51 file9 DEST_DIR]# ll total 0 -rw-r--r--. 1 root root 0 Aug 2 13:51 file1 -rw-r--r--. 1 root root 0 Aug 2 13:51 file10 -rw-r--r--. 1 root root 0 Aug 2 13:51 file2 -rw-r--r--. 1 root root 0 Aug 2 13:51 file3 -rw-r--r--. 1 root root 0 Aug 2 13:51 file4 -rw-r--r--. 1 root root 0 Aug 2 13:51 file5 -rw-r--r--. 1 root root 0 Aug 2 13:51 file6 -rw-r--r--. 1 root root 0 Aug 2 13:51 file7 -rw-r--r--. 1 root root 0 Aug 2 13:51 file8 -rw-r--r--. 1 root root 0 Aug 2 13:51 file9
Hurray! The contents of source directory SRC_DIR have been successfully synchronized to target directory.
Furthermore, you can review the log and status files to verify the status of the replication to confirm whether it's completed or not.
# tail -10 /var/log/lsyncd/lsyncd.log Thu Aug 2 14:03:16 2018 Normal: --- Startup --- Thu Aug 2 14:03:16 2018 Normal: recursive startup rsync: /root/SRC_DIR/ -> /root/DEST_DIR/ Thu Aug 2 14:03:16 2018 Normal: Startup of /root/SRC_DIR/ -> /root/DEST_DIR/ finished. # more /var/log/lsyncd/lsyncd.status Lsyncd status report at Thu Aug 2 14:03:27 2018 Sync1 source=/root/SRC_DIR/ There are 0 delays Excluding: nothing. Inotify watching 1 directories 1: /root/SRC_DIR/
Synchronize multiple local folders
In order to sync multiple folders to one or more target directories, we will need to update the configuration file with more sync command statements with our required source and target directories.
sync{ default.rsync, source='source1', target='target1' }
sync{ default.rsync, source=' 'source2', target='target2' }
For example, please see my Lsyncd configuration file /etc/lsyncd.conf to sync two of my folders namely /root/SRC and /etc/nginx to a target directory located at /backup below:
----
-- User configuration file for lsyncd.
--
-- Simple example for default rsync.
--
settings {
logfile = "/var/log/lsyncd/lsyncd.log",
statusFile = "/var/log/lsyncd/lsyncd.status"
}
sync{ default.rsync, source='/root/SRC', target='/backup/SRC' }
sync{ default.rsync, source='/etc/nginx', target='/backup/Nginx_bkup' }
After updating the configuration file with the required changes, you can restart the lsyncd process to initiate the rsync process. Once done, you can verify the log files to confirm its status.
# tail /var/log/lsyncd/lsyncd.log Mon Aug 6 08:36:16 2018 Normal: recursive startup rsync: /root/SRC/ -> /backup/SRC/ Mon Aug 6 08:36:16 2018 Normal: recursive startup rsync: /etc/nginx/ -> /backup/Nginx_bkup/ Mon Aug 6 08:36:16 2018 Normal: Startup of "/root/SRC/" finished. Mon Aug 6 08:36:16 2018 Normal: Startup of "/etc/nginx/" finished.
Similarly, you can use this "default-rsync" sync command statements multiple times when you want to sync same source directories to multiple targets or multiple source directories to the same target directory.
Synchronize to remote directories
In order to initiate remote directory synchronization, we will need to set up passwordless SSH login. This will help Lsyncd to automatically replicate the contents of the local directory to a remote directory without user intervention. Since we are doing synchronization across two servers. we can take a source server and a target server to explain this process more clearly. Please see the SRC and DEST server IPs below:
SRC_ SERVER IP : 45.33.113.94 DEST_SERVER IP: 45.33.121.82
Step 1) Creating SSH keys on Source server for Passwordless login
We need to create an SSH key for the source server and copy the public key over to the target server to enhance the SSH connection between the servers during the account sync. You can generate the RSA keys using the command below:
#ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:T+p4P/5WTw/JA7B/B+uonDNF6KvqDpSl2frl4i3Ma8Q root@li1050-94 The key's randomart image is: +---[RSA 2048]----+ | | | . | | . + | | * o o . | | =.. S..o o + | | . .E +. o O o| | o+ o .o = *.| | o**..=.o . o| | =O*=+*B. | +----[SHA256]-----+
Step 2) Copying the public keys to the target server
Now copy the public keys over to the target server to enable password-less logins.
# ssh-copy-id root@45.33.121.82 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host '45.33.121.82 (45.33.121.82)' can't be established. ECDSA key fingerprint is SHA256:qI+CBEAw9MX+XfXQ1P0NmXVg0tBkWnmjeE0p1wWHzpM. ECDSA key fingerprint is MD5:62:d9:cc:a5:8b:7a:ef:fd:5e:b8:be:a2:75:3a:0c:20. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@45.33.121.82's password: Number of key(s) added: 1
Here, 45.33.121.82 is the destination server IP. Now we will be able to connect to our remote destination server from the source without any passwords.
# ssh root@45.33.121.82
Step 3) Creating a target directory for remote synchronization
Let's create a target directory namely Remote_Dir on the destination server. Once it's done you can logout the target server.
#mkdir Remote_Dir
Step 4) Modify the Lsyncd configuration file to enable Remote sync
You can back up the current lsyncd configuration file at /etc/lsyncd.conf and copy the sample lsyncd configuration file for remote sync located at /usr/share/doc/lsyncd-2.2.2/examples/lrsyncssh.lua to the main lsyncd configuration file /etc/lsyncd.conf. Once it's done you can edit the source directory, host and target directory accordingly in the configuration file. Please see my lsyncd configuration file for remote sync as per my setup below:
Copy the sample configuration to retain its proper syntax.
# cp /usr/share/doc/lsyncd-2.2.2/examples/lrsyncssh.lua /etc/lsyncd.conf
Edit the configuration file accordingly.
# cat /etc/lsyncd.conf
----
-- User configuration file for lsyncd.
--
-- Simple example for default rsync, but executing moves through on the target.
--
settings {
logfile = "/var/log/lsyncd/lsyncd.log",
statusFile = "/var/log/lsyncd/lsyncd.status"
}
sync{default.rsyncssh, source="/root/SRC_DIR", host="45.33.121.82", targetdir="/root/Remote_Dir"}
Step 5) Restart the lsyncd service on the source
Once the configuration file is proper edited and saved, you can restart the lsyncd service to make these changes effective. Now login to the remote server (DEST_SERVER) and confirm the contents on the remote Rsync directory namely /root/Remote_Dir.
#ssh root@45.33.121.82 #cd /root/Remote_Dir # ll total 8 drwxr-xr-x 2 root root 4096 Aug 2 14:08 ./ drwx------ 9 root root 4096 Aug 5 06:52 ../ -rw-r--r-- 1 root root 0 Aug 2 13:51 file1 -rw-r--r-- 1 root root 0 Aug 2 13:51 file10 -rw-r--r-- 1 root root 0 Aug 2 13:51 file2 -rw-r--r-- 1 root root 0 Aug 2 13:51 file3 -rw-r--r-- 1 root root 0 Aug 2 13:51 file4 -rw-r--r-- 1 root root 0 Aug 2 13:51 file5 -rw-r--r-- 1 root root 0 Aug 2 13:51 file6 -rw-r--r-- 1 root root 0 Aug 2 13:51 file7 -rw-r--r-- 1 root root 0 Aug 2 13:51 file8 -rw-r--r-- 1 root root 0 Aug 2 13:51 file9
Howdy! you can see all the files from the source directory on the local system to be replicated on the target directory in the destination server. You can even verify the successful completion of the rsync process viewing the lsyncd log files on the source.
# tail /var/log/lsyncd/lsyncd.log
Sun Aug 5 07:04:42 2018 Normal: --- Startup ---
Sun Aug 5 07:04:42 2018 Normal: recursive startup rsync: /root/SRC_DIR/ -> 45.33.121.82:/root/Remote_Dir/
Sun Aug 5 07:04:43 2018 Normal: Startup of "/root/SRC_DIR/" finished: 0
Sun Aug 5 07:13:48 2018 Normal: Rsyncing list
/
Sun Aug 5 07:13:49 2018 Normal: Finished (list): 0
Synchronize across multiple remote servers
Previously, we described how to sync a folder across a remote server. Similarly, you can use the default.rsyncssh command statements multiple times to sync the required source folders over multiple remote destination folders. But we need to make sure to enable password-less SSH login to replicate the contents of the local directories over to multiple remote directories without user intervention.
sync{default.rsyncssh, source="source1", host="host1", targetdir="target1"}
sync{default.rsyncssh, source="source2", host="host2", targetdir="target2}
I'll explain this process with a simple example of transferring the /etc/nginx folder from my SRC server to the remote servers DEST 1 and DEST 2.
The first step to this process will be generating an RSA key on my source server and copying its public keys over to my remote servers DEST 1 and DEST 2 as explained above. We need to repeat the same procedure from Step 1 to Step 3 for the two remote servers DEST 1 and DEST 2 to ensure password-less SSH login.
Secondly, we need to modify the lsyncd configuration file with multiple default.rsyncssh command statements with the required source, host and target folders. Please see my lsyncd configuration file for this sync process below:
# cat /etc/lsyncd.conf
----
-- User configuration file for lsyncd.
--
-- Simple example for default rsync, but executing moves through on the target.
--
settings {
logfile = "/var/log/lsyncd/lsyncd.log",
statusFile = "/var/log/lsyncd/lsyncd.status"
}
sync{default.rsyncssh, source="/etc/nginx", host="45.33.121.82", targetdir="/backup/nginx"}
sync{default.rsyncssh, source="/etc/nginx", host="45.33.113.94", targetdir="/backup/nginx"}
After making these configuration changes, you can restart the lsyncd service to start this sync process. Once, it's done you can verify the log files to confirm its status.
# tail /var/log/lsyncd/lsyncd.log Mon Aug 6 09:15:55 2018 Normal: recursive startup rsync: /etc/nginx/ -> 45.33.121.182:/backup/nginx/ Mon Aug 6 09:15:55 2018 Normal: recursive startup rsync: /etc/nginx/ -> 45.33.113.194:/backup/nginx/ Mon Aug 6 09:15:56 2018 Normal: Startup of "/etc/nginx/" finished: 0 Mon Aug 6 09:15:56 2018 Normal: Startup of "/etc/nginx/" finished: 0
Install Lsyncd on Debian/Ubuntu 18.04
On Debian and its derivatives like Ubuntu, Linux Mint etc, you can install lsyncd using the simple apt command as below:
#apt install lsyncd
You can confirm the installed lsyncd version using this command:
# lsyncd -version Version: 2.1.6
Configuration on Debian/Ubuntu
On Ubuntu based systems, it won't provide any default lsyncd configuration file. It is recommended to manually create those configuration files as per our purpose. We can get sample configuration files at the location./usr/share/doc/lsyncd/examples/ These sample example configuration files provide us with a basic idea of what/how it does the synchronization.
:/usr/share/doc/lsyncd/examples# ll total 48 drwxr-xr-x 2 root root 4096 Aug 2 07:34 ./ drwxr-xr-x 3 root root 4096 Aug 2 07:34 ../ -rw-r--r-- 1 root root 715 Oct 15 2015 lalarm.lua -rw-r--r-- 1 root root 1057 Oct 15 2015 lbash.lua -rw-r--r-- 1 root root 534 Oct 15 2015 lecho.lua -rw-r--r-- 1 root root 3376 Oct 15 2015 lftp.lua -rw-r--r-- 1 root root 2278 Oct 15 2015 lgforce.lua -rw-r--r-- 1 root root 2737 Oct 15 2015 limagemagic.lua -rw-r--r-- 1 root root 2770 Oct 15 2015 lpostcmd.lua -rw-r--r-- 1 root root 213 Oct 15 2015 lrsync.lua -rw-r--r-- 1 root root 204 Oct 15 2015 lrsyncssh.lua -rw-r--r-- 1 root root 4047 Oct 15 2015 lsayirc.lua
All these configuration files are written in Lua programming language. Please see the lsyncd sample configuration for a simple local rsync below:
:/usr/share/doc/lsyncd/examples# cat lrsync.lua
----
-- User configuration file for lsyncd.
--
-- Simple example for default rsync.
--
settings = {
statusFile = "/tmp/lsyncd.stat",
statusInterval = 1,
}
sync{
default.rsync,
source="src",
target="trg",
}
This procedure is exactly the same comparing to CentOS systems described above. Firstly, we will need to set up passwordless SSH login generating the RSA keys on the source server. This will ensure Lsyncd to automatically replicate the contents to a remote directory without user intervention. The key points to remember during remote sync comparing local sync is that we will need to change default.rsync to default.rsyncssh to enable rsync over ssh, and we should replace the "target" variable with the "host" and "targeted" variables. In addition, we will need to maintain the configuration file location as /etc/lsyncd/lsyncd.conf.lua on Ubuntu/Debian systems. Rest of the lsyncd configuration procedures are the same in all Linux Sytems.
Subscribe to Article
Print Article
Email Article to Friend
Export to PDF
Export to MS Word