Secure Remote Logging to Central Log Server Using RSYSLOG on CentOS 6 / CentOS 7 and stunnel

Secure Remote Logging to Central Log Server Using RSYSLOG on CentOS 6 / CentOS 7 and stunnel

 
Secure Remote Logging to Central Log Server via Encrypted stunnel Tunneling
Here is how I set up a secure remote rsyslog server on my VPS that accepts and records system logs from various clients on my home and cloud hobbyist networks.

The remote syslog server runs rsyslog daemon on CentOS 7, and the clients are a mixture of numerous operating systems, including Linux CentOS 6 and CentOS7, Windows, BSD, Routers running DD-WRT etc.

The basic idea is that local rsyslog daemons forward logs to stunnel also running locally. stunnel then forwards them to the remote server using a secure tunnel over TCP/IP. A peer stunnel daemon instance running on the remote server then forwards the incoming logs to the local rsyslog daemon on the remote server.

Server setup (CentOS 7)

In my setup, the server-side stunnel listens on port 65515 and forwards logs to rsyslog which listens on port 65514.

Install stunnel and add a script to start and stop it using the systemctl facility.

# yum install openssl stunnel

Add a file /usr/lib/systemd/system/stunnel.service with the following contents:

[Unit]
Description=SSL tunnel for network daemons
After=syslog.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/stunnel /etc/stunnel/stunnel.conf
ExecStop=/bin/kill -9 $(pgrep stunnel)

[Install]
WantedBy=multi-user.target




Add a file /etc/stunnel/stunnel.conf containing the following:

client = no
fips = no
cert = /etc/stunnel/stunnel.pem
[rsyslog]
accept = 65515
connect = 127.0.0.1:65514
 
Generate the server's certificate. The following will generate a certificate with an expiry of 100 years. Important: enter the fully qualified host-name of the server for the question "Common Name". Answer the questions the best you can or want to. I like to put in meaningful information for all the questions, including my real email address for the last question.
 
# cd /etc/stunnel
# openssl req -new -x509 -days 36500 -nodes -out stunnel.pem -keyout stunnel.pem
Generating a 2048 bit RSA private key
.............................+++
.+++
writing new private key to 'stunnel.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:MD
Locality Name (eg, city) [Default City]:Germantown
Organization Name (eg, company) [Default Company Ltd]:Supratim Sanyal's Hobbyist Lab
Organizational Unit Name (eg, section) []:Basement Server
Common Name (eg, your name or your server's hostname) []:sanyalnet-cloud-vps2.freeddns.org
Email Address []: youremailhere@emaildomain.com
 
You should now see the newly generated certificate in the /etc/stunnel directory:
 
# ls -lrt
total 8
-rw------- 1 root root  105 Oct 26 20:54 stunnel.conf
-rw------- 1 root root 3278 Oct 26 23:04 stunnel.pem
 
Now configure the rsyslog system log daemon on this remote syslog server to listen to logs at the port indicated by connect line in the stunnel config file which you created above. Edit the file /etc/rsyslog.conf and look for, uncomment and adjust the following two lines (they were at around line 18 on my installation of CentOS 7):
 
# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 65514
 
The last configuration step is to open up the stunnel listening port to the external internet on the firewall. We also need rules to allow all traffic on the localhost (127.0.0.1) interface "lo" so that network traffic between stunnel and rsyslog flows internally over localhost. We then add firewall reject rules to reject everything else.
 

# iptables -A INPUT -i lo -j ACCEPT

# iptables -A OUTPUT -o lo -j ACCEPT
# iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 65515 -j ACCEPT
# iptables -A INPUT -j REJECT --reject-with icmp-host-prohibited

# iptables -A FORWARD -j REJECT --reject-with icmp-host-prohibited

 
Of course, to make these rules permanent, I recommend you edit and add them to the default iptables startup rules file /etc/sysconfig/iptables, adjusting for any existing other rules you may already have. In particular, for security, make sure you do not open up the local rsyslog listening port 65514 on your firewall; this is important since rsyslog binds to all interfaces and there is no easy way to force it to bind to the localhost (127.0.0.1) interface.
 
We are done configuring the rsyslog server, and can now enable and start up stunnel, and restart rsyslog, for the changes to take effect.
 
# systemctl enable stunnel
Created symlink from /etc/systemd/system/multi-user.target.wants/stunnel.service to /usr/lib/systemd/system/stunnel.service.
# systemctl start stunnel
# systemctl restart rsyslog

Perform some basic validations to check if everything is working as expected. Verify the stunnel daemon is up and running:
 
# systemctl status stunnel
 stunnel.service - SSL tunnel for network daemons
   Loaded: loaded (/usr/lib/systemd/system/stunnel.service; enabled; vendor preset: disabled)
   Active: active (exited) since Wed 2016-10-26 22:57:49 UTC; 18min ago
 Main PID: 675 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/stunnel.service
           ├─696 /bin/stunnel /etc/stunnel/stunnel.conf
           ├─697 /bin/stunnel /etc/stunnel/stunnel.conf
           ├─698 /bin/stunnel /etc/stunnel/stunnel.conf
           ├─699 /bin/stunnel /etc/stunnel/stunnel.conf
           ├─700 /bin/stunnel /etc/stunnel/stunnel.conf
           └─701 /bin/stunnel /etc/stunnel/stunnel.conf
 
Oct 26 22:57:48 sanyalnet-cloud-vps2.freeddns.org systemd[1]: Starting SSL tunnel for network daemons...
Oct 26 22:57:49 sanyalnet-cloud-vps2.freeddns.org systemd[1]: Started SSL tunnel for network daemons.
Oct 26 23:16:00 sanyalnet-cloud-vps2.freeddns.org systemd[1]: Started SSL tunnel for network daemons.
 
Verify rsyslog is working with the changed configuration:
 
# systemctl status rsyslog
 rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2016-10-26 23:16:04 UTC; 2min 6s ago
 Main PID: 2756 (rsyslogd)
   CGroup: /system.slice/rsyslog.service
           └─2756 /usr/sbin/rsyslogd -n
 
Oct 26 23:16:04 sanyalnet-cloud-vps2.freeddns.org systemd[1]: Starting System Logging Service...
Oct 26 23:16:04 sanyalnet-cloud-vps2.freeddns.org systemd[1]: Started System Logging Service.
 
Verify that stunnel's startup activity is logged in the /var/log/secure logfile:
 
[root@sanyalnet-cloud-vps2 ~]# cat /var/log/secure
...
...
...

Oct 26 22:57:49 sanyalnet-cloud-vps2 stunnel: LOG5[675:140205220452416]: stunnel 4.56 on x86_64-redhat-linux-gnu platform

Oct 26 22:57:49 sanyalnet-cloud-vps2 stunnel: LOG5[675:140205220452416]: Compiled/running with OpenSSL 1.0.1e-fips 11 Feb 2013

Oct 26 22:57:49 sanyalnet-cloud-vps2 stunnel: LOG5[675:140205220452416]: Threading:PTHREAD Sockets:POLL,IPv6 SSL:ENGINE,OCSP,FIPS Auth:LIBWRAP
Oct 26 22:57:49 sanyalnet-cloud-vps2 stunnel: LOG5[675:140205220452416]: Reading configuration from file /etc/stunnel/stunnel.conf
Oct 26 22:57:49 sanyalnet-cloud-vps2 stunnel: LOG5[675:140205220452416]: FIPS mode is disabled
Oct 26 22:57:49 sanyalnet-cloud-vps2 stunnel: LOG5[675:140205220452416]: Configuration successful
 
Verify the correct server network ports are open:
 
# netstat -pan | egrep "stunnel|rsyslog"
tcp        0      0 0.0.0.0:65514           0.0.0.0:*               LISTEN      2756/rsyslogd
tcp        0      0 0.0.0.0:65515           0.0.0.0:*               LISTEN      701/stunnel
tcp        0      0 127.0.0.1:65514         127.0.0.1:42938         ESTABLISHED 2756/rsyslogd
tcp        0      0 127.0.0.1:42938         127.0.0.1:65514         ESTABLISHED 701/stunnel
...
...
 

Client setup (CentOS 6)

In this example client running CentOS 6, we will configure the rsyslog daemon to forward all logs to the remote syslog server created above over a secure encrypted network link implemented by stunnel.
 
Install stunnel and add a script to start and stop it using the CentOS 6 service management facility.

# yum install openssl stunnel
 
Add a file /etc/init.d/stunnel with the following contents:
 
#!/bin/sh
#
# $Id: stunnel.init.in,v 1.0 2016/10/26 21:11:58 mschimek Exp $
# Startup script for the stunnel daemon, RedHat style.
# See http://supratim-sanyal.blogspot.com/2016/10/secure-remote-logging-to-central-log.html
#
# chkconfig: - 90 10
# description: stunnel daemon to establish encrypted secure network channel between two hosts
#
# Supratim Sanyal <supratim at riseup dot net>
#
 
. /etc/init.d/functions
 
prefix=/usr
exec_prefix=/usr
 
[ -x /usr/bin/stunnel ]  || exit 1
 
RETVAL=0
 
start(){
        echo -n $"Starting stunnel secure sockets tunnel daemon: "
        daemon stunnel
        RETVAL=$?
        echo
        touch /var/lock/subsys/stunnel
        return $RETVAL
}
 
stop(){
        echo -n $"Stopping stunnel secure sockets tunnel daemon: "
        killproc stunnel
        echo
        RETVAL=$?
        rm -f /var/lock/subsys/stunnel
        return $RETVAL
}
 
restart(){
        stop
        start
}
 
case "$1" in
        start)
                start
                ;;
        stop)
                stop
                ;;
        restart)
                restart
                ;;
        status)
                status stunnel
                ;;
        condrestart)
                [ -e /var/lock/subsys/stunnel ] && restart
                ;;
        *)
                echo $"Usage: $0 {start|stop|status|restart|condrestart}"
                RETVAL=1
esac
 
exit $RETVAL
 
 
Add the stunnel init script to the list of chkconfig managed init scripts, make it executable, and enable it, but do not start it yet.
 
# chmod +x /etc/init.d/stunnel
# chkconfig --add stunnel
# chkconfig stunnel on
# chkconfig --list stunnel
stunnel         0:off   1:off   2:on    3:on    4:on    5:on    6:off
# service stunnel status
stunnel is stopped
 
Add a file /etc/stunnel/stunnel.conf with the following contents, replacing <rsyslog-remote-server-hostname> with your actual fully qualified network-reachable remote syslog server hostname (or IP address):
 
client = yes
fips = no
[rsyslogd]
accept = 127.0.0.1:65514
connect = <rsyslog-remote-server-hostname>:65515
 
Note: you do not need a certificate on stunnel client side.
 
Edit the file /etc/rsyslog.conf and add the following at the bottom. If, however, you already have an existing forwarding rule for some other syslog server, insert the following lines below the existing $WorkDirectory but above the existing forwarding rule-sets.
 
$WorkDirectory /var/log # where to place spool files
# begin forwarding rule to stunnel port for passing on to VPS (see /etc/stunnel/stunnel.conf)
$ActionQueueType LinkedList
$ActionQueueFileName logs-spooled-4-VPS
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on
*.* @@127.0.0.1:65514
# ### end of the forwarding rule ###
 
 
Configuration is complete at this point. Start / restart the necessary daemons.
 
# service restart stunnel
# service restart rsyslog
 

Client setup (CentOS 7)

On a client running CentOS 7, we will configure the rsyslog daemon to forward all logs to the remote syslog server created above over a secure encrypted network link implemented by stunnel.
 
Install stunnel and add a script to start and stop it using the systemctl facility.

# yum install openssl stunnel

Add a file /usr/lib/systemd/system/stunnel.service with the following contents:

[Unit]
Description=SSL tunnel for network daemons
After=syslog.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/stunnel /etc/stunnel/stunnel.conf
ExecStop=/bin/kill -9 $(pgrep stunnel)

[Install]
WantedBy=multi-user.target



Enable the stunnel daemon (but do not start it yet):

# systemctl enable stunnel
Created symlink from /etc/systemd/system/multi-user.target.wants/stunnel.service to /usr/lib/systemd/system/stunnel.service.
 

Add a file /etc/stunnel/stunnel.conf with the following contents, replacing <rsyslog-remote-server-hostname> with your actual fully qualified network-reachable remote syslog server hostname (or IP address):
 
client = yes
fips = no
[rsyslogd]
accept = 127.0.0.1:65514
connect = <rsyslog-remote-server-hostname>:65515
 
Note: you do not need a certificate on stunnel client side.
 
Edit the file /etc/rsyslog.conf and add the following at the bottom. If, however, you already have an existing forwarding rule for some other syslog server, insert the following lines below the existing $WorkDirectory but above the existing forwarding rule-sets.
 
$WorkDirectory /var/log # where to place spool files
# begin forwarding rule to stunnel port for passing on to VPS (see /etc/stunnel/stunnel.conf)
$ActionQueueType LinkedList
$ActionQueueFileName logs-spooled-4-VPS
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on
*.* @@127.0.0.1:65514
# ### end of the forwarding rule ###
 
 
Configuration is complete at this point. Start / restart the necessary daemons.

# systemctl start stunnel
# systemctl restart rsyslog
 
 

Testing

Perform some basic validations to check if everything on the rsyslog and stunnel daemons on the client side are working as expected. Verify the stunnel and rsyslog daemons are up and running:
 
# service stunnel status
stunnel (pid  13781) is running...
# service rsyslog status
rsyslogd (pid  11285) is running...
 
Verify stunnel has connected at both the rsyslog port and peer stunnel on the remote server's port:
 
# cat /var/log/secure
Oct 26 20:57:53 dormarth stunnel: LOG5[10866:139787722659776]: stunnel 4.29 on x86_64-redhat-linux-gnu with OpenSSL 1.0.1e-fips 11 Feb 2013
Oct 26 20:57:53 dormarth stunnel: LOG5[10866:139787722659776]: Threading:PTHREAD SSL:ENGINE,FIPS Sockets:POLL,IPv6 Auth:LIBWRAP
Oct 26 20:57:53 dormarth stunnel: LOG5[10866:139787722659776]: 500 clients allowed
Oct 26 20:58:00 dormarth stunnel: LOG5[10872:139787722655488]: rsyslogd accepted connection from 127.0.0.1:44734
Oct 26 20:58:00 dormarth stunnel: LOG5[10872:139787722655488]: connect_blocking: connected 64.137.228.122:65515
Oct 26 20:58:00 dormarth stunnel: LOG5[10872:139787722655488]: rsyslogd connected remote server from 10.42.2.2:46658
 
Verify the network ports that you expecting are indeed in use:
 
# netstat -pan | egrep "rsyslog|stunnel"
tcp        0      0 127.0.0.1:65514             0.0.0.0:*                   LISTEN      13781/stunnel
tcp        0      0 127.0.0.1:49450             127.0.0.1:65514             ESTABLISHED 11285/rsyslogd
tcp        0      1 10.42.2.2:58520             64.137.228.85:65515         SYN_SENT    13781/stunnel
tcp      499      0 127.0.0.1:65514             127.0.0.1:49450             ESTABLISHED 13781/stunnel
 
Finally, send something to local logger and verify you can see it on the remote server. On the server:

# tail -f /var/log/messages

On the client host:
 
# logger "`date` tunnel test"
 
and verify on the server you see the "<date> tunnel test" line in the remote syslog file:

Oct 27 02:35:08 dormarth rumtuk: Thu Oct 27 02:35:08 UTC 2016 tunnel test
 
If you have the wireshark package installed, you may also want to run tshark on the stunnel port to verify the messages are indeed going across encrypted. For example, on the server, here is the traffic on port 65515 when the same logger command as above is repeated on the client:
 
# tshark -i ens33 | grep 65515
Running as user "root" and group "root". This could be dangerous.
Capturing on 'ens33'
 
80  64 5.467663041 108.31.178.247 -> 64.137.248.212 TCP 161 fjdmimgr > 65515 [PSH, ACK] Seq=1 Ack=1 Win=160 Len=107
 65 5.467829924 64.137.248.212 -> 108.31.178.247 TCP 54 65515 > fjdmimgr [ACK] Seq=1 Ack=108 Win=497 Len=0
 69 6.097558428 108.31.178.247 -> 64.137.248.212 TCP 161 fjdmimgr > 65515 [PSH, ACK] Seq=108 Ack=1 Win=160 Len=107
 70 6.097749638 64.137.248.212 -> 108.31.178.247 TCP 54 65515 > fjdmimgr [ACK] Seq=1 Ack=215 Win=497 Len=0
199 190 19.123101437 108.31.178.247 -> 64.137.248.212 TCP 157 fjdmimgr > 65515 [PSH, ACK] Seq=215 Ack=1 Win=160 Len=103
191 19.123287145 64.137.248.212 -> 108.31.178.247 TCP 54 65515 > fjdmimgr [ACK] Seq=1 Ack=318 Win=497 Len=0
405
0 (0)
Article Rating (No Votes)
Rate this article
Attachments
There are no attachments for this article.
Comments
There are no comments for this article. Be the first to post a comment.
Full Name
Email Address
Security Code Security Code
Related Articles RSS Feed
linux ssh How to Hide the OpenSSH Version Details when Telnet to Port 22
Viewed 1442 times since Wed, Apr 22, 2020
RHCS6: Create a new Logical Volume / Global Filesystem 2 (GFS2)
Viewed 1122 times since Sun, Jun 3, 2018
How log rotation works with logrotate
Viewed 2570 times since Fri, Nov 30, 2018
Tilix: Advanced Tiling Terminal Emulator for Power Users
Viewed 3429 times since Thu, Apr 18, 2019
Linux – Securing your important files with XFS extendend attributes
Viewed 6400 times since Wed, Jul 25, 2018
Create a Linux Swap File
Viewed 1407 times since Fri, Jun 8, 2018
Improve security with polyinstantiation
Viewed 10659 times since Fri, May 15, 2020
Linux Audit The Linux security blog about Auditing, Hardening, and Compliance lynis
Viewed 921 times since Thu, Jan 16, 2020
RHEL: XFS basic operations
Viewed 12954 times since Sat, Jun 2, 2018
Top 10 darmowych i publicznych serwerów DNS
Viewed 1280 times since Tue, May 22, 2018