Though I think we would all agree that it’s important, the unfortunate fact is that myself and other members of my “team” don’t necessarily document things as well as we always should. I’m fairly certain that a lot of knowledge would be gone forever if one of us fell asleep in a bathtub and drown while reading a UNIX book (otherwise referred to as the “bus factor”).

Recently, I’ve been trying to remind myself to better document more of the things I do and, in the process, discovered etckeeper.

etckeeper is a collection of tools to let /etc be stored in a git, mercurial, darcs, or bzr repository. It hooks into apt (and other package managers including yum and pacman-g2) to automatically commit changes made to /etc during package upgrades.

On Ubuntu, etckeeper will use bzr by default and the official documentation does a good job of explaining using bzr with etckeeper. I prefer git myself, so this article will basically be a conversion of that document to git.

I started out with a clean installation of Ubuntu 10.04 LTS to have a clean slate to work from.

Since I want to use git, the first thing I’ll need to do is install the git-core package (and its dependencies) and tell git who I am:

$ sudo apt-get -y install git-core
$ git config --global user.name "Jeremy L. Gaddis"
$ git config --global user.email jeremy@lab.evilrouters.net

Next, install etckeeper:

$ sudo apt-get -y install etckeeper

We’ll need to tell etckeeper that we want to use git instead of bzr. Open up /etc/etckeeper/etckeeper.conf in your favorite text editor. The first five lines look like this:

# The VCS to use.
# VCS="hg"
# VCS="git"
VCS="bzr"
# VCS="darcs"

We want it to look like this:

# The VCS to use.
# VCS="hg"
VCS="git"
# VCS="bzr"
# VCS="darcs"

With that change made, we can initialize the repository and do an initial commit to get things started:

$ sudo etckeeper init
Initialized empty Git repository in /etc/.git/
$ sudo etckeeper commit "Initial commit."

We could stop right here and be done. The Ubuntu etckeeper package sets up a cronjob that will run daily and auto-commit any changes to files in or under the /etc directory that we haven’t explicitly committed.

To check if there are any uncommitted changes, we can use the “git status” command:

$ cd /etc
$ sudo git status
# On branch master
nothing to commit (working directory) clean

“git log” will show us a history of commits:

$ sudo git log
commit b32d9b32f8c8cf9fe2a5d2070d6c08024bc497c5
Author: root <root@ubuntu.lab.evilrouters.net>
Date:   Fri Feb 18 05:08:39 2011 -0500

    Initial commit.

To see the power of etckeeper we need to make some changes to our config files under /etc. etckeeper hooks into apt (via /etc/apt/apt.conf.d/05etckeeper) before and after the installation of any packages which gives the ability to track the installation, upgrading, and removal of software packages as well! This is a clean installation and I haven’t yet installed the security updates (intentionally) so let’s do that and see what changes are made:

$ sudo apt-get update
$ sudo apt-get -y upgrade

apt upgraded 84 packages, so let’s take a look at what just happened:

$ sudo git log --summary -1
commit 1a2b75dd6bb2781c6785dfb063d959d55bccc248
Author: root <root@ubuntu.lab.evilrouters.net>
Date:   Fri Feb 18 05:22:20 2011 -0500

    committing changes in /etc after apt run

    Package changes:
    -apparmor 2.5-0ubuntu3
    -apparmor-utils 2.5-0ubuntu3
    +apparmor 2.5.1-0ubuntu0.10.04.3
    +apparmor-utils 2.5.1-0ubuntu0.10.04.3
    -apt 0.7.25.3ubuntu7
    -apt-transport-https 0.7.25.3ubuntu7
    -apt-utils 0.7.25.3ubuntu7
    +apt 0.7.25.3ubuntu9.3
    +apt-transport-https 0.7.25.3ubuntu9.3
    +apt-utils 0.7.25.3ubuntu9.3
    -at 3.1.11-1ubuntu5
    -base-files 5.0.0ubuntu20
    +at 3.1.11-1ubuntu5.1
    +base-files 5.0.0ubuntu20.10.04.3
    -bind9-host 1:9.7.0.dfsg.P1-1
    +bind9-host 1:9.7.0.dfsg.P1-1ubuntu0.1
    -bsdutils 1:2.17.2-0ubuntu1
    +bsdutils 1:2.17.2-0ubuntu1.10.04.2
    -byobu 2.68-0ubuntu1
    -bzip2 1.0.5-4
    +byobu 2.68-0ubuntu1.1
    +bzip2 1.0.5-4ubuntu0.1
    -coreutils 7.4-2ubuntu2
    +coreutils 7.4-2ubuntu3
    -dmsetup 2:1.02.39-1ubuntu4
    -dnsutils 1:9.7.0.dfsg.P1-1
    +dmsetup 2:1.02.39-1ubuntu4.1
    +dnsutils 1:9.7.0.dfsg.P1-1ubuntu0.1
    -dpkg 1.15.5.6ubuntu4
    -e2fslibs 1.41.11-1ubuntu2
    -e2fsprogs 1.41.11-1ubuntu2
    +dpkg 1.15.5.6ubuntu4.5
    +e2fslibs 1.41.11-1ubuntu2.1
    +e2fsprogs 1.41.11-1ubuntu2.1
    -fuse-utils 2.8.1-1.1ubuntu2
    +fuse-utils 2.8.1-1.1ubuntu2.2
    -gzip 1.3.12-9ubuntu1
    +gzip 1.3.12-9ubuntu1.1
    -ifupdown 0.6.8ubuntu29
    +ifupdown 0.6.8ubuntu29.2
    -language-selector-common 0.5.7
    +language-selector-common 0.5.8
    -libapparmor-perl 2.5-0ubuntu3
    -libapparmor1 2.5-0ubuntu3
    +libapparmor-perl 2.5.1-0ubuntu0.10.04.3
    +libapparmor1 2.5.1-0ubuntu0.10.04.3
    -libbind9-60 1:9.7.0.dfsg.P1-1
    -libblkid1 2.17.2-0ubuntu1
    +libbind9-60 1:9.7.0.dfsg.P1-1ubuntu0.1
    +libblkid1 2.17.2-0ubuntu1.10.04.2
    -libbz2-1.0 1.0.5-4
    -libc-bin 2.11.1-0ubuntu7
    -libc6 2.11.1-0ubuntu7
    +libbz2-1.0 1.0.5-4ubuntu0.1
    +libc-bin 2.11.1-0ubuntu7.8
    +libc6 2.11.1-0ubuntu7.8
    -libcomerr2 1.41.11-1ubuntu2
    +libcomerr2 1.41.11-1ubuntu2.1
    -libdbus-1-3 1.2.16-2ubuntu4
    +libdbus-1-3 1.2.16-2ubuntu4.1
    -libdevmapper1.02.1 2:1.02.39-1ubuntu4
    +libdevmapper1.02.1 2:1.02.39-1ubuntu4.1
    -libdns64 1:9.7.0.dfsg.P1-1
    +libdns64 1:9.7.0.dfsg.P1-1ubuntu0.1
    -libfuse2 2.8.1-1.1ubuntu2
    +libfuse2 2.8.1-1.1ubuntu2.2
    -libglib2.0-0 2.24.0-0ubuntu4
    +libglib2.0-0 2.24.1-0ubuntu1
    -libgssapi-krb5-2 1.8.1+dfsg-2
    +libgssapi-krb5-2 1.8.1+dfsg-2ubuntu0.6
    -libisc60 1:9.7.0.dfsg.P1-1
    -libisccc60 1:9.7.0.dfsg.P1-1
    -libisccfg60 1:9.7.0.dfsg.P1-1
    +libisc60 1:9.7.0.dfsg.P1-1ubuntu0.1
    +libisccc60 1:9.7.0.dfsg.P1-1ubuntu0.1
    +libisccfg60 1:9.7.0.dfsg.P1-1ubuntu0.1
    -libk5crypto3 1.8.1+dfsg-2
    +libk5crypto3 1.8.1+dfsg-2ubuntu0.6
    -libkrb5-3 1.8.1+dfsg-2
    -libkrb5support0 1.8.1+dfsg-2
    -libldap-2.4-2 2.4.21-0ubuntu5
    +libkrb5-3 1.8.1+dfsg-2ubuntu0.6
    +libkrb5support0 1.8.1+dfsg-2ubuntu0.6
    +libldap-2.4-2 2.4.21-0ubuntu5.3
    -liblwres60 1:9.7.0.dfsg.P1-1
    +liblwres60 1:9.7.0.dfsg.P1-1ubuntu0.1
    -libpam-modules 1.1.1-2ubuntu2
    -libpam-runtime 1.1.1-2ubuntu2
    -libpam0g 1.1.1-2ubuntu2
    +libpam-modules 1.1.1-2ubuntu5
    +libpam-runtime 1.1.1-2ubuntu5
    +libpam0g 1.1.1-2ubuntu5
    -libparted0debian1 2.2-5ubuntu5
    +libparted0debian1 2.2-5ubuntu5.1
    -libpcsclite1 1.5.3-1ubuntu4
    -libplymouth2 0.8.2-2ubuntu2
    -libpng12-0 1.2.42-1ubuntu2
    +libpcsclite1 1.5.3-1ubuntu4.1
    +libplymouth2 0.8.2-2ubuntu2.2
    +libpng12-0 1.2.42-1ubuntu2.1
    -libss2 1.41.11-1ubuntu2
    -libssl0.9.8 0.9.8k-7ubuntu8
    +libss2 1.41.11-1ubuntu2.1
    +libssl0.9.8 0.9.8k-7ubuntu8.6
    -libudev0 151-12
    +libudev0 151-12.3
    -libusb-0.1-4 2:0.1.12-14
    -libuuid1 2.17.2-0ubuntu1
    +libusb-0.1-4 2:0.1.12-14ubuntu0.2
    +libuuid1 2.17.2-0ubuntu1.10.04.2
    -libwww-perl 5.834-1
    +libwww-perl 5.834-1ubuntu0.1
    -libxml2 2.7.6.dfsg-1ubuntu1
    +libxml2 2.7.6.dfsg-1ubuntu1.1
    -linux-firmware 1.34
    +linux-firmware 1.34.3
    -login 1:4.1.4.2-1ubuntu2
    +login 1:4.1.4.2-1ubuntu2.2
    -man-db 2.5.7-2
    +man-db 2.5.7-2ubuntu1
    -mount 2.17.2-0ubuntu1
    -mountall 2.14
    +mount 2.17.2-0ubuntu1.10.04.2
    +mountall 2.15.3
    -openssh-client 1:5.3p1-3ubuntu3
    -openssh-server 1:5.3p1-3ubuntu3
    -openssl 0.9.8k-7ubuntu8
    +openssh-client 1:5.3p1-3ubuntu5
    +openssh-server 1:5.3p1-3ubuntu5
    +openssl 0.9.8k-7ubuntu8.6
    -parted 2.2-5ubuntu5
    -passwd 1:4.1.4.2-1ubuntu2
    +parted 2.2-5ubuntu5.1
    +passwd 1:4.1.4.2-1ubuntu2.2
    -plymouth 0.8.2-2ubuntu2
    -plymouth-theme-ubuntu-text 0.8.2-2ubuntu2
    +plymouth 0.8.2-2ubuntu2.2
    +plymouth-theme-ubuntu-text 0.8.2-2ubuntu2.2
    -python-apt 0.7.94.2ubuntu6
    +python-apt 0.7.94.2ubuntu6.2
    -python-lazr.restfulclient 0.9.11-1ubuntu1
    +python-lazr.restfulclient 0.9.11-1ubuntu1.1
    -rsyslog 4.2.0-2ubuntu8
    -screen 4.0.3-14ubuntu1
    +rsyslog 4.2.0-2ubuntu8.1
    +screen 4.0.3-14ubuntu1.2
    -sudo 1.7.2p1-1ubuntu5
    +sudo 1.7.2p1-1ubuntu5.3
    -tar 1.22-2
    +tar 1.22-2ubuntu1
    -tzdata 2010i-1
    +tzdata 2010o-0ubuntu0.10.04
    -udev 151-12
    +udev 151-12.3
    -update-manager-core 1:0.134.7
    +update-manager-core 1:0.134.11
    -upstart 0.6.5-6
    -ureadahead 0.100.0-4.1
    +upstart 0.6.5-8
    +ureadahead 0.100.0-4.1.3
    -util-linux 2.17.2-0ubuntu1
    -uuid-runtime 2.17.2-0ubuntu1
    +util-linux 2.17.2-0ubuntu1.10.04.2
    +uuid-runtime 2.17.2-0ubuntu1.10.04.2
    -w3m 0.5.2-2.1ubuntu1
    -wget 1.12-1.1ubuntu2
    +w3m 0.5.2-2.1ubuntu1.1
    +wget 1.12-1.1ubuntu2.1

 create mode 100644 apparmor.d/abstractions/dbus-session
 mode change 100755 => 120000 init.d/screen-cleanup
 create mode 100644 init/screen-cleanup.conf
 delete mode 120000 rcS.d/S70screen-cleanup

Without any extra work on our part, we now have a permanent record of what just happened during the apt-get upgrade process.

Next up, let’s make a change to some config files and see what happens. First, I’m going to install the apache2 package:

$ sudo apt-get -y install apache2

Let’s see what changes just occurred in /etc:

$ sudo git log --summary 1
commit 29511f0bac9ec81be799ae13a46a3f27c676d365
Author: root <root@ubuntu.lab.evilrouters.net>
Date:   Fri Feb 18 05:28:20 2011 -0500

    committing changes in /etc after apt run

    Package changes:
    +apache2 2.2.14-5ubuntu8.4
    +apache2-mpm-worker 2.2.14-5ubuntu8.4
    +apache2-utils 2.2.14-5ubuntu8.4
    +apache2.2-bin 2.2.14-5ubuntu8.4
    +apache2.2-common 2.2.14-5ubuntu8.4
    +libapr1 1.3.8-1build1
    +libaprutil1 1.3.9+dfsg-3ubuntu0.10.04.1
    +libaprutil1-dbd-sqlite3 1.3.9+dfsg-3ubuntu0.10.04.1
    +libaprutil1-ldap 1.3.9+dfsg-3ubuntu0.10.04.1
    +ssl-cert 1.0.23ubuntu2

 create mode 100644 apache2/apache2.conf
 create mode 100644 apache2/conf.d/charset
 create mode 100644 apache2/conf.d/localized-error-pages
 create mode 100644 apache2/conf.d/security
 create mode 100644 apache2/envvars
 create mode 100644 apache2/httpd.conf
 create mode 100644 apache2/magic
 create mode 100644 apache2/mods-available/actions.conf
 create mode 100644 apache2/mods-available/actions.load
 create mode 100644 apache2/mods-available/alias.conf
 create mode 100644 apache2/mods-available/alias.load
 create mode 100644 apache2/mods-available/asis.load
 create mode 100644 apache2/mods-available/auth_basic.load
 create mode 100644 apache2/mods-available/auth_digest.load
 create mode 100644 apache2/mods-available/authn_alias.load
 create mode 100644 apache2/mods-available/authn_anon.load
 create mode 100644 apache2/mods-available/authn_dbd.load
 create mode 100644 apache2/mods-available/authn_dbm.load
 create mode 100644 apache2/mods-available/authn_default.load
 create mode 100644 apache2/mods-available/authn_file.load
 create mode 100644 apache2/mods-available/authnz_ldap.load
 create mode 100644 apache2/mods-available/authz_dbm.load
 create mode 100644 apache2/mods-available/authz_default.load
 create mode 100644 apache2/mods-available/authz_groupfile.load
 create mode 100644 apache2/mods-available/authz_host.load
 create mode 100644 apache2/mods-available/authz_owner.load
 create mode 100644 apache2/mods-available/authz_user.load
 create mode 100644 apache2/mods-available/autoindex.conf
 create mode 100644 apache2/mods-available/autoindex.load
 create mode 100644 apache2/mods-available/cache.load
 create mode 100644 apache2/mods-available/cern_meta.load
 create mode 100644 apache2/mods-available/cgi.load
 create mode 100644 apache2/mods-available/cgid.conf
 create mode 100644 apache2/mods-available/cgid.load
 create mode 100644 apache2/mods-available/charset_lite.load
 create mode 100644 apache2/mods-available/dav.load
 create mode 100644 apache2/mods-available/dav_fs.conf
 create mode 100644 apache2/mods-available/dav_fs.load
 create mode 100644 apache2/mods-available/dav_lock.load
 create mode 100644 apache2/mods-available/dbd.load
 create mode 100644 apache2/mods-available/deflate.conf
 create mode 100644 apache2/mods-available/deflate.load
 create mode 100644 apache2/mods-available/dir.conf
 create mode 100644 apache2/mods-available/dir.load
 create mode 100644 apache2/mods-available/disk_cache.conf
 create mode 100644 apache2/mods-available/disk_cache.load
 create mode 100644 apache2/mods-available/dump_io.load
 create mode 100644 apache2/mods-available/env.load
 create mode 100644 apache2/mods-available/expires.load
 create mode 100644 apache2/mods-available/ext_filter.load
 create mode 100644 apache2/mods-available/file_cache.load
 create mode 100644 apache2/mods-available/filter.load
 create mode 100644 apache2/mods-available/headers.load
 create mode 100644 apache2/mods-available/ident.load
 create mode 100644 apache2/mods-available/imagemap.load
 create mode 100644 apache2/mods-available/include.load
 create mode 100644 apache2/mods-available/info.conf
 create mode 100644 apache2/mods-available/info.load
 create mode 100644 apache2/mods-available/ldap.load
 create mode 100644 apache2/mods-available/log_forensic.load
 create mode 100644 apache2/mods-available/mem_cache.conf
 create mode 100644 apache2/mods-available/mem_cache.load
 create mode 100644 apache2/mods-available/mime.conf
 create mode 100644 apache2/mods-available/mime.load
 create mode 100644 apache2/mods-available/mime_magic.conf
 create mode 100644 apache2/mods-available/mime_magic.load
 create mode 100644 apache2/mods-available/negotiation.conf
 create mode 100644 apache2/mods-available/negotiation.load
 create mode 100644 apache2/mods-available/proxy.conf
 create mode 100644 apache2/mods-available/proxy.load
 create mode 100644 apache2/mods-available/proxy_ajp.load
 create mode 100644 apache2/mods-available/proxy_balancer.load
 create mode 100644 apache2/mods-available/proxy_connect.load
 create mode 100644 apache2/mods-available/proxy_ftp.load
 create mode 100644 apache2/mods-available/proxy_http.load
 create mode 100644 apache2/mods-available/proxy_scgi.load
 create mode 100644 apache2/mods-available/reqtimeout.conf
 create mode 100644 apache2/mods-available/reqtimeout.load
 create mode 100644 apache2/mods-available/rewrite.load
 create mode 100644 apache2/mods-available/setenvif.conf
 create mode 100644 apache2/mods-available/setenvif.load
 create mode 100644 apache2/mods-available/speling.load
 create mode 100644 apache2/mods-available/ssl.conf
 create mode 100644 apache2/mods-available/ssl.load
 create mode 100644 apache2/mods-available/status.conf
 create mode 100644 apache2/mods-available/status.load
 create mode 100644 apache2/mods-available/substitute.load
 create mode 100644 apache2/mods-available/suexec.load
 create mode 100644 apache2/mods-available/unique_id.load
 create mode 100644 apache2/mods-available/userdir.conf
 create mode 100644 apache2/mods-available/userdir.load
 create mode 100644 apache2/mods-available/usertrack.load
 create mode 100644 apache2/mods-available/version.load
 create mode 100644 apache2/mods-available/vhost_alias.load
 create mode 120000 apache2/mods-enabled/alias.conf
 create mode 120000 apache2/mods-enabled/alias.load
 create mode 120000 apache2/mods-enabled/auth_basic.load
 create mode 120000 apache2/mods-enabled/authn_file.load
 create mode 120000 apache2/mods-enabled/authz_default.load
 create mode 120000 apache2/mods-enabled/authz_groupfile.load
 create mode 120000 apache2/mods-enabled/authz_host.load
 create mode 120000 apache2/mods-enabled/authz_user.load
 create mode 120000 apache2/mods-enabled/autoindex.conf
 create mode 120000 apache2/mods-enabled/autoindex.load
 create mode 120000 apache2/mods-enabled/cgid.conf
 create mode 120000 apache2/mods-enabled/cgid.load
 create mode 120000 apache2/mods-enabled/deflate.conf
 create mode 120000 apache2/mods-enabled/deflate.load
 create mode 120000 apache2/mods-enabled/dir.conf
 create mode 120000 apache2/mods-enabled/dir.load
 create mode 120000 apache2/mods-enabled/env.load
 create mode 120000 apache2/mods-enabled/mime.conf
 create mode 120000 apache2/mods-enabled/mime.load
 create mode 120000 apache2/mods-enabled/negotiation.conf
 create mode 120000 apache2/mods-enabled/negotiation.load
 create mode 120000 apache2/mods-enabled/reqtimeout.conf
 create mode 120000 apache2/mods-enabled/reqtimeout.load
 create mode 120000 apache2/mods-enabled/setenvif.conf
 create mode 120000 apache2/mods-enabled/setenvif.load
 create mode 120000 apache2/mods-enabled/status.conf
 create mode 120000 apache2/mods-enabled/status.load
 create mode 100644 apache2/ports.conf
 create mode 100644 apache2/sites-available/default
 create mode 100644 apache2/sites-available/default-ssl
 create mode 120000 apache2/sites-enabled/000-default
 create mode 100644 bash_completion.d/apache2.2-common
 create mode 100755 cron.daily/apache2
 create mode 100644 default/apache2
 create mode 100755 init.d/apache2
 create mode 100644 logrotate.d/apache2
 create mode 120000 rc0.d/K09apache2
 create mode 120000 rc1.d/K09apache2
 create mode 120000 rc2.d/S91apache2
 create mode 120000 rc3.d/S91apache2
 create mode 120000 rc4.d/S91apache2
 create mode 120000 rc5.d/S91apache2
 create mode 120000 rc6.d/K09apache2
 create mode 120000 ssl/certs/e8891fff
 create mode 100644 ssl/certs/ssl-cert-snakeoil.pem
 create mode 100644 ssl/private/ssl-cert-snakeoil.key
 create mode 100644 ufw/applications.d/apache2.2-common

Last, let’s make some changes to the Apache configuration files. I opened up /etc/apache2/sites-available/default and changed one line:

ServerAdmin webmaster@localhost

…to…

ServerAdmin jeremy@lab.evilrouters.net

$ sudo git status
# On branch master
# Changed but not updated:
#   (use "git add ..." to update what will be committed)
#   (use "git checkout -- ..." to discard changes in working directory)
#
#	modified:   apache2/sites-available/default
#
no changes added to commit (use "git add" and/or "git commit -a")

This shows us that the file /etc/apache2/sites-available/default has been modified since it was last committed. Let’s go ahead and commit the change:

$ sudo etckeeper commit "Updated web server admin's e-mail address."
[master 9904a73] Updated web server admin's e-mail address.
 Author: Jeremy L. Gaddis <jeremy@lab.evilrouters.net>
 1 files changed, 1 insertions(+), 1 deletions(-)

This commit will now show up in the log. Let’s see what, exactly, changed since the previous commit:

$ sudo git log -1 -p
commit 9904a73e009a32a883ec51e1e5bcfe566051c620
Author: Jeremy L. Gaddis <jeremy@lab.evilrouters.net>
Date:   Fri Feb 18 05:42:47 2011 -0500

    Updated web server admin's e-mail address.

diff --git a/apache2/sites-available/default b/apache2/sites-available/default
index b346156..d55f25f 100644
--- a/apache2/sites-available/default
+++ b/apache2/sites-available/default
@@ -1,5 +1,5 @@
 <VirtualHost *:80>
-       ServerAdmin webmaster@localhost
+       ServerAdmin jeremy@lab.evilrouters.net

        DocumentRoot /var/www
        <Directory />

Note that we don’t have to explicitly commit our changes, due to the cronjob that was set up during the installation of etckeeper. If we didn’t manually commit, the changes would have been automatically committed — at the latest — the next time the cronjob ran (which is configured via /etc/crontab).

That’s it! We will now have a full history of any changes made to files in /etc with zero effort beyond the initial installation and configuration of etckeeper, in compliance with Veteran Unix admin trait No. 4 (ironically enough, number one is the only one of those nine traits I don’t completely agree with). Go forth (to /etc) and multiply modify (your config files)!