I recently had the opportunity of giving a talk in the HGS MathComp Fellows Seminar. This is a rather informal venue where doctoral students of my graduate school may present their thesis work or give introductions to their field. I quite enjoyed giving the talk as the atmosphere was very encouraging.

For my talk, I chose the rather tongue-in-cheek title "Oh my god, it's full of data!", which is a very unsubtle reference to 2001 Space Odyssey. Checkout my slides for more information. I might even get my 15 minutes of internet fame, as the talks are usually recorded and uploaded to YouTube. Let's see how that is going to work out.

Posted Thursday afternoon, May 2nd, 2013 Tags:

This is part II of my journey into the wonderful world of building my own NAS. Check out part I for an explanation of the hardware I used.

Basic installation

I quickly chose FreeBSD as the operating system for the NAS because it is the one operating system that I am familiar with that also seems suitable for servers. See the FreeBSD wiki for some other compelling reasons.

After grabbing the memstick image from the FTP server, I started installing the base system. I you have never done this before, I would recommend reading the FreeBSD handbook first, in particular its section about the installation. They describe it way better than I possibly could. Assuming you now know what you are doing, you should disable journaling for all partitions on your SSD. You can do so prior to committing the file system changes in the FreeBSD installer by selecting the advanced options of your partitioning scheme. If you do not install the system on an SSD, you need not heed my advice.

Apart from the journaling, I went with the standard options. After the setup finished, I of course downloaded the most recent ports collections snapshots via portsnap fetch extract.

Linux procfs compatibility

I like to use htop for additional sysadmin coolness, so I always enable the Linux compatibility first. I first added the line linux_enable="YES to /etc/rc.conf. I then created the necessary directories using mkdir -p /usr/compat/linux/proc, and for compatibility reasons, create a symbolic link via ln -s /usr/compat /compat. Last, I added the following line into /etc/fstab:

linproc         /compat/linux/proc      linprocfs       rw      0       0

After a reboot, the new file system should be mounted and finally, the port htop can be installed.

powerd configuration

In the first article, I talked at lengths about trying to reduce the power consumption of the NAS. This is where powerd comes in handy. I added the following lines to /etc/rc.conf:

powerd_enable="YES"
powerd_flags="-a adaptive -b adaptive"
economy_cx_lowest="C2"
performance_cx_lowest="C2"

This basically tells powerd to always try to adjust the system performance, depending on the current load of the system. The other two options instruct the CPU to use the C2 state as its lowest state, regardless of the system load. The CPU in my NAS is only capable of C2. If your CPU manages the C3 state, the results will even be better. Use sysctl dev.cpu.0.cx_supported to query the supported states for the first CPU, for example. See the FreeBSD wiki for further power consumption tuning tips.

E-mail configuration

For the time being, I decided not to run my own mail server on the NAS because I do not think I am qualified to do so. Since I still want to rely on the reporting capabilities of daemons, I installed ssmtp as an alternative. Here's an excerpt of its man-page:

ssmtp is a send-only sendmail emulator for machines which normally pick their mail up from a centralized mailhub (via pop, imap, nfs mounts or other means). It provides the functionality required for humans and programs to send mail via the standard or /usr/bin/mail user agents.

In a previous post, I already talked about how to set up ssmtp for Google Mail. This is exactly how I proceeded this time. To completely disable sendmail, the default MTA in FreeBSD, I added the following lines to /etc/rc.conf:

sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

Furthermore, I modified /etc/mail/mailer.conf and replaced all entries with ssmtp. To wit:

sendmail        /usr/local/sbin/ssmtp
send-mail       /usr/local/sbin/ssmtp
mailq           /usr/local/sbin/ssmtp
newaliases      /usr/local/sbin/ssmtp
hoststat        /usr/bin/true
purgestat       /usr/bin/true

This procedure seems to be the official way of replacing sendmail with ssmtp. At least, it is suggested by the maintainers of the ssmtp port.

As a last touch, I disabled some of the periodic reports that are sent by FreeBSD. These are the relevant parts of the /etc/periodic.conf:

daily_status_network_enable="NO"
daily_status_mailq_enable="NO"
daily_status_mail_rejects_enable="NO"
daily_status_mail_rejects_enable="NO"

This leaves with periodic e-mails about the disk status, user accounts, and some other stuff. I don't need to be informed about the network or mail status.

Installing NTFS write support

Some of my data is still on hard drives that were regularly read and written to by Microsoft Windows (you may want to shudder now and shun me, so go ahead). Since the default NTFS driver included in FreeBSD is rather slow and cannot actually write to these disks, I installed the port fusefs-ntfs. Its configuration was again very simple and required only the line fusefs_enable="YES" in /etc/rc.conf, as well as the line vfs.usermount=1 in /etc/sysctl.conf. After rebooting, you should be able to use the driver by issuing ntfs-3g /dev/da0s1 /mnt, for example.

Conclusion

This concludes the basic system setup. For the next part, I plan on explaining my ZFS configuration as well as some tools that simplify managing the server. Stay tuned.

Posted late Saturday evening, May 4th, 2013 Tags:

For a long time, rdiff-backup has been my preferred backup solution. Recently, however, I started looking for an alternative because its performance simply was not up to par any more–it took over 20 minutes to process my home directory although less than 1 MiB of files had changed.

Luckily, I stumbled over Obnam. Joey Hess already adopted it as an additional backup solution, so I figured I could give it a try as well. And I was pleasantly surprised: Configuration is very easy, the backup is sufficiently fast, there's support for GPG-based encryption, and it offers deduplication. What's not to love?

So, my current backup setup for my laptop has a central .obnam.conf configuration file with the following content:

[config]

repository = sftp://example/home/example/backup.hostname.example.com

exclude = \.o$, \.tmp$, /Trash/, ...

Of course, the exclude list is a tad longer in reality. This is the only tedious thing about Obnam: All excluded files and directories have to be specified as a regular expression in a single line. I have several folders I do not wish to be transferred, such as caches, downloads, temporary storage, and the like. For each of these folders, I either added a /foldername/ or the full path to the exclude line so that Obnam does not include for regular backups. Here are some recommendations for stuff to exclude:

  • \.o$
  • $HOME/.cache
  • $HOME/.thumbnails
  • /build/
  • /Downloads/
  • /Trash/
  • A variety of auxiliary files for LaTeX sources, temporary backup files for editors, and so on

With a sufficient exclusion list, I was now able to do backups quite comfortably using obnam backup $HOME. But there's more: Since the backup usually does not take very long, I decided to let it run automatically. I created a small shell script for this:

#!/bin/sh

notify-send "Starting obnam backup..."

obnam backup $HOME 

if [ "$?" -ne 0 ]; then
  notify-send "Unable to finish obnam backup."
  exit 1
else
  notify-send "Finished obnam backup."
fi

notify-send is a nice little program that allows scripts to send notifications to a window manager. Both awesome and GNOME 3 show small notification bubbles for a small period of time. Finally, I added an entry to my personal crontab, instructing Obnam to run every day at 8 o'clock in the evening:

0 20 * * * DISPLAY=:0 /home/example/backup

Note that the DISPLAY=:0 variable is required so that the notifications of the script are shown. For systems that are only powered on sporadically, it might also make sense to create a configuration for anacron. So far, the normal cron variant served me well, though.

If you want to use the script yourself, you might want to incorporate a more powerful logging and notification capability in case Obnam does not complete its run. For my own use (and for now), I am content with checking the backup generations created by Obnam using obnam generations every once in a while.

Posted Sunday afternoon, May 26th, 2013 Tags: