SonicChicken blog

Looking over the last few posts, I seem to be stuck in a rut. I apologize to readers looking for stories about our remodeling project, progress on Barb’s quilt shop, cooking, or other tales of home. Believe me when I say that I’d much rather be writing about those things! Sadly I’ve been attending the school of hard knocks for the last month, and I want to record some of the things I’ve learned in the hopes of helping other people in similar situations. Heck, it may even come in handy for me in the future, although I sincerely hope not!

The system I’m currently fixing has a high-reliability disk setup. The server has two fibre channel interfaces, each connected to a separate RAID-5 array. The server takes care of mirroring, while the external disk arrays implement the RAID-5. This is all done with an Apple Xserve server and an Xserve RAID enclosure.

It looks like one of the RAID arrays got corrupted during our recent power outage, as diskutility (the GUI tool) is reporting “mirror degraded”. I used diskutil (CLI) and the RAID Admin utility to check the RAID arrays, but they both reported no errors on either side of the mirror. A little time on Google turned up a post which suggested using iostat to determine which side of the mirror had failed, but when I tried rebuilding the mirror using those instructions I kept getting “Error -9980″. Some more digging and it looks like the “broken” side of the mirror was unmounted. I tried several things to get that half of the mirror to reconnect, even going so far as shutting down the server and the RAID and rebooting. No luck, although I was able to get ‘diskutil repairMirror raid-disk slice from-disk to-disk‘ accepted. (It didn’t actually do anything, though. But at least it didn’t error out anymore.)

Finally I noticed that one of the RAID controller cards was behaving oddly. It reported status=okay, but it wouldn’t allow connection from the RAID Admin utility (over the LAN). I had been ignoring it, since I was connected to the other controller, but I decided to connect directly to the flaky one and see if there was something it wasn’t reporting to the controller I had been using to diagnose the problem. I found that the IP address of the dodgy controller wasn’t correct, and the passwords for monitoring and management access weren’t working. I shut down the server again, and this time when I shut down the RAID I pulled the power cords. (I had found out that without pulling the power cords, the array isn’t actually powered off; rather, it’s in “sleep” mode.) I powered up the RAID (but not the server!), then reset the RAID controller (just the password reset, not the full-blown one). Between power-cycling the RAID and resetting the controller, I was able to get the RAID Admin utility to connect to the array. Then when I rebooted the server, the disk mirror started rebuilding without any other action on my part.

The mirror seems to be rebuilding right now. The RAID arrays are 1.225 TB (1.1 TiB) each, with ATA100 disks. (It’s a few years old.) At the rate it’s been going, I expect the rebuild to take about 11 hours total.

We had some trouble with our FreeBSD systems over the holiday shutdown a few weeks ago. Our backup generator didn’t kick in during an extended power outage, and our UPS’s didn’t provide enough runtime to see us through. Result: all our systems crashed. (Before anybody mentions it… Yes, I know I should have installed NUT and configured our network to shutdown in such cases.)

When I got to the office and started powering systems back up, I noticed the expected messages about the filesystems not being dismounted properly. After all the systems were up, I went back around and started logging in to survey the damage. Here’s an example of what happened when I tried to run fsck on one of our filesystems:

# fsck -y /tmp
** /dev/twed0s1e (NO WRITE)
** Last Mounted on /tmp
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
.
.
.

I had never seen “(NO WRITE)” show up on fsck before. A little searching turned up this post which explains that (NO WRITE) means the filesystem is mounted, thus fsck cannot write to it. I went back and rebooted each system into single-user mode. Then I was able to fsck all the local filesystems and reboot cleanly.

# init 1
# fsck -y -t ufs
# reboot

We are currently building about a dozen FreeBSD servers at work. The build process is pretty well documented, but it’s a bit labor intensive. So I had one of my student assistants look into automating the process somewhat. This post discusses our assumptions and initial conditions, along with our results and recommendations. Hopefully this can help others trying to implement similar plans.

We implemented this on FreeBSD versions 6.2, 6.3, 6.4. We will also be testing this on version 7.1 later this month. We are focusing on the i386 architecture, and we would like to try large (≥3 GiB) memory configurations with the amd architecture as well.

All of our systems have a common set of basic tools:

• portmaster
• portupgrade
• portaudit
• spinner
• screen
• postfix
• nagios
• sudo
• bash
• emacs
• lynx

Ideally we would like to install all of these common tools with one command that doesn’t require further user intervention. We could have created a package for this, but I’m pretty sure we would need to build a separate package for every possible FreeBSD version and architecture — that’s 4 O/S versions x 2 architectures = 8 packages, which doesn’t seem worthwhile for ~12 systems.

Another possible approach would have been creation of a “metaport”, a single pseudo-application listing all of the tools as dependencies. I didn’t pursue this because it doesn’t (by itself) address the user intervention issue.

One more goal of this project was the automation/scripting of some common system configuration tasks such as enabling/starting NTP, setting up network printers, etc. This doesn’t have anything to do with the ports/packages system, so we needed to write a script for this part. Most of this code is pretty obvious and mundane, but I’ve included it in the article for the sake of completeness.

The trickiest part of this whole project was figuring out how to do unattended installation of ports with options, e.g. screen. Normally when you install such a port, you will be presented with an ncurses menu of options for that port. Each of these options should correspond to a WITH_* / WITHOUT_* variable that controls how the port is built, possibly affecting the port’s dependencies, etc. The options menu is displayed when the ‘config’ target is built:

# cd /usr/ports/sysutils/screen
# make config
(ncurses option menu appears, user chooses which options to enable/disable)
# grep ^WITH /var/db/ports/screen/options
WITHOUT_CJK=true
WITH_INFO=true
WITH_MAN=true
WITH_NETHACK=true
WITH_XTERM_256=true
WITHOUT_HOSTINLOCKED=true
WITHOUT_SHOWENC=true

The options are saved in /var/db/ports/portname/options, as shown above. Once you have figured out which options to enable/disable on one system (interactively), you can copy them into a port-specific section of /etc/make.conf as follows:

# echo '# Options for sysutils/screen' >>/etc/make.conf
# echo 'if ${.CURDIR:M*ports/sysutils/screen}' >>/etc/make.conf
# grep ^WITH /var/db/ports/screen/options >>/etc/make.conf
# echo '.endif' >>/etc/make.conf

We’ll now test this by removing the config file and building the port with the BATCH=yes flag set. This should prevent the ncurses config screen from popping up.

# cd /usr/ports/sysutils/screen
# make rmconfig clean
(We're using /bin/sh for our installation script, as well as for testing.)
(If you're using csh, you'll need to use *setenv* instead of *export*.)
# export BATCH=yes
# make install clean
(Lots and lots of output, but no configuration prompting!)

The components of the solution are all in place! Now all we need is to put them together and write the script, but that’ll have to wait until next week.