Tag Archive for 'sysadmin'

23
May

ADODB on FreeBSD

Recently I’ve had one of my student assistants working on setting up a web server. She’s pretty much got it, except that she hasn’t been able to establish a connection from the web server to our database server.

We’re wanting to use ADODB to connect to a mssql server. She figured out how to install the databases/adodb5 port (and how to get ADODB included in the PHP configuration), but that wasn’t enough by itself. I thought there might be a PHP extension required, and a little Unix pipe revealed the missing module:

# grep -i ^php /usr/ports/INDEX-7 | grep -v ^php4 | grep -i adodb

The command above searches the ports index for lines starting with “php” (or “PHP”), ignores those which begin with “php4″ (I’m not interested in version 4 stuff), and searches everything that’s matched so far for “adodb” (or “ADODB”). Results:

php-adodb-ext... (long index line snipped)

A quick check of /var/db/pkg verified that this module hadn’t been installed yet. I installed the module via portmaster(8) and added the new extension per the pkg-message instructions:

# portmaster databases/php-adodb-ext
.
.
.
(installation stuff snipped)
# cat >>/usr/local/etc/php/extensions.ini
extension=adodb.so

Now we’re getting a different error:

PHP Fatal error:  Call to undefined function mssql_get_last_message() in /usr/local/share/adodb/drivers/adodb-mssql.inc.php on line 501

Google indicates the problem is likely that the mssql extension for PHP isn’t installed. I ran the same pipe that I did for ADODB, this time searching for mssql:

# grep -i ^php /usr/ports/INDEX-7 | grep -v ^php4 | grep -i adodb

php5-mssql... (long index line snipped)

So another portmaster(8) installation, this time for php5-mssql. One of php5-mssql’s dependencies is freetds-msdblib, which has some configuration options. The only one I’m setting is MSDBLIB, which enables MS SQLserver (in lieu of the old Sybase DB).

This port (php5-mssql) automatically registers itself as a PHP extension, so you don’t have to add a line to /usr/local/etc/php/extensions.ini yourself.

I’m still getting errors, but they’re starting to look like problems with the database connection string instead of server configuration now. Progress, of a sort. But that’s enough for now–It’s almost 8:00 pm on Friday night before a 3-day weekend, long past time for me to get on with my own life!

27
Jan

Xserve disk mirror repair

Looking over the last few posts, I seem to be stuck in a rut. I apologize to readers looking for stories about our remodeling project, progress on Barb’s quilt shop, cooking, or other tales of home. Believe me when I say that I’d much rather be writing about those things! Sadly I’ve been attending the school of hard knocks for the last month, and I want to record some of the things I’ve learned in the hopes of helping other people in similar situations. Heck, it may even come in handy for me in the future, although I sincerely hope not!

The system I’m currently fixing has a high-reliability disk setup. The server has two fibre channel interfaces, each connected to a separate RAID-5 array. The server takes care of mirroring, while the external disk arrays implement the RAID-5. This is all done with an Apple Xserve server and an Xserve RAID enclosure.

It looks like one of the RAID arrays got corrupted during our recent power outage, as diskutility (the GUI tool) is reporting “mirror degraded”. I used diskutil (CLI) and the RAID Admin utility to check the RAID arrays, but they both reported no errors on either side of the mirror. A little time on Google turned up a post which suggested using iostat to determine which side of the mirror had failed, but when I tried rebuilding the mirror using those instructions I kept getting “Error -9980″. Some more digging and it looks like the “broken” side of the mirror was unmounted. I tried several things to get that half of the mirror to reconnect, even going so far as shutting down the server and the RAID and rebooting. No luck, although I was able to get ‘diskutil repairMirror raid-disk slice from-disk to-disk‘ accepted. (It didn’t actually do anything, though. But at least it didn’t error out anymore.)

Finally I noticed that one of the RAID controller cards was behaving oddly. It reported status=okay, but it wouldn’t allow connection from the RAID Admin utility (over the LAN). I had been ignoring it, since I was connected to the other controller, but I decided to connect directly to the flaky one and see if there was something it wasn’t reporting to the controller I had been using to diagnose the problem. I found that the IP address of the dodgy controller wasn’t correct, and the passwords for monitoring and management access weren’t working. I shut down the server again, and this time when I shut down the RAID I pulled the power cords. (I had found out that without pulling the power cords, the array isn’t actually powered off; rather, it’s in “sleep” mode.) I powered up the RAID (but not the server!), then reset the RAID controller (just the password reset, not the full-blown one). Between power-cycling the RAID and resetting the controller, I was able to get the RAID Admin utility to connect to the array. Then when I rebooted the server, the disk mirror started rebuilding without any other action on my part.

The mirror seems to be rebuilding right now. The RAID arrays are 1.225 TB (1.1 TiB) each, with ATA100 disks. (It’s a few years old.) At the rate it’s been going, I expect the rebuild to take about 11 hours total.

16
Jan

Duplicate DHCP leases

We’ve been getting the following types of error messages on our DHCP server (running on FreeBSD):

Jan 12 07:01:52 *someserver* dhcpd: uid lease 192.168.100.200 for client 00:12:34:56:78:9a is duplicate on 192.168.100/24

Google turned up a helpful post which told me what the problem was, but I didn’t care for the suggested fix. Initially I concentrated on the MAC addresses listed in the error log, seeing if they were also defined in our dhcpd.conf file:

grep "is duplicate" /var/log/messages \
    | cut -f 11 -d' ' \
    | sort -u \
    | grep -C --file=- /usr/local/etc/dhcpd.conf

But this doesn’t seem like the correct approach. What we really ought to be doing is comparing our dhcpd.leases file to dhcpd.conf, looking for MAC addresses that are listed in both. Anything that’s listed with a fixed-address in dhcpd.conf probably shouldn’t be the leases file, IMHO. That’s probably a bit too complicated for a bunch of shell commands piped together, since we’d need to parse a multi-line DHCP entry from each file. I think a perl script might do the job.

14
Jan

Background fsck problems

We had some trouble with our FreeBSD systems over the holiday shutdown a few weeks ago. Our backup generator didn’t kick in during an extended power outage, and our UPS’s didn’t provide enough runtime to see us through. Result: all our systems crashed. (Before anybody mentions it… Yes, I know I should have installed NUT and configured our network to shutdown in such cases.)

When I got to the office and started powering systems back up, I noticed the expected messages about the filesystems not being dismounted properly. After all the systems were up, I went back around and started logging in to survey the damage. Here’s an example of what happened when I tried to run fsck on one of our filesystems:

# fsck -y /tmp
** /dev/twed0s1e (NO WRITE)
** Last Mounted on /tmp
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
.
.
.

I had never seen “(NO WRITE)” show up on fsck before. A little searching turned up this post which explains that (NO WRITE) means the filesystem is mounted, thus fsck cannot write to it. I went back and rebooted each system into single-user mode. Then I was able to fsck all the local filesystems and reboot cleanly.

# init 1
# fsck -y -t ufs
# reboot
11
Jan

Unattended port installation and FreeBSD setup

We are currently building about a dozen FreeBSD servers at work. The build process is pretty well documented, but it’s a bit labor intensive. So I had one of my student assistants look into automating the process somewhat. This post discusses our assumptions and initial conditions, along with our results and recommendations. Hopefully this can help others trying to implement similar plans.

We implemented this on FreeBSD versions 6.2, 6.3, 6.4. We will also be testing this on version 7.1 later this month. We are focusing on the i386 architecture, and we would like to try large (≥3 GiB) memory configurations with the amd architecture as well.

All of our systems have a common set of basic tools:

Ideally we would like to install all of these common tools with one command that doesn’t require further user intervention. We could have created a package for this, but I’m pretty sure we would need to build a separate package for every possible FreeBSD version and architecture — that’s 4 O/S versions x 2 architectures = 8 packages, which doesn’t seem worthwhile for ~12 systems.

Another possible approach would have been creation of a “metaport”, a single pseudo-application listing all of the tools as dependencies. I didn’t pursue this because it doesn’t (by itself) address the user intervention issue.

One more goal of this project was the automation/scripting of some common system configuration tasks such as enabling/starting NTP, setting up network printers, etc. This doesn’t have anything to do with the ports/packages system, so we needed to write a script for this part. Most of this code is pretty obvious and mundane, but I’ve included it in the article for the sake of completeness.

The trickiest part of this whole project was figuring out how to do unattended installation of ports with options, e.g. screen. Normally when you install such a port, you will be presented with an ncurses menu of options for that port. Each of these options should correspond to a WITH_* / WITHOUT_* variable that controls how the port is built, possibly affecting the port’s dependencies, etc. The options menu is displayed when the ‘config’ target is built:

# cd /usr/ports/sysutils/screen
# make config
(ncurses option menu appears, user chooses which options to enable/disable)
# grep ^WITH /var/db/ports/screen/options
WITHOUT_CJK=true
WITH_INFO=true
WITH_MAN=true
WITH_NETHACK=true
WITH_XTERM_256=true
WITHOUT_HOSTINLOCKED=true
WITHOUT_SHOWENC=true

The options are saved in /var/db/ports/portname/options, as shown above. Once you have figured out which options to enable/disable on one system (interactively), you can copy them into a port-specific section of /etc/make.conf as follows:

# echo '# Options for sysutils/screen' >>/etc/make.conf
# echo 'if ${.CURDIR:M*ports/sysutils/screen}' >>/etc/make.conf
# grep ^WITH /var/db/ports/screen/options >>/etc/make.conf
# echo '.endif' >>/etc/make.conf

We’ll now test this by removing the config file and building the port with the BATCH=yes flag set. This should prevent the ncurses config screen from popping up.

# cd /usr/ports/sysutils/screen
# make rmconfig clean
(We're using /bin/sh for our installation script, as well as for testing.)
(If you're using csh, you'll need to use *setenv* instead of *export*.)
# export BATCH=yes
# make install clean
(Lots and lots of output, but no configuration prompting!)

The components of the solution are all in place! Now all we need is to put them together and write the script, but that’ll have to wait until next week.

04
Jun

FreeBSD openssl-stable broken

While upgrading some software on one of our FreeBSD servers at work, I ran into the following problem with openssl-stable:

# cd /usr/ports/security/openssl-stable
# make clean
===>  Cleaning for openssl-stable-0.9.7m
# make
===>  Extracting for openssl-stable-0.9.7m
=> No MD5 checksum recorded for openssl-0.9.7m.tar.gz.
=> No SHA256 checksum recorded for openssl-0.9.7m.tar.gz.
=> No suitable checksum found for openssl-0.9.7m.tar.gz.
*** Error code 1
Stop in /usr/ports/security/openssl-stable.

The problem is caused by a recent change in openssl which deleted the checksum information for the stable version (0.9.7m) of openssl. The fix is pretty simple: just restore the checksum info for 0.9.7m to /usr/ports/security/openssl/distinfo. I’ve submitted a PR to the project which contains a patch file for folks who’d rather use patch(1) than copy-pasting three lines of text. ;)

13
May

Kyocera EP510DN printer

We got a new Kyocera printer at work a few days ago. The model we’re trying out is an “EP510DN“. We had a heck of a time finding information for this printer on the Kyocera web site. In desperation, we called their sales/support people who informed us that a Kyocera “EPwhatever” is an EcoPro unit, all of which show up on the KyoceraEcoPro domain. (Personally, seems like having completely different domain names for the same company is a good way to dilute your brand.) As soon as I added “EcoPro” to my Google query, I got meaningful results on the first page.

We are setting this up on a FreeBSD print server. We had DHCP and DNS configured and tested, so we created a printcap(5) entry for the printer, and got… errors (from the /var/log/lpd-errs log file):

lpd[32955]: /dev/lp: No such file or directory

After much head-banging, I finally figured out the problem–there was some whitespace at the end of one of the lines in the printcap. Doh!

Some other notes and information about this printer:

  • Web-based configuration doesn’t support https:// protocol. :(
  • The configuration username is “admin”. Default password is “”. Get IP configured on the printer, pull up the web interface, and change it–ASAP! (Ideally you would set the password on a private network, before putting it on the Internets. Not that it’ll do you much good–see previous note.) I’ve seen a pretty gnarly security exploit for Kyocera printers, which seems to bypass what little security is available, but I haven’t tested it on our unit yet.
  • The printer supports Bonjour protocol, although it doesn’t seem to be a “real” PostScript printer. (My Mac sees it as a “Generic PostScript” device.)
  • The EP510DN PPD file is available for *nix operating systems. I’m still looking for a Mac driver for this printer.

I don’t know if we’ll keep this printer or send it back. The print speed is definitely nice, and print quality seems alright. I’m a bit disturbed about all the network services that this printer provides, without securing them in any meaningful way. I’ll try out the cracking suggestion above and see if this model is vulnerable.

03
Feb

Doh!

I am so screwed… I ran the portupgrade utility on one of our FreeBSD servers at work. This was a production system that we used for printing certificates for our students, preparing address labels for USPS shipments, and several other small (but crucial!) database applications. When I realized that the upgrade had roached all these business-critical apps, I tried to restore the system and found that my backups weren’t usable. ARRRRGH! From now on, I’m going to verify my backups by generating a “table of contents” from the backup media after the backup is complete.

for ($i=1; $i<=100; $i++) {
    print "$i. I will *never* run portupgrade(1) again without making sure I have a *known good* backup!\n";
}



March 2010
M T W T F S S
« Dec    
1234567
891011121314
15161718192021
22232425262728
293031  

Tags

a wok for all seasons abramoff accessibility acetylene active directory activism adodb aerial photo afrofunk aim4tree air america airport express alite alito amtrak andante andy antenna anti failure any browser apache appetizer apple apple pie applescript arden arcade arden park ariana arnold artichoke ashland astronomy attic attika audio avid reader backup baen bart bash batch bathroom bbq bee beef begonia bgg bike bike hikers bike journal bikejournal biketalk bill oreilly bittorrent blacklist blog bloggercode blogroll blueberry boardgamegeek boardgames bonjour book bookcrossing boot failure bootcamp borderline chaos bugmenot bush business card butter cabinetry caira caitlin california callsign candamir caroline carpet cat cateye cdrom celebration cellular phone century cfengine cheese chicken chico chimney chinese chipotle chocolate christmas chumby city bikes civilization cjd claire club club fed co op coconut coffee comfort food community computer concord congress conquest conservation consumer reports copyright costco crazybikerchick crepe crock pot cruftbox css csus curl currency custard cvntrak dagoba daniel goetz darwinports database datarescue date format death delicious monster democracy now Democrat demolition dessert dhcp dickens diff disaster discrimination discworld dishwasher disk mirror diskutil diskutility diskwarrior dixie belle dock doors downing street memo drm dryer drywall dtv dump ed burke eff eggs eject election electric car electric truck electrical electricity elvgren emacs email energy efficiency eric meyer esr etext euthanasia facebook family faq faux news fcc feed feedreader fence finder firefox fireworks first4internet first amendment fishmeal flash flickr flooring fondue foreign exchange foundation fox and goose framing freebsd freedback freetds fsck fundraiser fyq garage garden geek geotag geourl ginger gnutar go gonuls google google earth google maps gpl gracenote graywater greek green group concat gtd gunthers gutters gwbush gymnastics hacking halloween ham ham radio handspring hardwood harry potter headmeta heimdal high speed photograph high tech inv hoax hobbies hobby hodgins holiday holly home power magazine horcrux houseblogs how to html humor hvac ilovejackdaniels imac impeach init inspection insulation intel internet iostat ipod ipodder iq iraq isight iso itunes ivins iwantsandy java city javascript jean reno jet lag joint chiefs joseph jott journalized blue judges cup juliette binoche kctc kerberos khs kimkerry kitchen kyocera la salles la times labor landscape laundry law leatherbys legal lego leopard lhc liberal library linkedin ljurban lodi logic logo lungren lynx maaloufs mac macaroni macbook mafc mailinator make.conf malware map marin markdown martin yan mason media media clip media reform meetup metric mexican microsoft mike comfort military milonic mindstorms mlk model railroading modoc hall monterey morning sedition motorcycle movie moving mssql munchkin murray mushroom music mynetflix mysql n3jim nagios naked lounge nerd score netflix network new urbanism new year New Yorker nofollow nut nutella oatmeal obama obcz occidental occupation omni openssl oracle organic osx owp paint pair palast palm panasonic parallels party patch peak adventures peak oil pear pedestrian village perl petaluma php ping pingback pingomatic pingomation pingqueue pirate pl sql planet bike plasma playroom entertainment plugin plugin:preview theme plugin:text control plugin:ultimate tag warrior plumbing podcast podsite politics portaudit portmaster ports portupgrade postfix potato powerbook Pozsars printer programming project gutenberg pryde punk puzzle qiana qrz quilt quilt shop quiz quote r5 records racism radio RAID railroad recent comments recipe redalt remodel repairmirror resaurant resolution restaurant restore review richmond rio americano robotics roger niello ron montana roof rootkit rss sacbee sacog sacramento sacramento spotlight magazine salad samba san francisco sandwich santa cruz scam scandal school days scplugin screen security session report shakespeare shuffle sigma siriradha sitemap slim jim snl snopes social social network software soma somafab sonicchicken sonicchicken blog sony spam spam karma spellcheck spinach spinner spirit spoiler etiquette spring hill cheese sql sqlserver stairs standards statcounter stephen king stop loss stucco stunt subversion sudo sudoku superbowl superhero supreme court sushi svn sysadmin tacoma tags plugin takara technology telephone television template terms and conditions textcontrol thai kitchen thanksgiving the well seasoned wok theater theme theme:borderline chaos theme:elvgren theme:elvgren wordpress google code theme:fluid blue theme:fullwidth theme:head theme:journalized theme:journalized blue theme:psycho theme:redoable theme:sharepoint like theme:simple green theme:spirit theme:steam theme:wuhan theme:yadda theme switcher thomas tigris euphrates tile tinfoil hat toc tofu tower theater training transit transit oriented development tribute u haul uneasy rhetoric university village unix ad notes ups van zant veal veterans day video vie viking hobby vote smart w3c wall warboss washer washington post weather web web developer web server wikipedia windows windows server 2003 wishlist woodworking wordpress workplace wpblacklist wpgatekeeper wsj xserve yaaarr yard