It's not all fun and games

I'm wearing my sysadmin hat today. Last night
I started an upgrade for trajan (Debian Sarge).
When I got in early this morning it wouldn't
boot, just hung at "LI". The usual panicky
minutes before verifying that the nightly
backups completed.

Next, attempt to boot up on Knoppix
and verify the filesystems are okay.

I attempted to rebuild LILO (yes, I use GRUB
for all the new installs):

# mount /dev/hda3 /mnt/hda3
# chroot /mnt/hda3
# /sbin/lilo

But Knoppix won't write lilo to /dev/hda,
even after I've given it write access. This
is one of the few times Knoppix has let me
down. Fortunately, I've got another Linux
Rescue CD with the 2.4 kernel. Booting up
on it finally allows me to rebuild LILO. My
decision process was much more time consuming
than appears above, but that was the general
flow. Stupid Jeff: I didn't have a rescue
diskette built, at least not one that was
readily available. A boot diskette would have
saved valuable time.

Unfortunately there are now some residual
problems from the upgrade. Python spews some
wierd linkage warnings (fixed by recompiling
the source), but Apache won't allow any users
to log on.

An awful lot of cursing/head-scratching later
reveals a couple of very obscure Apache bugs:

The gist of the matter is that I must swap the
order in which the following modules appear in
the configuration.

LoadModule auth_module /usr/lib/apache/1.3/mod_auth_ssl.so
LoadModule apache_ssl_module /usr/lib/apache/1.3/libssl.so

Also, I had to turn off SSLFakeBasicAuth, which
was on by default.

Quite a nerve-wracking morning.