Site Outage

Anything Goes (within reason)

Moderators: FAOA Administrators, FAOA Moderators

Post Reply
User avatar
Firehawk
Posts: 1234
Joined: Fri Jan 11, 2002 5:19 pm
Location: Indianapolis, IN
Contact:

Site Outage

Post by Firehawk » Thu Aug 30, 2007 8:49 pm

Sorry for the outage, everything should be back to normal.

Will explain more later tonight after I get something to eat.

Brian
Last edited by Firehawk on Thu Sep 13, 2007 1:26 pm, edited 1 time in total.

User avatar
Firehawk
Posts: 1234
Joined: Fri Jan 11, 2002 5:19 pm
Location: Indianapolis, IN
Contact:

Post by Firehawk » Fri Aug 31, 2007 12:46 am

[SHORT NON-TECH VERSION]
Brian hosed the box and spent 24+ hours trying to fix it.
Its fixed and back to normal now.
[/SHORT NON-TECH VERSION]

So here the longer story of that happened.

Yesterday the server was getting hammered by some machine over seas trying to send a boat load of mail through it.

I setup some filtering, but found a problem in sendmail that would allow that machine to make things even worse. I figure it was time to finally setup the firewall on the server, but found I didn't have the kernel built with firewall support. Turns out it was built as a shared library and I stupidly tried to load it (mistake #1). Trying to load it made the kernel panic and lock the box.

I got a hold of the colo guys and got them to reboot the box. It takes a while to boot since it has to check 500G of disk when it is not rebooted cleanly. It boots up and everything is looking good and then it disappears again. Turns out the colo guys didn't see it come up the first time and figures it needed another kick and rebooted it again (mistake #2). When it cam up this time it checked the 500G and found minor problems, but wanted human intervention.

My machine has serial console access enable, this mean that I can get all the boot up messages and control remotely (just like being at the keyboard and monitor). It seems the boot loader I am running has some bugs because there is no way to redirect the output from the serial console back to the keyboard and monitor output. This normally isn't a problem, but currently it is because console access isn't working right now (mistake #3). So the machine was booting up, detects the machine wasn't cleanly rebooted and checks the disk. While checking the disk it finds a few problems which is enough to force human intervention, but it is sending the prompt to the console port (which isn't working) instead of the keyboard and monitor.

So I have spent the last 24+ hours trying to figure out how to disable serial access and get keyboard and monitor working so I could literally type:

Code: Select all

enter
fsck -y 
y
y
and wait 20mins while the disks are being checked and then reboot the machine cleanly.

Needless to say, it has been a long day.

Brian

The Silverdude
Posts: 56
Joined: Tue Oct 03, 2006 1:56 pm
Location: Severn, MD

Post by The Silverdude » Fri Aug 31, 2007 7:08 pm

Bummer about those problems! What OS are you running? I'm guessing something Linux based? Hopefully you have the firewall going now, and those mad overseas spammers won't be taking over the site again!
451 RWHP Black 2002 Trans Am Firehawk # 1133, 383 LS1, ported/polished heads, headers, American Racing longtube, and a bunch of other cool stuff...

Matt B
Posts: 784
Joined: Tue Mar 19, 2002 4:01 pm
Location: Des Moines, Ia

Post by Matt B » Fri Aug 31, 2007 7:47 pm

Don't worry about it Brian.. we appreciate your time and effort to keep it all going!!!

Matt

User avatar
Firehawk
Posts: 1234
Joined: Fri Jan 11, 2002 5:19 pm
Location: Indianapolis, IN
Contact:

Post by Firehawk » Fri Aug 31, 2007 8:48 pm

The Silverdude wrote:What OS are you running? I'm guessing something Linux based?
Running FreeBSD. At the time it had the best 64bit and multiple processor support. Used to run Linux (and still do at home), but need to take another look at it.
The Silverdude wrote:Hopefully you have the firewall going now, and those mad overseas spammers won't be taking over the site again!
Nope, no firewall. I really don't need it as I think I will just hack sendmail to do what I want it to. Currently when I configure sendmail to deny connections from a ip address it will be nice and still allow it to connect, but deny all commands. I think I will change that to just deny the connection all together and not let it consume any resources.

Brian

Post Reply