LambdaMOO Status Blog

snowstorm expected

2012-01-17T16:05:00.002-08:00

So the various forecasters are talking about there being a huge snow storm starting tomorrow (Wed.) morning, which necessarily includes the possibility that snow-laden trees will take out our power supply. Hopefully this will all be a false alarm, but if not and LM happens to go down, that'll probably be why.

Seattle / Qwest network fail

2011-10-05T13:22:00.000-07:00

So we seem to be currently in the middle of a network outage ("Seattle area" and "Qwest CLEC circuits" almost certainly includes me and there isn't any other good explanation for why I haven't been able to ping anyone in the outside world despite having power-cycled the modem twice now).

It seems to have started sometime before 11:20am PDT this morning, still ongoing as of this post (1:20pm), and no ETA for resolution. More on this story as it develops. Or you can just watch this page yourself for updates.

Update: (5:45pm PDT) and... we're back. Bleah.

more network fun

2011-09-25T09:17:00.000-07:00

Looks like we had another random network outage around 4am PDT this morning, which I discovered when I came back from my sleep study around 8am. I also took advantage of the opportunity to try the reboot I said I was going to do at the end of the last message but never quite got to that day, and yay, it's getting the right network driver now, so no more worries about reboots that happen when I'm away from the house (or, at least, not because of this issue).

(Note to self: when changing things under /lib/modules, don't forget to rerun depmod.)

fun with network drivers

2011-09-18T18:13:00.000-07:00

Well, that was fun

We do what appears to be a routine upgrade (of Debian 6.0.0 to 6.0.2) and on reboot it decides eth0 (a.k.a. The Network Interface That Matters) doesn't exist anymore. Just in case you were wondering why we didn't come back right away after that shutdown today . . .

Note to self for future reference: kernel modules installed in /lib/modules/<version>/updates will take precedence over everything else and won't get overwritten by kernel upgrades (knock wood), which is what you need to do when a module that comes with the stable kernel (like, say, e1000e.ko) happens to be broken and you need to override it with a version that actually works.

I'm going to need to try another reboot later tonight just to make sure this is working the way I think it is (lie)

Getting settled in

2011-06-02T16:48:00.005-07:00

and the mail server is moved over. Next up: websites.

-20 to Gateway for making the power button insanely accessible; far too easy to press accidentally whilst one is leaning over the machine messing with the cables in the back -- the source of today's brief outage.

Seriously, what were they thinking here? I.e., why would you ever need to be able to get to a power button quickly? ("Captain! The warp core is about to blow!" "Commence EMERGENCY SHUTDOWN now before it's too late!!" and then everybody dies as the helmsman tries to get the childproof cover off of the... yeah, right).

I really hope there's a way to disable this in software...

In other news, ext3 filesystems are the best thing since sliced bread.

Update (Saturday 11:59pm): Done moving the database server and the web server (including the old ftp site content). Which means everything of note is on 'hagen' now. Yay.

New Box (was: Light at the end of the tunnel.)

2011-05-26T17:17:00.006-07:00

so having endured a few weeks of not being able to stay up for more than about 3 days at a time and still not a whole lot closer to figuring out what's really going wrong -- best guess at this point seems to have to do with the way it just completely freezes up and dies when the mail server gets heavily loaded, which suggests that maybe that overheating problem that Jay and I were worried about lo these many years ago is now finally biting us, i.e., one of the processors is getting too hot and misbehaving.

And since 'alberich' (server box) is now nearly 8 years old and doesn't owe me anything, it's not clear that this is worth trying to fix. So...

NEW BOX; just picked it up today.
Introducing 'hagen', Intel Core i3, quad-core, 3.2GHz, 6GB of memory.

Debian install is proceeding apace. The only small problem is
I have to get on a plane in about 6 hours. This may actually be enough time.

But if not, then we're stuck on alberich for another 5 days, in which case freezeups leading to multi-hour outages will remain possible...

Update (6:30am): and DONE! or at least the LambdaMOO part of it has been successfully moved over and is now running on the new box, and it looks like people are able to get to it. Enjoy.

Other services (notably the lambda.moo.mud.org website and mailing lists)
are still on alberich, will have to wait until I get back, and may possibly go down.

had to reboot again this morning

2011-04-30T11:45:00.003-07:00

So we were doing fine until a week ago, and then alberich (server box) has now randomly frozen up four times in one week. No idea what's going on yet -- I have a sinking feeling it might be a hardware issue (this box is now 8 years old...)

slog slog slog

2011-03-04T07:20:00.007-08:00

extended downtime hasn't happened yet but is going to.

suffice it to say, it's happening (finally). The box has been successfully flattened (cf. Tokyo/Dresden 1945) and there are a number of happy new CLEAN ext3 filesystems now. And it boots with a modern bootloader. And it can talk to the network. And /home is back. And various other things. So yay, so far.

At the current rate things are going, I don't think I should promise anything being back up before the end of the day.

Update 3/5 7am: well,... only missed that one by about 7 hours. But we are back on the air! And that is most likely it (knock wood) for the long downtimes and general flakiness -- at least until the next windstorm or something.

squeeze upgrade

2011-02-26T00:00:00.003-08:00

So... Debian 6.0 is out.

For various reasons I expect this upgrade to not be smooth and there will almost certainly be extended periods (read: hours) of downtime in the next few days.

Go outside; play in the snow, or something.

Update (3/1): still in progress, extended downtime hasn't happened yet but is going to.

and more wind

2010-12-18T12:29:00.000-08:00

big windstorm last night, plus lots of rain to loosen up the soil. A tree went down blocking Island Crest Way just south of us. Power was out for about 5 hours this morning. We're back on the air, but there may be more of this coming.

windy tonight

2010-12-14T00:40:00.002-08:00

and it's also been raining really hard the last couple of days, meaning lots of soft soil and power poles that can fall over.

And I do not like these power flickers that are getting closer and closer together in just the last few minutes.

it's that time of year again

2010-11-22T22:59:00.003-08:00

Seattle area just got its first major snowstorm. Between the sheets of ice on the road and the persnickety ABS on my Toyota, the drive home from dinner was extra-special fun.

However, for you folks, what matters will be the variously encumbered tree branches positioned over the powerlines leading houseward. We had one power fluctuation already tonight. Oddly, alberich (where the MOO server runs) managed to stay up throughout, but all of the other machines including the firewall and the DSL modem had to reboot; hence all of the dropped connections.

Anyway, expect other interruptions; this cold spell is apparently going to last at least a few days. The school district has already decided to punt on tomorrow.

Downy Softness

2010-03-23T17:13:00.000-07:00

So yeah, we're down. Rog's on it. Do Not Panic.

~H

server panic

2010-03-23T16:54:00.003-07:00

we know exactly what caused it. working on a fix.

yes, we've been having windstorms

2009-11-16T23:39:00.003-08:00

I tend to think that when they warn about 40-50mph gusts it's not such a big deal, but we did get a 2-second power glitch this evening during which time the hardware clock mysteriously gained about 2 hours -- given the interval, I am less certain that this had anything to do with DST or UTC-vs-local nonsense this time. Buh.

Fortunately I was sitting right there and caught it in time to not have to do (yet another) timewarp. Yay.

Update:
Well okay, having looked back over the last five years of blog and seeing that we were doing a timewarp pretty much every fall for at least 3 years in a row -- missed last year for some reason -- and having also seen that my noting of the hardware clock being on localtime and not UTC was not actually accompanied by some kind of assertion that I was going to fix this as I thought I had. And lo and behold, after grepping the init scripts for every instance of hwclock I found a setting in /etc/default/rcS that I hadn't seen before that actually addresses the problem and was set the wrong way. Whee.

Also found out that the hardware clock, which is only used to keep time while the power is out, apparently only ever gets synched when the system does a clean shutdown. So if every shutdown for months on end is a power failure, all bets are off. While things up and running, hwclock can be wildly wrong and nothing will notice or care. So now it's making a lot more sense why a power failure, say a month after the DST switchover, results in an hour jump.

Anyway, we now definitively have the hwclock on UTC (rebooted while the network was down), so there is now a high level of confidence that we won't be having to do timewarps ever again (knock wood... I suppose it probably would also be nice to know why NTP wasn't taking care of this ... project for another day...)

Power outage

2009-09-06T20:33:00.003-07:00

We had some interesting rainstorms today; power has been out since about 10am this morning -- fairly small outage affecting about 25 customers, or so the recorded message claims. Said message also claims that things will be fixed by 1pm, and is now about 7 hours out of date. Yay.

According to the actual human we just got through to, they had some "equipment failure" and are still working on things. Current estimated time of completion is 10pm (PDT). We'll see how that goes.

(Also, even if they do succeed in getting things fixed by then, there will likely be some delay beyond that point, because we're not currently home at the moment, and the circuit breaker on that room is currently off because I have an instinctive mistrust of my surge protectors.)

8/11-12/2009 maintenance me harder

2009-08-03T02:07:00.003-07:00

On Tuesday evening, August 11th and Wednesday morning, August 12th 2009 we will be performing scheduled maintenance on the Seattle, WA Point of Presence (POP) through which your Broadband connection is routed. Maintenance will begin at 11:00 PM PDT Tuesday evening and end by 5:00 AM PDT Wednesday morning. While you may experience a service disruption during this time, it is unlikely that your service outage will extend through the entire time frame. Service interruptions during a maintenance event like this often last only a few minutes.

Network maintenance 7/22/2009, midnight-1am PDT

2009-07-21T21:39:00.002-07:00

Just got this from Speakeasy:

On Wednesday morning, July 22nd 2009, we will be performing emergency maintenance on the Seattle, WA Point of Presence (POP) through which your Broadband connection is routed. Maintenance will begin at 12:01 AM PDT and end at 1:00 AM PDT on Wednesday morning. While you may experience a service disruption during this time, it is unlikely that your service outage will extend through the entire time frame. Service interruptions during a maintenance event like this often last only a few minutes.

You now know as much as I do...

network maintenance, June 10-11, 2009

2009-06-03T12:42:00.003-07:00

latest missive from Speakeasy

On Wednesday evening, June 10th and Thursday morning, June 11th 2009 we will be performing scheduled maintenance on the Seattle, WA Point of Presence (POP) through which your Broadband connection is routed.

Maintenance will begin at 11:00 PM PDT Wednesday evening and end by 3:00 AM PDT Thursday morning. While you may experience a service disruption during this time, it is unlikely that your service outage will extend through the entire time frame. Service interruptions during a maintenance event like this often last only a few minutes.

Thank you in advance for your patience as we continuously work to improve our service to you.

and now you know as much as I do.

power outage

2009-05-01T17:29:00.003-07:00

Driving home I see a bunch of fire trucks and police cars clustered around on Island Crest Way, with lots of blinking lights, and guys standing around with binoculars looking up, a block or so away from the house. No obvious signs of creative driving but maybe I got there too late to see the tow truck.

Get home, power is out. Surprise.
Anyway, back up now.
Thus concludes today's fun for the day (we hope).

upgrade complete, sort of

2009-03-19T16:32:00.003-07:00

and I managed to do it with just one reboot.

Well okay, that's sort of a lie because I'm thinking now might be the right time to switch over to Grub (what with all of the Debian docs saying "WTF? You're still using LILO? .. well let's just say you probably won't want to be by the time 6.0 comes out..."). So there are probably a few more reboots in the cards if I go through with that.

But not until I finish picking up the other pieces that are a bit more urgent --- funny how it's never what you expect is going to be difficult (short story: none of the horrible things that the debian docs were predicting for the reboot -- in particular, the possible arbitrary renaming of disk devices that would make it difficult to find the root directory, LILO getting blown out of the water by a too-large kernel image, and about 10 other fun possibilities -- came to pass -- knock wood). Instead it's various annoying changes to the mailserver config. And I get to figure out why X doesn't work anymore. La la la... Not Your Problem, as they say (and figure that'll keep me out of trouble for at least a day or two while y'all can get back to MOOing...)

upgrade-o-rama

2009-03-18T23:58:00.004-07:00

Apparently, lenny has been released, which then means I run this magic script and then everything becomes ever so slightly different, presumably Better, but we'll see.

For more fun, this is another one of these "we need to bump the kernel, too" upgrades -- and there's specifically stuff about how they now deal with the AMD k7 models differently (the new world lumps them into the rest of the 686 boxes).

Which means we've probably got at least 2 reboots before everything will be totally happy. This being debian, everything has undoubtedly been thought through most carefully and things will Just Work. Or maybe not. Heads up.

Right now I'm running El Grandemundo Backup and reading through the release notes one more time. Figure sometime in the next 48 hours I will amass the courage to finally throw the switch.

and of course today is a completely sunny day

2008-12-29T17:03:00.000-08:00

and most of the snow has melted,

so I have no idea why the power went out...
... but it did.

weather's been especially sucky this week

2008-12-24T18:01:00.005-08:00

So, yes that was a power outage. The main problem at this point is that most of the back streets are still covered with a few inches of packed snow and ice, which in the past 24 hours has been trying to turn into slush and will most likely be solid ice by tomorrow.

Overall we've had about 10 inches or so over the past week, with no intervening warming spells to melt stuff -- which is quite unusual for this area -- never mind that a mere half inch of dusting is usually enough to turn the Seattle area into a useless traffic snarl. And there are very few plows capable of dealing with this, so even a week later, only the main arteries are anything approaching vaguely cleared

Thus, while the outage itself was relatively minor (affecting 100 houses, not like the civilization-ending disaster we had two years ago), it still takes Them a while to actually get to whatever pole has fallen over (or, more likely been knocked over by one of the 500,000 people around here who don't know how to drive in snow). Luckily there were apparently not very many of these at the same time, so they were able to get to ours in a mere four hours.

Current forecast is for more snow, changing (finally!) to rain sometime this weekend with next week to be consistently above freezing. We'll see how that goes.

Update (8:41pm PST): ... down and up again. Save early and often, I guess.

Raining now. Of course, I've just been reminded that when you have rain falling on top of heavy accumulations of snow on top of not-entirely-sturdy structures like decks and carports, said structures tend to collapse. And maybe knock over power poles.

And so the fun continues

We had a power failure today

2008-07-07T18:29:00.002-07:00

sometime after 3:11pm PDT (which was the last checkpoint).
but now we're back.
yay.

webserver

2008-02-05T04:53:00.001-08:00

so our faithful webserver of nearly 8 years finally bit the dust. Just in case you were wondering why you couldn't get to www.lambda.moo.mud.org or ipomoea.org or any of the other sites I run. This was basically the last of the services that remained to be moved from wotan to alberich, and wotan didn't last quite long enough -- probably wouldn't have ever lasted long enough seeing as as long as wotan stayed up, there was little reason to bother. Oh well.

I'm now learning about all of the differences between Apache2.0/Windows and Apache2.2/Debian-Linux. Fun.

Since the lambda site is all of one page, it's one of the easier ones to restore...

network outage

2007-07-30T18:13:00.000-07:00

yes we had another network outage today from about 9am to 2pm PDT.

(the usual story: ISP stopped responding to ping, reboot firewall, powercycle modem, wait a few minutes, then everything was happy again. No idea what actually failed.)

and yes this is more of a log-for-posterity post rather than something intended to be actually useful; sorry.

Server panic - 3rd time's the charm

2007-05-29T05:40:00.000-07:00

I guess the good news is that bug is evidently reproducible, at least in some sense of the word.

and it reproduced itself again the other day.

But I believe I've nailed it -- or rather I found something definitively broken that appears to account for what happened, and now we're running with the fix. As usual, we'll see how it goes.

upgrading to etch

2007-05-24T09:44:00.000-07:00

so Debian 4.0 is out, and I'm doing the nasty (one of my other machines died and so there's now a bunch of stuff I need to move to the moo server box which made the upgrade suddenly a bit more urgent, which is why you didn't see much advance notice on this).

And normally with Debian upgrades you're supposed to be able to keep things running throughout,which is way cool when you can do it. Except that this time it's not, since for various reasons the sarge->etch transition involves two kernel upgrades (one to get from 2.4 to 2.6 and the other to get from a 2.6 that sarge knows about to the 2.6 that etch knows about -- there's no overlap, see... oh well, it's free...) and some of the packages being altered are things like libc, which the moo server actually depends on (surprise), and after a few distubing "Waah, I can't find such and such in foo.so" messages because the process was suddenly bereft of its shared lib, I decided maybe I ought to shut it down for the duration... (I hope we got a clean checkpoint, otherwise we'll have lost about an hour; actually since it lookied like DNS stuff that was crashing and that's all a separate process anyway, I'm guessing things are fine, but we'll see how that goes).

Current task is dealing with hdparam (flashback to 2003: Jay sez, "Yes, you really do want to tune your hard drives. It'll be totally excellent...", he gives me the one-line command that does the trick; I wrap it in the 20 line init.d script one needs so that it properly fits into the boot sequence, and we're done. And then I go on to spend the next four years thinking about Other Things.

And now it seems that etch has its own script that's about to overwrite mine; it's only 300 lines and the corresponding config file is only about 150. Most of it is about RAID admittedly, but somewhere in there is the magic option to set that will produce the single line that actually does whatever it was that we evidently needed to do back in 2003. And I kinda wanna get it right the first time, because,... well, ...disk.)

so that's where I am at the moment. I like to think this won't take too much longer.

Meanwhile, it's a very nice day outside (at least here in the PNW anyway). Enjoy.

Update (1:48pm PDT): we made it through the second kernel reboot and still no evil on the console. yay. And of course, X is now b0rken (after only just having gotten it to work under sarge just a few days ago), but hey, not your problem, as they say...

Server panic deja vue

2007-04-06T11:07:00.000-07:00

I guess the good news is that bug is evidently reproducible, at least in some sense of the word.

Server panic

2007-01-23T19:00:00.000-08:00

woo. first server panic in a while. The short story is something bad happened in notify() at 9:25pm (PST) last night, but I don't know much beyond that at this point.

Meanwhile, we've reverted to a 9:19pm checkpoint.

back on the air

2006-12-21T00:38:00.000-08:00

I'll believe it when I see it

Okay, I believe it now. Wires are still down in the road at 62nd, but evidently those ones are not a prerequisite for us having power, at least so far as I can infer from the fact that we do in fact have power now. And internet. And everything else that matters.

Machines seem to have survived, so we're now back up.

Just in time for storm #2 (actually, for us, this one is looking to be Just Rain, so far [knock wood]).

progress report

2006-12-19T13:03:00.000-08:00

Still no work done yet on the tree that went down at 63rd other from clearing a lane so that the traffic can get through. This is one of three sets of lines (that I know about) that are down in our neighborhood; the furthest appears to have been fixed (yesterday), or at least doesn't still have wires supporting a tree trunk dangling 6 feet off the ground above a bunch of orange cones. For the second set, they'd gotten rid of the tree back on Friday, but the wires are still flat on the road with people driving over them. Granted, I don't know for sure which wires are prerequisites for our getting power back -- there is some surprising connectivity in places -- but I'm guessing it's not a bad measure of how much has been done and how much remains.

There are still something on the order of 150,000 people without power in the Puget Sound area. November was one of the rainiest months on record, thus the soil was nice and soft, meaning the storm arrived at the worst possible time. And now, apparently They are having to rebuild everything. If you imagine that the storm explicitly targetted and destroyed every substation and transformer pole east of Seattle, you wouldn't be too far off.

In other news we're fine and the house appears to have emerged unscathed. Temperature has been threatening to go below freezing but that hasn't quite happened yet, so the pipes are intact, and hopefully the leaky toilet will keep things that way. And even though the furnace needs electricity for the blower, the gas hot water heater does not.

It could be argued that we lucked out in having our biggest tree go down six weeks ago; it was about 50 feet tall and landed in the driveway, missing the house and both of the cars -- the idea being that if it had survived until last Thursday, when the wind was coming directly from the south, it would have been blown straight into the living room (which, I'll grant, is not actually where LambdaMOO is, but this still would have been cause for some sadness...)

Current word is maybe Friday. At this point, I'll believe it when I see it.

There's also evidently another storm on the way, albeit a much smaller one, though I like to think that all of the trees that are currently inclined to go over would have already been pushed by the big storm and thus already be gone, so...

Update (3:51 pm PST): woohoo. Tree at 63rd has been cleared. The wires there still look dodgy, but one thing at a time... Wires lying in the road on Island Crest at 62nd are still there.

No news is bad news

2006-12-18T17:40:00.000-08:00

emma (or is she chloe on lambda?) posts from her local library, "It doesn't look like it's going to be up any time really soon, as there are two major lines down within a few blocks of our house." And it's very cold with no heat, as well. Send warm thoughts to wrog and emma/chloe!

Serious power outage!

2006-12-15T15:12:00.000-08:00

Just talked to wrog on the phone. Things are very grim in his neighborhood -- a large tree is down a block away and took out some power lines. Essentially his entire town has no power and things are worse right where he is. So, be happy if it's less than "several days".

Get a life :-)

still windy

2006-12-14T17:02:00.000-08:00

There have been some disturbing power flickers in the last half hour. Just so that you know.

Update (9:40pm): Looks like I called that one right. We got hit again at around 8:50. Whee.

windy today

2006-12-13T14:23:00.000-08:00

So we had two (2) power failures in rapid succession: 1:47pm (PST) for about 7 minutes, then things were up for 6 minutes (but no net) and then back out again at 2:00pm and stayed out for another 9 minutes. (the combined outages would most likely have toasted the UPS if we still had UPS, though, yes, I should still do something about that...)

The fun part is that after the first power failure, alberich's clock went to 9:56pm. Apparently, the BIOS is now indeed on UTC (as intended) but something in OS Land wasn't expecting that. And of course, LambdaMOO dutifully started up 8 hours in the future, though the external network was down, so nobody got a chance to get in and look at tomorrow's newspapers or anything.

Then the second power failure hit and killed it before it could checkpoint. On the 2nd reboot, the OS finally got its act together and set the clock correctly. So none of that ever happened and we dodged a small bullet, at least as far as LambdaMOO is concerned (and with UPS we'd have only gone down once and then things would have stayed confused. so see? it's good that we didn't have UPS! ...)

Admittedly, other daemons on alberich may well be confused for the rest of the day, but hopefully they'll cope (if not, then you'll be hearing yet more about this,... whee...)

impending time displacement

2006-11-25T03:27:00.000-08:00

Yes, we had a brief power failure the other night (11/23).

And, of course, when the system came back up it had gained an hour (looks like the BIOS clock is not in UTC after all... what fun). And of course setting clocks backwards is a big nono... tends to massively confuse certain applications (e.g., the moo server).

So I think I'll just play it safe and take the machine down for an hour, tentatively tonight 11/26 12:01am to 1am PST

Update (11/26): well, that was fun. Some games with tickadj and we didn't have to go down after all (something to remember for next time, except there shouldn't be a next time, but anyway...)

line problems

2006-08-01T23:50:00.000-07:00

For the past few hours, we've been getting lots of packet lossage (70-90%). DSL signal would keep dropping, come back up, last for about 10 seconds, then drop again; wash, rinse repeat. Speakeasy talked to Qwest and they poked around for a bit, and suddenly things are fine again.

But they're still a bit mystified, so sometime tomorrow a tech is coming out to check the line. We'll see what happens.

Update (8/2): tech didn't find any problems between service panel and central office, but did notice that the green circuit (internal line where the DSL modem was living) was "off hook" (which isn't supposed to actually affect DSL signals, but does mean that somewhere on that line, there's a at least a short and perhaps other badness as well). So now the DSL modem is on the brown circuit, which is arguably a better place for it anyway, seeing as way fewer jacks connect to that one.

Scheduled Network Maintenance 8/3 (midn-3am PDT)

2006-07-28T05:13:00.000-07:00

This just in:

On Wednesday night/Thursday morning, August 2nd and August 3rd 2006, we will be performing scheduled maintenance on the Point of Presence (POP) through which your Broadband connection is routed.

Maintenance will begin at 11:59 PM PDT on Wednesday night and end by 03:00 AM PDT on Thursday morning. While you will experience a service disruption during this time, it is unlikely that your service outage will extend through the entire time frame. Service interruptions during a routine maintenance event like this often last only a few minutes.

Back up

2006-06-26T17:43:00.000-07:00

Looks like our network name problems are resolved!

Update on DNS

2006-06-26T13:17:00.000-07:00

mud.org is back in business, but now there's some new problem with lambda.moo.mud.org. I've emailed wrog. Hopefully he'll get it and be able to deal with the problem soon.

LambdaMOO DNS lossage

2006-06-25T20:30:00.000-07:00

The mud.org domain is nonfunctional, thus LambdaMOO is unreachable except by direct IP address (which is 69.17.114.80) until this is repaired. Some bureaucratic screwup; the owner of mud.org states it is paid up until 2007, but of course bureaucracies don't work on weekends. Hopefully tomorrow it'll be taken care of.

Update (wrog): In the meantime you can use lambdamoo.fyigm.com

power outage

2006-06-09T16:13:00.000-07:00

we had a blackout around 1:45 or so.
this one was a little odd in that there was no accompanying boom from the street outside, no storm activity or anything else. And once we got through the phone queue and actually started talking to Somebody about it, it mysteriously came back on again. I'm guessing somebody tripped over a cord in the central office or something.

Looks like everything came through with flying colors this time.

network outage

2006-04-13T21:30:00.000-07:00

we appear to be cut off at the moment (down for about 20 minutes so far). Speakeasy's network status board is blank, so either this is something extremely local or extremely new. More when I finish putting Pip to bed.

Update (12:10am 4/14): we're back. It seems our circuit needed to be "rebuilt", or at least, a "rebuild" fixed it. yay.

We have Hail

2006-03-10T18:11:00.000-08:00

and the power went bip at around 6pm PST.

Fortunately this time, no network cards seem to have been fried. I would like to think that, as natural selection takes its toll and the old/diseased transformers are culled from the herd, the power grid will adapt, survive, and become all the more stronger for its experience. Or something.

And yes, I should fix the UPS, which would have made a difference this time. I suck.

Network Outage (2/17)

2006-02-17T10:04:00.000-08:00

external network seems to have gone down sometime around 8:52am PST today (Friday 2/17).

I've done all of the usual tests and am still not getting ping from the gateway.

Speakeasy's status page is not as yet showing an outage however their callcenter is reporting a "high volume" and I waited on hold for 30 minutes without getting anybody. so I'm guessing Something Bad is going down and they're in Headless Chicken Mode at the moment.

Update (11:30am): yay, we're back.

LambdaMOO IP address

2006-01-26T20:18:00.000-08:00

We apparently had some DNS trouble today. Sorry about that. For future reference, LambdaMOO's usual IP address is 69.17.114.80. Thanks to Jaybird for the suggestion to post it here.

Scheduled Network Maintenance (1/12 mid-4am)

2006-01-09T03:34:00.000-08:00

This just in from Speakeasy:

On Wednesday night/Thursday morning, Jan 11th and Jan 12th 2006, we will be performing scheduled maintenance on the Point of Presence (POP) through which your Broadband connection is routed.
Maintenance will begin at 12:00 AM PST and end by 04:00 AM PST on Thursday morning, and while you will experience a service disruption during this time, it is unlikely that your service outage will extend for the length of the maintenance time frame. Routine maintenance events such as these often have service interruptions that last just a few minutes.

bad UPS, no biscuit

2005-12-11T15:22:00.000-08:00

so the power went out. It was only for about a minute but the UPS turns out to be hosed as well. Everything's plugged directly into the wall now... meaning there'll eventually be more downtime when the UPS is fixed/replaced -- more on this story as it develops.

network was also out for a while -- DSL modem evidently doesn't like power failures either.

And for more fun, alberich's clock mysteriously jumped forward an hour once again apparently this only happens on power-on reboots, though this time I managed to catch it before LambdaMOO got confused (and if anything else is, hopefully that'll sort itself out in the next 10 minutes or so...)

Time Warp

2005-11-01T12:50:00.000-08:00

On November 1, yduJ reported that the time seemed to be off by one hour. Rog bashed it back. Shortly thereafter problems started occurring, apparently involving idle seconds. Krate attempted to figure out what was going on, but while he was doing so, it crashed.

We lost 6 minutes; we decided not to use the panic database.

DSL is down - no joy

2005-10-31T10:45:00.000-08:00

When I woke up this morning all of the machines had rebooted. However my alarm clock, which goes blinky even if the power is out for just a few seconds, was steady. so I'm guessing something wonky happened with the UPS -- it seems to be fine now, but I'm still disturbed.

Been on the phone with speakeasy. No network issues on their end. On my end, even with a laptop plugged straight into the DSL modem plugged straight into the wall, there's still no ping, even though the DSL light is showing green. QWest apparently tried rebooting the dslam and still no dice.

So nobody has any clue what's wrong apart from the possibility that modems apparently can fail in strange and unfathomable ways and still show green lights.
They're sending out a new modem which should arrive tomorrow morning.

Meaning we're off the air at least until then.

Update 1:47pm: Well, well. Looks like they found something to fix after all. We are back up.

yoyo mode

2005-06-13T16:51:00.000-07:00

OS upgrade (to Debian 3.1) is in progress.

Current status: packages appear to have upgraded sucessfully; now to reboot.

Update 5:42pm: even though they said there's no kernel upgrade in this part of the process, something decided to overwrite the boot block anyway to force booting from the CDROM and the CDROM seems to only know how to do a fresh install. meaning this is going to take longer than I thought. bleah.

Update 7:15pm: ok, we have our old boot block back. certain other things look decidedly wonky, but we're up for now.

Scheduled network maintenance (6/16/2005 12mn-3am)

2005-06-09T01:57:00.000-07:00

This just in:

On Wednesday night/Thursday morning, June 15th and June 16th 2005, we will be performing scheduled maintenance on the Point of Presence (POP) through which your Broadband connection is routed. Maintenance will begin at 12:00 AM PDT and end by 03:00 AM PDT on Thursday morning, and while you will experience a service disruption during this time, it is unlikely that your service outage will extend for the length of the maintenance time frame. Routine maintenance events such as these often have service interruptions that last just a few minutes.

As usual, this is all I know.

Update (6/15): This appears to have been postponed. More later.

More emergency network maintenance (2/16 12m-4am PST)

2005-02-15T20:43:00.000-08:00

This just in:

This maintainence will be performed on Tuesday night / Wednesday morning, February 15th and 16th, 2005. As a result, your service will be interrupted for up to 4 hours, beginning at 12:00am PST Wednesday Morning and ending by 4:00am PST Wednesday morning.
We apologize for the short notice of this maintenance. Whenever possible, we notify you of scheduled maintenance seven days in advance.

I guess whatever it was they worked on last week didn't quite get fixed.

Update (4:51am 2/16): Hmm. Network seems to have come back up at around 4am but fell over again at around 4:20. Bleah. Status page says "The Seattle POP is undergoing emergency maintenance to correct instability issues. Outage is expected to last ... until 4am Wednesday". Bedtime.

Update (2pm): Looks like things finally came back around 6am. Oddly enough, the status page still has this in the "unresolved" column; whether they forgot to update or they're still working on something is anybody's guess.

Emergency network maintenance (2/9 11pm-3am PST)

2005-02-08T18:23:00.000-08:00

This just in:

On Wednesday night / Thursday morning, February 9th and February 10th 2005, we will be performing emergency maintenance on the Backhaul circuit through which your Broadband connection is routed. As a result, your service will be interrupted for up to 4 hours, beginning at 11:00pm PST Wednesday night and ending by 03:00am PST Thursday morning.

You now know as much as I do.

Update (12:06am 2/10): and whaddayaknow, we have indeed been interrupted. Status screen says, "We will be performing a code reversal on the Seattle ERX in order to resolve a conflict which has resulted in recent intermittency." Woo.

Update (2:30am): and we're back.

Update (4:30am): still seems to be a bit of weirdness. "Post scheduled maintenance we are seeing some reports of sync no surf and are actively investigating." urk...

Scheduled network maintenance (1/19 11pm-3am PST)

2005-01-13T16:06:00.000-08:00

Looks like it's that time of the month again:

On Wednesday night / Thursday morning, January 19th and 20th 2005, we will be performing scheduled maintenance on the Point of Presence (POP) through which your Broadband connection is routed.

Maintenance will begin at 11:00 PM PST and end by 03:00 AM PST on Thursday morning, and while you will experience a service disruption during this time, it is unlikely that your service outage will extend for the length of the maintenance time frame. Routine maintenance events such as these often have service interruptions that last just a few minutes.

so I'm guessing they're not planning to do a whole lot this time.

Scheduled network maintenance

2004-12-01T13:26:00.000-08:00

Latest email from Speakeasy:

On Wednesday night / Thursday morning, December 8th and 9th 2004, we will be performing scheduled maintenance on the Point of Presence (POP) through which your Broadband connection is routed. As a result, your service will be interrupted for up to 4 hours, beginning at 11:00PM PST and ending by 4:00AM PST Thursday morning.

Thank you in advance for your patience as we continuously work to improve our service to you.

More on this story as it develops.

Update: (12/9, 2:15am PST)
We were cut off from about 1:15 to 2am. System status message says "This work will ... conclude on or by 2am local time," from which I infer that they're done now.

Network Outage

2004-11-17T22:06:00.000-08:00

Looks like we've been disconnected since 8:02pm PST. Speakeasy status hotline reports, "We are experiencing a problem with the DS3 servicing QWest-partnered customers in the Seattle area."
No ETA on a fix, yet.

Update: (10:27pm PST)
"After remote troubleshooting, a technician has been dispatched to the Central Office. ETA for arrival is 1 hour."
Still no ETA for repair.

Update: (Thu. 1:55am PST)
"Outage issue has been isolated and repair has been implemented at this time." I suppose it won't be too long now, but they're still not giving an ETA.

Update: (Thu. 2:40am PST)
Heyhey, we have ping. And I see people connecting again. Yay.

Bedtime pour moi.

Made it

2004-11-12T16:53:00.000-08:00

Things seem to have survived the trip and we are back up at the new IP.

(DNS appears to have updated, but see previous post about what to do if you're not getting the right (69....) IP for lambda.moo.mud.org)

Update: FTP site and mail are back as well.

Road trip

2004-11-12T15:56:00.000-08:00

Server box is now on the road. If all goes well, we'll be back up in about 30-60 minutes.

There will be a new IP address.
lambda.moo.mud.org's old DNS entry is set to time out about 15 minutes from now, so you should just be getting it automatically. But if you still find it resolving to the 67. address and you can't convince your DNS cache to flush itself, try new.lambda.moo.mud.org

Firewall rebooted

2004-11-06T15:11:00.000-08:00

Apparently, this time it had gotten into a state where old connections were okay, but new connection attempts were losing (and it had stopped responding to pings).

First post

2004-11-06T01:17:00.000-08:00

Hm. That was easier than I thought it would be.

And yes, we're still up, so far as I can tell.