Seattle Area Wind Storm

Actually, it's really more of a big ass wind storm sitting off the entire West Coast of ths US (short version: You don't want to be in a small craft in the water anywhere between Mendocino and British Columbia right about now). What is extra-special fun is this one involves unusual winds from the east that are more likely to confuse the trees and make them fall over.

And I'm now seeing the sequence of brief-but-getting-longer power flickers and sounds of pole transformers blowing up that usually converges to Out For A While. Just a heads up.

NOAA says things are going to remain At Least Annoying until around 4am.

Update (10pm):  Looks like we finally got nailed. 300K customers without power so this will indeed probably be a while...

Update (1am, Wed):  "As of 1 a.m. we have 474,032 customers out of power. We anticipate a multiple day outage...". Well, then...

Update (7pm, Wed):  Latest ETR for our area is 8pm, 11/23,
...as in Saturday (!!)

Update (4pm, Fri):  Outage map now says they're down to 118,540 customers without power. But that still includes us. At least one neighborhood within a half-mile now has a crew assigned, so that's something.

Update (7:30pm, Fri):  Looks like we are back up. It's a bit confusing since, from the logs, it looks like the machine rebooted at around 3:15pm but we evidently didn't get the network back until about 15 minutes ago. Anyway, enjoy.

phone line work

So they're working on our phone line.

Evidently there's a short of some kind in some cable out in the neighborhood that's been preventing dial tone from happening (that we haven't noticed because we mostly don't use the land-line anymore) that's evidently not been affecting the internet service (and right now they're doing experiments and internet still seems to be up [guy says, "It's on a different frequency"]) -- and hopefully it'll continue to stay up. But here's a heads up just in case something stupid happens.

Update (6:30pm):  And done. We have dial tone. And the network never actually got disconnected, so yay for that.

Power Failure

Power's been out for about 15 minutes now. There's been heavy rain the past few days and it looks like they're dealing with rather a lot tonight, so I'm guessing this one may take somewhat longer to come back. We'll see.

Update (1am):  They're now giving me an ETA of 2:30am.

Update (6am):  There are now 380 outages affecting 25,000 people in the Seattle area. Last text message status update said, "Work is underway" (no further details), but PSE website's outage map is saying ours is still in the "Outage reported" phase (no cause located, no crew assigned). Looks like this is going to be a while.

Update (6:55am):  "Assessing damage." They've apparently gotten as far as figuring out that "Trees/vegetation" had something to do with it. Progress!

Update (8:20am):  "Crew assigned" + activity observed at a plausible location while driving W to school. Now down to 15,900 customers without power.

Update (9:10am):  ...and we are back. Yay.

server panic

So... this was not your imagination:  We did, in fact, have a server panic on Saturday around 3:30pm, and I didn't actually notice until Tuesday. Suffice it to say, I need to set better alarms for this sort of thing. I also evidently let myself get distracted between bringing it back up and making the obligatory blog post. Not my best week.

Anyway, something stupid happened in a verb_args() call, and that's as much as I know at this point. I've turned up the checkpointing frequency to once per hour (it was at 24 hours and we lost about 23 hours of activity).

For future reference, if you're noticing LM down and you're not seeing anything about it here and it's been more than an hour (+ usual allowances for sleep if it's middle-of-the-night PST), you should feel free to DM me on Mastodon (@wrog@mastodon.murkworks.net) or Failbook -- sad to say, these channels seem to be more reliable than email these days.

Network Outage

We went off the air at around maybe 11:30am. Automated phone tech support thingie says they're working on it and it should be resolved by 10am tomorrow (8/9/2022).

And... that's all I know at this point. Bleah.

Update (2pm Mon.):
Now they're saying tomorrow (Tuesday) at 5pm. ++Bleah.

Update (7:30pm Mon.):
Oh, weird. It looks like things came back five hours ago (2:27pm if I believe what connected_seconds() is telling me for various people) and they didn't notify me the way they said they would. Which might mean it's going be bouncing a bit. But for now, things seem to be up again.

Wind storms

Things have been windy. Power is now out. And the outages are sufficiently numerous that it may be a while (we don't have a repair ETA yet...)

Update: we've been back up since 2pm-ish.

Debian 11.0 (bullseye)

There's a new Debian to upgrade to as of today. I will be doing this as soon as I convince myself it won't be causing anything to die horribly, and then finish a round of backups. Probably tonight or tomorrow sometime.

Update (Sunday, 10:19pm PDT):  The downside of staying up for 300+ days straight is that everything has to fsck now. Hopefully this will go well, but it may take a while.

Update (Sunday, 10:49pm PDT):  libcrypt.so.1: cannot open shared object file: No such file or directory (and before you ask, yes, the package libcrypt is installed, so it looks like I get to go hunting through Changelogs to see WTF happened. This is going to take way, way longer.

Update (Monday, 11:30pm PDT):  The good news is I've successfully created an environment on my Debian 11 virtual machine that can run the moo server executable. There remains the small matter of deploying to my actual machine and not getting bitten by further surprises. If all goes well we'll be back on the air sometime tomorrow. Longer term, there's stuff to talk about (I'll go into detail on *Server, but the tl;dr is we're going to want to be a 64-bit MOO by the time Debian 12 comes out...)

Update (Tuesday, 11:15pm PDT):  And we are back up. Yay. (Thanks to Ken Fox and the other folks at Waterpoint for saving me a lot of time...)