Please explain why Tool Labs goes up and down like a yo-yo

[EricBarbour]

https://tools.wmflabs.org

This is getting insane. There is no way to know if a given tool will be working on a given day or not, there is no warning of outages (or at least they're well-hidden), and the coders who are supposed to be "maintaining" them are nowhere to be found online. Especially the tools that process article histories, like "Revision history statistics", which we need all the time to figure out editwars.

No one holds the developers' feet to a fire when things fail. This isn't even "anarchy", it's a studied and deliberate attempt to be obtuse and erratic. Almost swear they were making the whole thing like a Python sketch. Or are they deliberately letting tools like this drop, because they want to cover something up?

[Jim]

When all this stuff was on the German server it used to run pretty well.

WMF have just failed to replicate the functionality.

At the moment I tend towards: Never attribute to malice that which is adequately explained by stupidity. or incompetence.

But I certainly think they care less than they should about whether this stuff works. It's never really made them look good that the best tools for analysing the site were not created by them. Kinda embarrassing.

[Poetlister]

That's just German efficiency and the usual WMF programmers. Can we start a campaign to have an all-German WMF board, not to mention an all-German ArbCom?

[lilburne]

That's just German efficiency and the usual WMF programmers. Can we start a campaign to have an all-German WMF board, not to mention an all-German ArbCom?

I propose 50% Saibo and 50% Niabot.

[Lukeno94]

I'd love to know what the uptime statistics are for Tool Labs, because I bet they're less than 50%. Certainly, less than 10% of the time when I want to use the Tool Labs it actually works as desired; otherwise, it times out to a white screen or gives me an error of some kind. I've got a whole bunch of retro PCs that might be able to double their server power if they want...

[Randy from Boise]

https://tools.wmflabs.org

This is getting insane. There is no way to know if a given tool will be working on a given day or not, there is no warning of outages (or at least they're well-hidden), and the coders who are supposed to be "maintaining" them are nowhere to be found online. Especially the tools that process article histories, like "Revision history statistics", which we need all the time to figure out editwars.

No one holds the developers' feet to a fire when things fail. This isn't even "anarchy", it's a studied and deliberate attempt to be obtuse and erratic. Almost swear they were making the whole thing like a Python sketch. Or are they deliberately letting tools like this drop, because they want to cover something up?

Not a Python sketch, more like Brazil.

RfB

[Kelly Martin]

You're asking about operational reliability from an organization that used to serve the entire PHP codebase for all of Wikimedia from a single, nonredundant NFS root server with no backup? The only reason they don't do this anymore is because it once failed and they had to scramble to fix it. Only after they'd been burned did they take any proactive measures to ensure uptime.

Wikimedia Engineering is a laughingstock. Especially given the budget and degree of loose cash that they have. Even today they tend to run with far less redundancy than would be considered acceptable in a reputable organization.

[Vigilant]

I'd love to know what the uptime statistics are for Tool Labs, because I bet they're less than 50%. Certainly, less than 10% of the time when I want to use the Tool Labs it actually works as desired; otherwise, it times out to a white screen or gives me an error of some kind. I've got a whole bunch of retro PCs that might be able to double their server power if they want...

Someone should write a meta tool that tests uptime for each WMF labs tool and posts the results in realtime.

[Johnny Au]

How can we be able to see detailed analysis of each user's editing history?

It works half the time at best.

[Lukeno94]

I'd love to know what the uptime statistics are for Tool Labs, because I bet they're less than 50%. Certainly, less than 10% of the time when I want to use the Tool Labs it actually works as desired; otherwise, it times out to a white screen or gives me an error of some kind. I've got a whole bunch of retro PCs that might be able to double their server power if they want...

Someone should write a meta tool that tests uptime for each WMF labs tool and posts the results in realtime.

I would imagine the "enormous" strain of such a tool would probably manage to crash the WMF Tools Lab server on its own!

[Vigilant]

I'd love to know what the uptime statistics are for Tool Labs, because I bet they're less than 50%. Certainly, less than 10% of the time when I want to use the Tool Labs it actually works as desired; otherwise, it times out to a white screen or gives me an error of some kind. I've got a whole bunch of retro PCs that might be able to double their server power if they want...

Someone should write a meta tool that tests uptime for each WMF labs tool and posts the results in realtime.

I would imagine the "enormous" strain of such a tool would probably manage to crash the WMF Tools Lab server on its own!

Sounds like win-win to me.

If the tools were each pinged once per minute, do you think the WMF would cry "Havoc" and claim DDoS?

[EricBarbour]

You're asking about operational reliability from an organization that used to serve the entire PHP codebase for all of Wikimedia from a single, nonredundant NFS root server with no backup? The only reason they don't do this anymore is because it once failed and they had to scramble to fix it. Only after they'd been burned did they take any proactive measures to ensure uptime.

Wikimedia Engineering is a laughingstock. Especially given the budget and degree of loose cash that they have. Even today they tend to run with far less redundancy than would be considered acceptable in a reputable organization.

So it seems. And you're talking about that mysterious September 2006 "database crash", I take it. Allegedly millions of diffs simply vanished, and the backup was incomplete. There is so little hard information remaining about what happened, I have no material for the book wiki.

[tarantino]

A related discussion is going on at User_talk:Cyberpower678, who helps run X!'s tools, which are broken at this moment.

[Poetlister]

It wouldn't be difficult to copy histories into an Excel spreadsheet and write a script to analyse them, but it would take a bit of time.

[Sitush]

Labs has been crap from the day it went live. It will likely stay crap unless WMF start employing engineers who actually know what they are doing, and supporting the tool writers who have been put in an unconscionable position by the decision to dictate to the community.

[Hex]

...unless WMF start employing engineers who actually know what they are doing...

[Anroth]

I know that is meant to be a pig, but it looks like a Moomin.

[Poetlister]

I know that is meant to be a pig, but it looks like a Moomin.

It's meant to be Andy Mabbett.

[Kelly Martin]

You're asking about operational reliability from an organization that used to serve the entire PHP codebase for all of Wikimedia from a single, nonredundant NFS root server with no backup? The only reason they don't do this anymore is because it once failed and they had to scramble to fix it. Only after they'd been burned did they take any proactive measures to ensure uptime.

Wikimedia Engineering is a laughingstock. Especially given the budget and degree of loose cash that they have. Even today they tend to run with far less redundancy than would be considered acceptable in a reputable organization.

So it seems. And you're talking about that mysterious September 2006 "database crash", I take it. Allegedly millions of diffs simply vanished, and the backup was incomplete. There is so little hard information remaining about what happened, I have no material for the book wiki.

No, that's a different incident entirely. I'm not knowledgeable of the situation behind that. The NFS crash caused substantial downtime but no data loss (they were able to piece together a working copy of the Mediawiki code, mainly from Brion's development computer and various email messages). There was another downtime caused once by the failure of the sole internal DNS server. Detect a pattern here, do you?

[Hex]

I know that is meant to be a pig, but it looks like a Moomin.

Can't unsee that now.

[Hex]

...they were able to piece together a working copy of the Mediawiki code, mainly from Brion's development computer and various email messages...

Christ on a fucking bicycle.

Is this chronicled in public email anywhere? I haven't read anything painfully funny recently.

[EricBarbour]

Christ on a fucking bicycle.

Is this chronicled in public email anywhere? I haven't read anything painfully funny recently.

We already went thru wikipedia-l and foundation-l archives from that period, and there was almost nothing said at the time. Damned if I can find anything on Vibber's talkpage either. They must have oversighted edits.

There was something in the Signpost on 21 August. Not sure if that was the same incident. Technical screwups were commonplace back then.

If you discover anything else, please pass it on.

[EricBarbour]

There's also the time Foundation-l melted down in 2005.....

[Lukeno94]

I think the WMF should hire the people who run IT Services at my university; since they took two months to fix an ethernet connection issue that affected the entire flat (maybe the whole block; it certainly did at one stage), and spent half the time burying their heads in the sand about it, I think they'd fit perfectly in the pool of incompetence.

[Notvelty]

I think the WMF should hire the people who run IT Services at my university; since they took two months to fix an ethernet connection issue that affected the entire flat (maybe the whole block; it certainly did at one stage), and spent half the time burying their heads in the sand about it, I think they'd fit perfectly in the pool of incompetence.

I'll let you in on a secret, Luke. IT services at universities aren't for the students. There may have been a professor of Post-Modern Humanities in Society or something who needed some help getting to the next level in Candy Crush/lost a cow on Farmville... you get the idea.

[SB_Johnny]

Christ on a fucking bicycle.

Is this chronicled in public email anywhere? I haven't read anything painfully funny recently.

We already went thru wikipedia-l and foundation-l archives from that period, and there was almost nothing said at the time. Damned if I can find anything on Vibber's talkpage either. They must have oversighted edits.

There was something in the Signpost on 21 August. Not sure if that was the same incident. Technical screwups were commonplace back then.

If you discover anything else, please pass it on.

I was on IRC that day "live as it happened", but I don't think I have logs. Freenode nearly broke under the weight, and after that they changed the policies for the technical channel (iirc it was nearly an IRC riot, but Cary Bass managed to pull a Malcolm X and dispersed the angry crowds).

[Kelly Martin]

...they were able to piece together a working copy of the Mediawiki code, mainly from Brion's development computer and various email messages...

Christ on a fucking bicycle.

Is this chronicled in public email anywhere? I haven't read anything painfully funny recently.

I heard about it via IRC. I rather doubt that it's been discussed "in writing".

[Kelly Martin]

There's also the time Foundation-l melted down in 2005.....

Was that the one caused by the fact that the WMF only had one email server for the entire organization (and it was also used for several other "minor" services)?

[Kelly Martin]

I think the WMF should hire the people who run IT Services at my university; since they took two months to fix an ethernet connection issue that affected the entire flat (maybe the whole block; it certainly did at one stage), and spent half the time burying their heads in the sand about it, I think they'd fit perfectly in the pool of incompetence.

Nah, that's not incompetence, it's just reflective of the high degree of care that university IT services has for students. If that had been a dean's Ethernet connection, it would have been fixed in a day or two, and if it had been in the lab where the IT geeks play Quake, it would have been detected when it was still in predictive fail, hotfailed to a live spare, and fixed before it actually failed at all. Don't confuse not caring for incompetence.

[Kelly Martin]

There was something in the Signpost on 21 August. Not sure if that was the same incident. Technical screwups were commonplace back then.

That one was genuinely not (really) their fault; Cogent screwed up. However, realistically, it was Brion's fault, because Brion was crap at managing Wikimedia's relationship with its transit and hosting providers. Wikimedia, in that time frame, paid 2x to 3x fair market value for its bandwidth and hosting services, because of Brion's abject disinterest in spending time negotiating contracts or pursuing competitive bids for external services. I remember that, at one point (2007, I think), I found that WMF was spending almost 10 times what was reasonable for IP transit, basically because they were still on a "low volume" plan and had merely scaled that plan up linearly to their transit volume, instead of negotiating a lower overall rate based on their actual, much higher volume. The fact that Brion was absolutely unsupervised (the Board was not competent to do so, and the WMF had no meaningful chief executive officer to provide that function until Sue Gardner was hired) did not help.

[Lukeno94]

I think the WMF should hire the people who run IT Services at my university; since they took two months to fix an ethernet connection issue that affected the entire flat (maybe the whole block; it certainly did at one stage), and spent half the time burying their heads in the sand about it, I think they'd fit perfectly in the pool of incompetence.

Nah, that's not incompetence, it's just reflective of the high degree of care that university IT services has for students. If that had been a dean's Ethernet connection, it would have been fixed in a day or two, and if it had been in the lab where the IT geeks play Quake, it would have been detected when it was still in predictive fail, hotfailed to a live spare, and fixed before it actually failed at all. Don't confuse not caring for incompetence.

The thing is, it *was* incompetence as well. They attempted a whole bunch of things that didn't work, and even fixed things that no one had asked them to (they made a whole bunch of networking changes for an Xbox One when I reported that my Xbox 360 was one of many devices that wasn't connecting, which was pretty amusing and dumb). So they would be a perfect fit for the WMF - incompetent and don't give a fuck about the end-user.

[EricBarbour]

There's also the time Foundation-l melted down in 2005.....

Was that the one caused by the fact that the WMF only had one email server for the entire organization (and it was also used for several other "minor" services)?

Yep. And once again, it was because Brion wasn't being supervised or supported properly. Like it or not, Sue Gardner made a number of positive changes, just by raising more money and hiring technical staff.

Let this be a warning to anyone who wants to hire Jimbo Wales or Florence Devouard or Cary Bass to run a corporation. Bad, bad idea.

[Hex]

However, realistically, it was Brion's fault, because Brion was crap at managing Wikimedia's relationship with its transit and hosting providers. Wikimedia, in that time frame, paid 2x to 3x fair market value for its bandwidth and hosting services, because of Brion's abject disinterest in spending time negotiating contracts or pursuing competitive bids for external services.

Quelle surprise. For geeks, having business conversations with those pesky, unpredictable human beings is unpleasant. Far more fun to write code instead. This is why no org worth its salt should ever put one in a position where communicative competence and business skills are required.