Wikidata's wb_items_per_site table dropped

[Giraffe Stapler]

Several complaints at the Village Pump turn out to be caused by the same thing. Apparently if you delete (or "drop") a particular table on Wikidata, it screws things up for all logged-in users. The incident report is worth reading.

Impact: For about 20 minutes, almost all content page parses were broken. They received, along with any other user interaction with the wb_items_per_site table in some way, a database error message instead of page content. This affected many (but not all or even most) page views for logged-in users, and many (possibly most, but not all) edits made to wikis. For 19 hours, all users looking at pages rendered since the incident start received some incorrect metadata on pageviews (infobox content, cross-language links, wikdata item relation). Also, Wikidata's checks for duplicate site links failed, leading to many hundreds of duplicate items being created.

Cause: Wikidata's wb_items_per_site secondary table, used for the rendering of every page connected to Wikidata, was dropped by a mis-configured weekly cron script which executed the update.php code path, which itself had been misconfigured for eight years to drop this table. This immediately led to a DBQueryError on content page loads (reads). The table was re-created as blank, at which point pages began to paint again (though wrongly).

[Vigilant]

Sounds like something that absolutely could have been tested for.

[Osborne]

Sounds like something that absolutely could have been tested for.

What is "testing"? -- unidentified WMF developer

[Dysklyver]

One day a misconfiguration will accidentally delete the entire database, and everyone will panic for a day as they restore from a backup.

[el84]

One day a misconfiguration will accidentally delete the entire database, and everyone will panic for a day as they restore from a backup.

Which will lead to a massive RfC, RfArb etc., because the ''''''wrong'''''' version was restored.

[Poetlister]

One day a misconfiguration will accidentally delete the entire database, and everyone will panic for a day as they restore from a backup.

Which will lead to a massive RfC, RfArb etc., because the ''''''wrong'''''' version was restored.

Is there an ArbCom on Wikidata?

[Death To Wikipedia]

The last major outage of Wikipedia was also, iirc, caused by a configuration problem during a supposedly routine software update, the sort of thing that gets spotted in professional code/change review.

To be honest, it's a wonder it stays up so long at all, given the amateurs involved are not really the best the voluntary sector has to offer, and the paid staff are most certainly not the cream of the crop, not remotely what you would find running such a huge website ordinarily.

Kudos for their level of transparency though, it must be so tempting to try and cover these things up and hope their slack-jawed edit-slaves blame it on the NSA or Putin's little green men.

Keep donating guys. You're not wasting your money at all.