Why this Site?

  • Our Mission:
  • We exist to shine the light of scrutiny into the dark crevices of Wikipedia and its related projects; to examine the corruption there, along with its structural flaws; and to inoculate the unsuspecting public against the torrent of misinformation, defamation, and general nonsense that issues forth from one of the world’s most frequently visited websites, the “encyclopedia that anyone can edit.”
  • How you can participate:
  •  Visit the Wikipediocracy Forum, a candid exchange of views between Wikipedia editors, administrators, critics, proponents, and the general public.
  • 'Like' our Wikipediocracy page on Facebook.
  •  Follow Wikipediocracy on Twitter!

Press Releases

  • Please click here for recent Wikipediocracy press releases.

Google Search

What can fact checkers learn from Wikipedia — how not to do things, perhaps? (Part 2)

by Kingsindian

[Second in a two-part series.]

In Part 1, we looked at how the mechanisms behind Wikipedia are totally inadequate for the purpose of fighting “fake news”. But perhaps, as the communication director of the Wikimedia Foundation (WMF) Juliet Barbara says: “It’s a good thing Wikipedia works in practice because, in theory, it’s a total disaster.” So let’s look at how Wikipedia does in practice.

Self-image

Katherine Maher

Image: Wikimedia Commons
License: CC BY-SA 4.0

Katherine Maher, the executive director of the WMF, in defending Wikipedia’s content, points to two studies:
“A 2012 study commissioned by Oxford University and the Wikimedia Foundation, for example, showed that when compared with other encyclopedic entries, Wikipedia articles scored higher overall with respect to accuracy, references and overall judgment when compared with articles from more traditional encyclopedias. Wikipedia articles were also generally seen as being more up-to-date, better-referenced and at least as comprehensive and neutral. This study followed a similar 2005 study from Nature that found Wikipedia articles on science as reliable as their counterparts from Encyclopedia Britannica.”

Let’s look at the studies in turn.

The Oxford University study is not, and never pretended to be, an assessment of the quality of Wikipedia articles. It bills itself as a “preliminary comparative study” and a “pilot study” aimed at “explor[ing] the viability of the methods used … for a possible future study on a larger scale.” There is a disclaimer in the “Results” section: “All of the results outlined below are based on a small sample studied for the purposes of piloting the study’s approach and methods, and these results cannot therefore be generalised to the wider output of the online encyclopaedias referred to.” As far as I can determine, no follow-up study was undertaken, possibly because it proved difficult to surmount several obstacles identified in the study. Since it has been five years since then, one can conclude that assessing article quality is not high on the WMF’s priorities.  Certainly, nothing in any of the WMF’s past few annual budgets featured a line-item for the process of assessing article quality.

The criticism of the Nature study can be outsourced to Andreas Kolbe’s excellent Wikipediocracy blog post, which demonstrates why the hype about the study is unwarranted and misleading. Also, see this contemporary post by Nicholas Carr, which makes several cogent points. Leaving aside the Nature study’s validity, an obvious question presents itself: why is a study from 2005 is being quoted in 2017? Everyone recognizes that Wikipedia has changed a lot in the meantime, so what is the relevance of a study from that far back? But it seems that the WMF and scores of Internet pontificators just love the thesis of the study, so it’s just too tempting to not keep quoting it for more than a decade now.

Reality
So what is the actual state of Wikipedia content? This is a large topic and many different aspects have been studied. A 2014 paper by Okoli et. al. reviews a large number of studies assessing Wikipedia content. Many of the studies are somewhat old, but no more than the studies quoted above. The review looked at four aspects of content: breadth of coverage, reliability/accuracy, up-to-dateness, and readability. In the following, I give a general outline of their findings about the first two aspects.

Breadth of coverage: Wikipedia, by its very nature is well suited to cover a wide variety of topics, both niche and popular. Wikipedia famously has articles for all episodes of Doctor Who and The Simpsons, as well as obscure English football players and US politicians. The studies reviewed buttressed this popular perception: most of them find that Wikipedia is reasonably comprehensive in its coverage of topics ranging from science, health, history and philosophy. The research typically finds that older topics have more omissions than newer ones, a case of “recentism” which Wikipedia is known to suffer from. Regarding the “gender gap” on Wikipedia, the studies find that Wikipedia has a higher number of women biographies, and the same proportion of notable women biographies as several other datasets; however, biographies of notable women are more likely to be missing than male ones.

Reliability/accuracy: Of course, coverage of topics does not imply that they are presented accurately and completely. Here, the record of Wikipedia is quite mixed, with some studies evaluating quite favourably compared to professional sources, while others pointed to deficiencies, sometimes quite serious ones. The review found that the Wikipedia articles in the health domain (which has been most comprehensively studied topic, due to the obvious life-and-death implications) have mixed evaluations of Wikipedia content. In other domains, Wikipedia was found to be “generally” or “reasonably” accurate — however, Wikipedia articles frequently contain “sins of omission” where relevant and important information is missing from the article.

Studies conducted more recently follow this trend of mixed evaluations. A 2014 study in the Journal of the American Osteopathic Association concluded: “Most Wikipedia articles representing the 10 most costly medical conditions in the United States contain many errors when checked against standard peer-reviewed sources. Caution should be used when using Wikipedia to answer questions regarding patient care.” On the flip side, a 2014 study of pharmacology-related articles found that “Wikipedia is an accurate and comprehensive source of drug-related information for undergraduate medical education.”

Summarizing the evidence above, it’s fair to say that the picture of Wikipedia as being “roughly of the same quality as other reference works” is misleading at best. Wikipedia is decent for some things and unreliable for others. Moreover, even within the same category, there is a lot of variability in articles.

Neutrality
One suggestion in the communications audit was to position Wikipedia as a provider of neutral information and tweet out such information when a controversy breaks. This suggestion is seriously faulty, which will become clear from an example.

In May 2016, Google marked the 95th birthday of Yuri Kochiyama with a Google Doodle. Kochiyama was a longtime activist for various causes and her views on many topics were controversial. At the time, there was a big furor in the media about the Google Doodle; among other things, Senator Pat Toomey of Pennsylvania demanded an apology from Google because of Kochiyama’s views on Mumia Abu Jamal and Osama bin Laden. Without commenting on the merits of various sides, this “natural experiment” nevertheless provides some insight into how Wikipedia operates.

In the immediate aftermath of the controversy, the Wikipedia page was very heavily edited. In fact, it received more revisions in a single day, 19 May, than it had received since the article was started (back in 2005). In a matter of days, the article was rewritten drastically (compare the version on 18 May with the version on 1 June). At the very least, the experience suggests that the idea of Wikipedia as a repository of “neutral” content during times of high controversy is seriously flawed.

A broader view of neutrality on Wikipedia
Did I just pick an unrepresentative example, or is the case described above typical? Here the work of Greenstein and Zhu is useful. In a series of papers, they investigate the bias of Wikipedia articles on US politics. Their work covers a lot of ground, and I’ll only mention a few relevant parts here.

Greenstein and Zhu use the following technique (originally due to Gertzkow and Shapiro 2010) to investigate bias. They look at phrases occurring in the Congressional Record and associate a number of “code words” with Democrats and Republicans. For instance, “war on terror”, “illegal alien” and “Saddam Hussein” are used more often by Republicans, while the phrases “war in Iraq”, “Rosa Parks” or “national wildlife” are used more often by Democrats. The authors then use the frequency of the code words to determine the “slant” of articles on Wikipedia and Encyclopedia Britannica.

The authors find that Wikipedia articles, on average, lean slightly to the left as compared to Britannica, but this effect disappears entirely when controlling for article length (Wikipedia articles tend to be much longer than Britannica articles). Again, however, this average conceals a lot of variability depending on how many revisions the article gets. Articles in the upper quartile of their sample get enough revisions to be more or less neutral, but the median article does not, and shows “considerable difference in slant and bias from their Britannia counterparts”.

Finally, a comment about the overall approach. Wikipedia, as we have seen in part 1, is primarily edited by people in relatively rich, English-speaking countries. Greenstein and Zhu confirm this: they find that both Republicans and Democrats are represented well on Wikipedia, with Democrats outnumbering Republicans by some amount. This pattern is typical of many fields like journalism and academia. On many other topics, say India/Pakistan, or Israel/Palestine, even this rough parity of representation likely does not hold. This is reflected, for instance, in terrorism-related articles on Wikipedia.

funhouse mirror

Image credit: kthypryn on flickr CC BY 2.0

Toward a more realistic view of Wikipedia
The primary innovation of Wikipedia was to produce a large supply of “sorta” correct information for no charge at all. I do not want to disparage this accomplishment — Wikipedia would not be a top 10 website if people didn’t find it useful. I have used Wikipedia for more than a decade, and have been an on and off contributor for roughly the same time. Many Wikipedia contributors like myself are acutely aware of its flaws, which are many and big. The relationship between the image of Wikipedia I have and the image which the WMF propagates is akin to the relationship between a photograph and the image in a funhouse mirror.

I’ll end with two quotes about my own attitude toward Wikipedia.

The first comes from Andreas’ earlier WO article: “Wikipedia is free. It’s understandable that people don’t like to look a gift horse in the mouth. But no one should fool themselves into thinking that the nag they got for free is an Arabian racehorse.”

The other comes from Amílcar Cabral, one of Africa’s erstwhile anti-colonial leaders. Cabral’s motto for his own organization, which the WMF would be wise to adopt, was: “Tell no lies, claim no easy victories.”

10 comments to What can fact checkers learn from Wikipedia — how not to do things, perhaps? (Part 2)

  • With regard to neutrality, I’ll just repeat my assertion that that particular administrator should be required to change their name to something more descriptive of a regular mortal. I know that they are responsible for having framed the 5 pillars to which all “right-hearted” Wikipedians adhere. Still, in the spirit of the motto you mention — “tell no lies, claim no easy victories” — it would be well for the Community to recognize that power users from the Democratic Underground should perhaps be identified as such.

  • Good part 2 Kingsindian! Sandra Rientjes, director Wikimedia-NL, once claimed one Dutch television the English people trusted more on Wikipedia than the BBC! Indeed, these study’s are often misinterpreted or misused.

  • A nicely done piece, balanced and sourced, critical of some of what merits criticism without engaging in histrionics. I think, perhaps, the overall level of accuracy is somewhat undersold here — WP content is better than what many hardcore WP critics pretend it is — but it is absolutely beyond question that content is uneven between subjects and within subjects. Particularly problematic are “hot button” biographies and topics, which frequently attract one-sided contingents of activists to do battle over content. “Soft” topics are generally accurate and uncontroversial, if sometimes suffering the “errors of omission” mentioned above due to the incomplete nature of the project.

  • Captain Custard

    The drug study found that Wikipedia articles were at the same reading level as a university textbook, and in addition suffered from a less uniform and more scattered presentation of information than such text books. And it only measured Wikipedia’s ability to collect and maintain information that was mature enough to make it into multiple university textbooks.

    Rather than simply summarise the study, which has been done many times now, largely by Wikipedians, who of course love it for its glowing conclusion, it would have been useful for the author of this blog to use their expert knowledge of Wikipedia and the WMF’s statements, to make some comment on these specific findings, as they reveal a lot about the true nature of Wikipedia, and do not paint it in such as good a light as the conclusion, which the author merely repeated here, unquestioned.

    That is the supposed purpose of this website, shining a light on things previously unknown.

    • Captain, if you keep whining about the quality of blog posts here, without submitting your own version of blog excellence, then we might have to demote you to First Lieutenant.

  • Kingsindian

    Hi Captain Custard,
    My aim was not to evaluate the studies, but give an overview of their general findings. I simply wanted to show that the picture painted by Maher is misleading, and for reaching this conclusion, it is enough to simply summarize the studies.

    It is correct that all studies have more nuanced conclusions than I could indicate here. Unfortunately, this was necessary due to reasons of space. The post is already rather long and dense, and I didn’t want to make it even more so.

  • Captain Custard

    A rather pointless exercise, don’t you think? (Editor’s note: Far less pointless than whining repeatedly about a Wikipedia criticism site that isn’t meeting your tough standards that are backed by zero divulged credentials.) It is a trivial task for anyone who actually looks, to realise that what Maher says, doesn’t really match reality. It doesn’t take much effort. Anyone who is actually looking into such things, won’t have learned anything new from this post. (Editor’s note: Good thing the post is not exclusively directed toward people actually looking into such things.) And obviously, anyone who isn’t bothering to look, will not be reading it. (Editor’s note: And yet, it seems the Captain who hasn’t looked into such things — because he’s too superior to stoop to such grunt work — was caught reading the post. Is that hypocrisy, irony, or logical flaw?) This site promotes itself as the preeminent body of Wikipedia investigators, the experts, the insiders. (Editor’s note: It does not, but do go on.) Thus it should be offering information not easily available elsewhere to those people who are bothering to look. (Editor’s note: Wow, that’s a stretch.) Those sort of inquiring people would have appreciated in depth analysis (Editor’s note: Probably “in-depth” would have worked better there.) of apparently glowing studies the Wikipediots are cock-a-hoop about, not redundant summary. Analysis which directly addresses the issue of the day – that the media space is dominated by the voices of Maher and the devoted editors of her encyclopedia, who are quite happy for the world to know the takeaway line of such positive studies, which you helpfully simply repeated for her here. The presence of negative studies in your piece as some kind of counterbalance will be easily brushed off by the cult for the usual reasons, which are very believable as far as readers will see, due to the established reputation of this site as a peddler of biased hit pieces written for motives no higher than revenge or harassment. (Editor’s note: According to reliable sources, such as Salon, the Washington Post, and Daily Dot, that is not the actual established reputation of this site, but the Captain is free to imagine as he sees fit.)

    To become a worthwhile critic, first you need to quit being a willing part of a broken project which you appear to accept is willingly and deliberately playing the PR game with the world, taking full advantage of the fact the vast majority of people are not even interested in hearing the truth of how Wikipedia really works. And then you need to start writing blog posts which offer (Editor’s note: Probably should go with “blog posts that offer” in this case.) information not found elsewhere. That is how this site will regain its lost reputation as a wielder of influence, a must read platform. (Editor’s note: Even if the platform was once “must-read”, it should still be described as “must-read”, with a hyphen, not the way Captain describes it.) Still, I am not confident, you sadly have a long history of not listenng (Editor’s note: Another typing mistake by the Captain who claims to know what makes for excellence in communication.) to good advice on how to be a better critic. It has led you to this point, where your most glowing praise (Editor’s note: Captain’s access on this site does not include private discussion folders where other forms of glowing praise have been heaped on Kingsindian.) is from Carrite, who has practically zero reputation among Wikipedians and critics alike as far as his knowledge of Wikipedia goes. (Editor’s note: Well, you described Carrite to a T, but even a blind squirrel will find the occasional nut.) (Editor’s note: Just kidding, Tim.)

    • Kingsindian

      Hi Captain Custard,
      Thanks for your advice.

      I think analysis of public statements made by the WMF executive director is worth carrying, even if some of the points cover ground which has been covered before. Writing a synthesis of already-known things can be a worthwhile activity, as worthwhile as writing things which are not covered elsewhere. I am willing to bet that not more than 1 in 10 people in the general public know about any of the studies quoted here. Isn’t it worthwhile to tell them?

      You may think my approach is wrong, and you are free to do so. As far not listening to advice: I do listen to others, but I do what I think is right.

  • My eyes! The punctuation does nothing!

  • Several lengthy but tedious comments from a certain IP address in Newcastle, UK have been repositioned in the Trash.