Why this Site?

  • Our Mission:
  • We exist to shine the light of scrutiny into the dark crevices of Wikipedia and its related projects; to examine the corruption there, along with its structural flaws; and to inoculate the unsuspecting public against the torrent of misinformation, defamation, and general nonsense that issues forth from one of the world’s most frequently visited websites, the “encyclopedia that anyone can edit.”
  • How you can participate:
  •  Visit the Wikipediocracy Forum, a candid exchange of views between Wikipedia editors, administrators, critics, proponents, and the general public.
  • 'Like' our Wikipediocracy page on Facebook.
  •  Follow Wikipediocracy on Twitter!

Press Releases

  • Please click here for recent Wikipediocracy press releases.

Google Search

Experiment concludes: Most misinformation inserted into Wikipedia may persist

by Gregory Kohs

A months-long experiment to deliberately insert misinformation into thirty different Wikipedia articles has been brought to an end, and the results may surprise you. In 63% of cases, the phony information persisted not for minutes or hours, but for weeks and months. Have you ever heard of Ecuadorian students dressed in formal three-piece suits, leading hiking tours of the Galapagos Islands? Did you know that during the testing of one of the first machines to make paper bags, two thumbs and a toe were lost to the cutting blade? And would it surprise you to learn that pain from inflammation is caused by the human body’s release of rhyolite, an igneous, volcanic rock?

None of these are true, but Wikipedia has been presenting these “facts” as truth now for more than six weeks. And the misinformation isn’t buried on seldom-viewed pages, either. Those three howlers alone have been viewed by over 125,000 Wikipedia readers thus far.

The second craziest thing of all may be that when I sought to roll back the damage I had caused Wikipedia, after fixing eight of the thirty articles, my User account was blocked by a site administrator. The most bizarre thing is what happened next: another editor set himself to work restoring the falsehoods, following the theory that a blocked editor’s edits must be reverted on sight.

How reliable is Wikipedia?

When Wikipedia first entered the scene in 2001, while lots of people thought how amazing it would be to have an encyclopedia about everything, a few people asked whether we could trust an encyclopedia that any old goofball can edit. Would the average reader be able to tell if what they encounter on Wikipedia is factual truth, or just creatively vandalized misinformation?

.

Over the years, a deceptive myth began to take hold as fact for many people, even ones with an advanced education — that Wikipedia’s community of editors was on constant watch for vandals, zapping their mischievous bits of misinformation, such that nearly all vandalism is very quickly reverted within minutes, if not seconds. The news media could report itself blue in the face about Wikipedia hoaxes like Amelia Bedelia or John Seigenthaler’s role in the Kennedy assassination or the record-breaking fake article about Aboriginal deity Jar’Edo Wens, or even about long-running personal vendettas like the one carried out for years by “Qworty”. But Wikipedia’s true believers went on record to spread their faith in the online encyclopedia’s mythically near-perfect record against misinformation. They’d dismiss these egregious cases as outlying incidents of rarity.

The same news media that would eat up those Wikipedia hoaxes would also regurgitate the evidence from “studies” that praised Wikipedia’s self-healing powers:

* Daily Dot said, “Numerous studies have shown that vandalism — sneaking curse words or lies or libel into articles — gets cleaned up pretty quickly on Wikipedia.”

* The Economist touted, “Normally, such vandalism is corrected within minutes on Wikipedia because other people see it, remove it and improve the entry with their own genuine insight.”

* National Public Radio featured a professor who said, “And I tried to simply delete that reference, and when I did so, within minutes, that page was restored”.

* Dan Gillmor’s 2008 book, We the Media, proclaimed, “Wikipedia draws strength from its volunteers who catch and fix every act of online vandalism. When the bad guys learn that someone will repair their damage within minutes, and therefore prevent the damage from being visible to the world, they tend to give up and move along to more vulnerable places.”

* The Washington Post explained, “Generally, clear violations are taken down within minutes, as was the case with recent commentary on President Bush.”

* And finally, professor Alex Halavais in 2004 ran a small test of Wikipedia’s ability to detect vandalism. He made 13 changes that he described as vandalism, and he planned to leave them there for two weeks. “He expected to prove that Wikipedia is unreliable and un-vetted. Instead, all the changes were detected and fixed within a couple of hours, and Halavais conceded that he was impressed with Wikipedia’s self-correcting nature.” The problem with Halavais’ facile test was that his edits were all made from one account, so once one blooper was discovered, they were all easily discovered.

And that’s why I felt the need for my own experiment in 2015. Is the widely-proclaimed “self-correcting” Wikipedia still as defensive against damage as it supposedly was a decade ago? My plan would sample 30 vandalized articles, about one per day, using different IP addresses for each attempt. Generally, I progressed from infrequently-viewed articles about obscure subjects (like Rufus Barringer and the Koegel Meat Company), to highly-viewed articles about well-known subjects like inflammation, the movie Up in the Air, and Newcastle upon Tyne. Most of my vandalism attempts were “buried” in the middle or the end of an article, but a few were tested toward the beginning of articles (where people are more likely to read). Sometimes I would provide a legitimate-looking reference source, even though the source did not support my editorial claim (something I call a “feint” in the research documentation); but other times I would provide no source. And sometimes I would try to “disguise” my bad content with other helpful and accurate content in the same edit; but other times I’d just plunk down the phony content on its own.

The ethics of experimentation

.

 

As my experiment was underway and I started to talk with others about it, I was told by more than one person that it wasn’t an ethical approach to research. I was even asked, “Do you have any empirical evidence for the existence of the belief” that people think Wikipedia quickly fixes most of the errors deliberately introduced to it. Cue April 5th’s episode of ’60 Minutes’, featuring millionaire co-founder of Wikipedia, Jimmy Wales, telling Morley Safer: “I actually think of the problems we have, vandalism is one of the minimal ones, because it’s so thoroughly monitored and so carefully looked after.” There’s your empirical evidence. I also believe that when airport security systems are tested by undercover law enforcement agents, it is a necessary measurement process, even if it happens to inconvenience many travelers.

 

Journalist Dan Murphy came to my defense, too, regarding the ethics of my experiment. He said:

The convenient lie in all this has been the definition of ‘vandalism’ as a 12-year-old inserting at the top of an article “Ralphy is a penis-head.” But lies and distortions? They persist for years and years. Wikipedia lies when it says most vandalism is removed quickly. It doesn’t even have a working definition of vandalism.

Furthermore, even though Wikipedia’s parent company, the Wikimedia Foundation, collected $5.7 million in surplus cash beyond expenses last fiscal year, the organization not only has never spent a dime to evaluate vandalism on Wikipedia, they have never enacted any study of Wikipedia’s content quality at all. So, I conclude that I have served an ethical purpose, doing the sort of due diligence that the Wikimedia Foundation cleverly has avoided for over a decade.

The secret to sticky misinformation

While my study of only 30 articles was limited in scope, it certainly proves an important point. If you properly format nonsense inserted into Wikipedia and especially include a reference source (even a bogus one) that conforms to Wikipedia’s style guidelines, there is a very good chance that your vandalism will persist indefinitely.

Had I not attempted to unravel my deliberate mistakes, I am quite sure that Wikipedia would still say that the Sagami Railway in Japan was initially set up in 1917 to transport corn and fresh spicy shrimp (can you imagine the odor?) along the Sagami River valley. Likewise, a letter from Abraham Lincoln to Edwin Stanton would still be falsely directed to Albert E. H. Johnson. And the legend of Bodhidharma turning a bridegroom into a goldfish would still be Wikipedia’s version of truth.

Furthermore, because my attempts to fix Wikipedia back to its previous state were halted by a Wikipedia administrator, at the time of this publication there are still eleven more boneheaded falsehoods in Wikipedia, waiting for some other volunteer to correct them.

.

One of the more delicious outcomes of my experiment has been the revelation of Wikipedia’s ridiculous governing rule sets and their policing. In other words, if a user named “Bumperdinck” without any stated credentials whatsoever works extensively on the article about inflammation, his Talk page is showered with praise from other editors. But if I try to correct Wikipedia so that it doesn’t say that inflamed human tissue produces volcanic rock, I cannot, because my account is blocked. Indeed, after my vandalism of the article about the Chenango Canal, I was still welcome to edit Wikipedia. But if I point out that another editor named “Hlkliman” who soon made several edits to the same article might presumably be Harvey L. Kliman, webmaster of the Chenango Canal Association website, then that is “outing”, and my postulation is removed from public view, and my account is blocked.

Perhaps in the coming weeks, I will publish more detailed accounts of the interesting stories related to my vandalism experiment, but in the meantime you can browse through my detailed notes on the project’s analysis sheet stored as a Google Document. Regardless of what you think about what I’ve done, there should be one clear takeaway from my results: Wikipedia’s purported “self-correcting” prowess is more myth than reality.

 

Image credits: Flickr/mufinn, Flickr/roger4336, Flickr/Ian E. Abbott ~ licensed under Creative Commons Attribution 2.0 Generic

33 comments to Experiment concludes: Most misinformation inserted into Wikipedia may persist

  • Ross McPherson

    A well written and well conceived test of Wikipedia standards. Hopefully 60 Minutes will give you equal time.

  • John Lilburne

    I’m reminded of the article on Thomas Rainsborough that for a year declared Thomas to be leading Ranter. Someone added the commentary to the article that he was a Leveller not a Ranter, only to be reverted by some called EponymousDunce or some such. And so Thomas remained a ranter for a few more months.

  • This story has been published now for about 14 hours. True Believers in Wikipedia have had access to the entire list of vandalized edits, all that time. Yet, the self-correcting Wikipedia *still* hasn’t gotten around to easily fixing five of the damaged articles.

  • User558748962

    This reminds me that a user, Legolas2186, was caught inserting fake references to get articles—mostly Madonna-related articles—promoted to Good Article (GA) and Featured Article (FA) status: see here. Legolas2186 disappeared after being called out, and was blocked in Jan 2012. A large cleanup operation to remove his non-existent citations ensued. There’s been three SPIs of the user here, the latest closed without action on 16 February 2015. This is a more insidious way of damaging Wikipedia; these articles were rated some of Wikipedia’s best-written articles. The GA and FA reviewers clearly didn’t check the sources properly until Binksternet eventually called him out.

    • HRIP7

      The Legolas2186 story was covered by Kevin Morris in the Daily Dot in 2013: How vandals are destroying Wikipedia from the inside.

      […] by all appearances Legolas2186 was a model, even exemplary, Wikipedian. In his five years as a volunteer editor, he’d racked up more than 36,000 edits, mostly on articles relating to Madonna, Lady Gaga, and other pop culture phenomenons.

      And he was well-rewarded. The community elevated 95 articles he worked on to “Good Article” status, an honor bestowed on less than 1 percent of the encyclopedia’s 4 million English language articles. Another two were crowned with an even rarer status: “Feature Articles,” the type of entry that’s featured on Wikipedia’s front page, some of the most valuable real estate on the Internet.

      But like his parallels in news media, Jayson Blair and Stephen Glass, Legolas was weaving together a portfolio of success with a web of cleverly constructed lies, false sources, and invented quotes. An investigation led by an enterprising team of Wikipedia editors dug up dozens of fabrications perpetrated by Legolas, who was later banished from the site. […]

  • It should be noted that very shortly after Wikipediocracy published this blog post, the Washington Post (a Top 5 US newspaper) published a related story by Caitlin Dewey. Bob Kosovsky, a vice president of the Wikimedia New York City organization, angered that Dewey would interview me for her story, quickly responded: “…it’s a poor article – one-sided, superficial reporting, barely worthy of a college freshman.” Also the same day, Dewey was saddled with a brand new Wikipedia biography about her.

  • Bernd

    Well, duh. The false edit relate to pieces of information that are mostly irrelevant. I’d expect that any encyclopedia (wiki or not) contains a large number of such minor errors because of sloppy research on part of the editors, even without any deliberate falsehoods. If you had tried to actually manipulate stuff that people consider to be serious, then I’m pretty sure you would have been caught at a much higher rate.

    • I will duly note that you don’t find Inflammation to be “serious stuff”. Same with Voluntary euthanasia, and Dance music. Or Road bicycle racing, or Copy testing. Next time, I will look for more “serious” topics.

      • Radiant Orchid

        I suspect that any attempt to insert false information into articles on such topics as videogames, highways, or internet memes would have been caught immediately. Does that make these things “serious stuff”?

  • Stefano Guidoni

    You forgot about pet theories. There are tons of articles on specific subjects that are property of a few users/administrators and their love for some fringe theory about the subject.

    And that aboriginal deity is a pussy: the article “Flower of Life”, a hoax generated outside of wikipedia by the New Age guru Drunvalo Melchizedek around the year 2000, is still on and rocking since June 2005. A couple more months and it will be a decade old.

  • Eagle

    An excellent article. The fundamental problem is that Wikipedia has created the role of “vandal fighter” instead of the role of “fact checker.” More people sign up for “vandal fighter” than for GA or FA reviewer. I always fact checked the articles that I reviewed for a GA, but many people do not. The public would hope that all Wikipedia articles would be fact-checked periodically, but in fact, even those nominated for GA are not.

  • Oliver

    Nice. I’m not surprised at all that your corrections were reverted. I think your experiment was well conceived and shows something important, but I wouldn’t represent it as a general test of vandalism. “Vandalism” is a Wikipedia-community euphemism, in effect. Were you deleting text or inserting gibberish, like some mad Visigoth at a keyboard? No, you were inserting high-quality counterfeit information–as a PR operative might. You were testing Wikipedia’s resistance and resilience against well-crafted false claims and the spread of disinformation. It just doesn’t pain me to hear that Wikipedia can be vandalized. It does upset me though to know it’s disseminating bunk. Which reminds me: You may have introduced bunk into many student reports, not to mention their impressionable minds, which you don’t seem to have owned up to. I think a little contrition on that count would be apt.

    • Hersch

      Any teacher who does not explicitly warn pupils against using Wikipedia as a source is probably in the wrong line of work.

      • Oliver

        I feel the same, but I wonder if most teachers really are so categorical. Even articles in reputable newspapers reference Wikipedia. It worries me that if there are legitimate uses, there’s not going to be a general prohibition on using it, and instead teachers will be teaching students instead to use it with care. I think a safe level of care is bound to be very hard to teach the 18 and under.

    • How about this, Oliver? I will apologize for my actions in this experiment, the moment that Jimmy Wales apologizes for falsely conveying the impression that errors are quickly fixed on Wikipedia almost all of the time.

  • Oliver

    D’oh, and errors. Most generally of all, I think your test tests what happens to introduced errors. I would guess errors are introduced more often than classical acts of vandalism are committed.

  • […] Caps Lock esetek felszámolásában, de ezen felül megáll a tudomány. Kohs a közelmúltban egy kísérletet is elvégzett, amely során 31 cikkbe helyezett el különféle hibákat, hogy aztán megnézze, mi […]

  • […] a recent experiment found that small errors purposefully introduced to Wikipedia lingered for very long periods of time, meaning that your chances of picking up a detail you think makes you sound smart will actually […]

  • Oliver

    @Stefano Guidoni, good catch re: pet theories. I think the Wikipedia line is that these get policed in the same way as vandalism, but I believe that’s supposing the theories are introduced into articles that already exist, for instance an article about lunar geology, into which some crank tries to introduce the green-cheese theory. But like a biographical article about the kid who sits next to you in home room, who’s not a public figure, a pet theory that’s about your obscure thesis topic, or that’s to explain something that is not widely seen even as a thing, I think is unlikely even to be seen…except and until its first victim or sucker sees it. There’s no peer-review and no threshold to publishing such theories, and “vandalism” glosses over this weakness/public threat completely.

  • Me

    Wikipedia has got a new longest hoax, https://en.wikipedia.org/wiki/Bec_dilxil_xastin lasted for over 10 years.

    • That may or may not be a hoax. Apparently some 50-year-old Arizona native American told the story of Bec dilxil xastin to a linguist back in the early 1900s.

  • TVD

    Exposed! Your experiment did just that. Its that simple. Now, its time for the powers that be to do something useful with the information. Maybe start by fixing those articles?

  • Erich

    How many of these “Wikipedia experiments” are you going to conduct before understanding that we ALL KNOW that Wikipedia is flawed. Stop vandalizing to a point. This is not news. How many school teachers have already inserted false infos on wiki articles related to the research subject they are giving their students the next day, for the sake of “uncovering” the plagiarism habit among them ?
    All vandals think they are right (or “funny”, or “clever”, or whatever) when making their edits and they all have “good reasons” (i.e. “experimenting”) but guess what, they are not.

    • Erich, you say “that we ALL KNOW that Wikipedia is flawed” — I guess you didn’t read the middle part of this blog post, nor Jimmy Wales’ own comment, “I actually think of the problems we have, vandalism is one of the minimal ones, because it’s so thoroughly monitored and so carefully looked after.”

    • Luchog

      Not true. In fact, many people seem to think that Wikipedia is a reliable source. I’ve dealt with dozens of them online and in person. The number of lazy journalists who have been caught plagiarizing false information from Wikipedia.

      In fact, not all of “us” know that Wikipedia is flawed. Thanks to the promotional efforts by Wales and his fanboys (disclaimer: I used to be one), and one piece of biased journalism in Nature that got repeatedly misreported, most people seem to think that it’s equivalent to Britannica in accuracy.

      People like you keep pointing to “vandals” as the problem. That’s nonsense. That’s merely pointing to a symptom and ignoring the disease. Its like giving sore throat lozenges to someone with a bad Streptococcus infection and claiming the disease is being effectively treated. It’s not. The problem with Wikipedia is systemic, it’s very core principle is inherently flawed and unworkable; and anything built on it stands on a foundation of sand. It’s just a matter of time before it completely collapses; and, unfortunately, it’s going to cause a lot of problems for a lot of people by the time it does.

  • Chris Crawford

    I am cross-posting this in response to a post you made on LiveScience that linked to this article.

    I read your article and the Google Docs page presenting the actual changes you made. While your experiment definitely shows that Wikipedia is not perfect, the falsehoods you inserted are so obscure and trivial that they do not demonstrate what I would call a significant weakness in Wikipedia.

    Almost every non-academic book contains some factual errors; I myself notice some factual errors while reading books, even some by academics, and I’m not even looking for them. Most good book reviews of nonfiction works will point out a few factual errors.

    Suppose you were to go to your local library and vet every nonfiction work there for factual errors. I’m sure that you could find plenty of errors that way, some of them more significant than the falsehoods you planted.

    The true significance of your experiment is to demonstrate the slippery nature of what we consider to be truth. Our perception of reality as recorded in our literature is not absolute; some facts are more reliable than other facts. If we insist that our sources of information contain only those facts of which we are absolutely certain, then the only sources of information we would have would be mathematical theorems (and Godel showed that even those might not be so certain).

    Rather than thinking in black-and-white terms of whether statements are true or false, I think it more useful to consider the combination of the reliability and the significance of statements. If I argue that Erasmus published Querula Pacis in 1520 instead of 1521, just how “wrong” is that statement? And how significant is the error? That depends upon the application of the statement. If I use it to support the hypothesis that Erasmus was growing increasingly concerned by the rift in the Church opened by Luther, the error matters not. If I use it to support the hypothesis that the inspiration for the book was a letter he received in late 1520, it’s significant.

    What if Erasmus had published Querula Pacis on January 1st, 1521, and I stated that he published it in 1520? How significant is my error? What if the publication process started on December 30th, but wasn’t complete until January 10th? Truth can be a slippery thing.

    I believe that Wikipedia now comprises the best initial source of information available, better than any other encyclopedia. It contains vastly more information, more easily accessible citations, better connections between pages and many other advantages. By any measure of “Truth”, Wikipedia contains more of it than any other single source. No, it’s not perfect. What is?

    • It’s my personal opinion that your assessment of the experimental errors as “obscure” and “trivial” is your personal opinion, but not backed by any objective measurement. And it’s my personal opinion that if you go to your local library and vet every nonfiction work there for factual errors (your suggestion), that you will find nowhere near the frequency or severity of errors that you’ll find in Wikipedia. Your closing paragraph suggests that “Truth” is something we quantify by the pound. An old saying goes something like this — pour a cup of fine wine into a barrel of sewage, and you have a barrel of sewage; pour a cup of sewage into a barrel of fine wine, and you have a barrel of sewage. That all said, I thank you for at least reading this blog post and reviewing the data set — that’s much more than the typical Wikipedian would ever do when presented with alarming evidence of information mismanagement.

  • Chris Crawford

    You’re certainly correct in pointing out the subjectivity of any assessment of obscurity or triviality. But subjectivity does not render a judgement useless. C’mon, seriously, do you really claim that changing the name of the chief engineer of a minor canal from John B. Jervis to Nathaniel Henry Hutton is NOT obscure or trivial?

    We’ll have to agree to disagree on the number of errors in nonfiction works. Let me put it this way: my personal library contains over 2,000 nonfiction works. By way of exercise, let me go through some of the books I have been reading over the last month:

    Collapse of the Bronze Age, by Manual Robbins. Contains some rather wild speculations on the actions of the Sea Peoples; ignores the evidence for destruction of cities by natural events and civil disorder.

    A More Perfect Heaven, by Dava Sobel. In attempting to protect the reader from Copernicus’ highly technical work, the author presents a great many simplifications and approximations that cannot by any stretch of the imagination be considered to be true; they are acceptable only because they insulate the reader from some truly abstruse material. But you would surely reject these simplifications as falsehood.

    1177 BC The Year Civilization Collapsed, by Eric H. Cline (Dr. Cline and I are collaborating on a side project). Dr. Cline declares that the havoc wrought by the Sea Peoples on marine trade contributed to economic collapse. I argue that, inasmuch as agriculture accounted for at least 99% of economic output, the loss of shipborne trade could not have caused economic collapse.

    The Measure of Reality, by Alfred W. Crosby. So far I haven’t found any errors, but I’ve only just begun.

    The point of this exercise is that books are just as subject to errors as any other medium. I have published a few books, and I can assure you, making absolutely certain that every detail is correct is simply impossible. You read, you re-read, the editor reads and re-reads, and things slip by you. I am currently editing another person’s book, and we’re not even trying to get every detail right; the subject matter is such that most of those details are not relevant to the book’s contents. I’m sure that there will be incorrect details in the book, especially with regard to some dates.

    I reject your analogy of wine and sewage. Its core idea is that perfection cannot be compromised. Since I maintain that perfection is unattainable, I consider the analogy inappropriate.

    I again suggest that you contemplate the question, “What is truth?” There is no such thing as a perfect truth, or a perfectly correct statement. Everything is an approximation. We accept these inaccuracies because we have no alternative. The question is best answered by deciding what you want to use a given truth for.

    • John lilburn

      Would you consider “in 1345, during the reign of King Richard II” to be a minor infraction? How about “Edward III divided his kingdom between his sons”? Or that “Thomas Rainsborough was a leading Ranter”? For many years British secular Jews were Israeli Atheists, and that Gerard Winstanley is both a Christian Atheist a Christian Communist, and a Housing rights activist.

  • Luchog

    Going back over this article, I wanted to mention something else.

    In response to a debate with a Wikipedia fanboy (disclaimer: I used to be one myself) about this very issue, I engaged on a similar, but smaller-scale experiment. Said fanboy was fond of repeating Jimbo’s assertion that vandalism was an insignificant issue (and still is, despite the evidence).

    There were multiple articles involved, but one in particular stood out. It was regarding a phenomenon from Japanese history that is quite popular with westerners, especially the type who make up the majority of Wikipedia’s user base; and could therefore be expected to be one that would be watched carefully.

    The article contained some popular misconceptions about the phenomenon, and some legends reported as truth. I corrected some of these, properly sourced, and added one absolutely egregious and unsourced error which anyone with at least half a brain could have caught. Some of my corrections were reverted, but the deliberate error was not. I watched the page for a bit over two years. It took about a year and a half before that error was finally removed, in what was a major overhaul of the page. The twisted part is that whoever corrected that error introduced a number of new ones, and the page as it stands is better than it was, but is still rife with errors and egregious omissions. Subsequent attempts to improve the page were met with reversions, and I had no interest in getting involved in an edit war, so I gave up on it.

  • […] the damage that subtle, intelligent, and well-established vandals can do, just see this page and this article. I KNOW IT HOW MUCH IT SUCKS. I MIGHT EVEN KNOW BETTER THAN YOU HOW MUCH IT SUCKS. We’re scaring […]

  • Skyrocket

    I am aware of a “vandalism reversion test edit” that was made in 2007 and still hasn’t been reverted. I look at it from time to time to see if it’s still there. It’s in an article about a reasonably populous suburb of an American city. It states a falsehood about municipal governance, and is unsourced. One wonders how many of these “tests” there are in the millions of articles in Wikipedia.