Wikipediocracy

I've thought for a while this would be a good idea. But I know nothing about statistics or survey design. Does anyone think it would be a good idea or have any ideas about structure?

I'm wedded to nothing but would presently prefer to have it as an ongoing thing - just asking readers questions as they arise.

Are you aware of this?

thekohser wrote:Are you aware of this?

Yep. And I started this last May, but it didn't attract much interest.

Here, I'm suggesting an en.Wikipedia-initiated survey of its readers on the en.Wikipedia website, not telephone survey initiated by the foundation.

An annual survey of English Wikipedia readers, randomly sampled, is an excellent idea. Because of it being an excellent idea, I give it little chance of being properly executed (or executed at all) by the Wikipedia "community". It will take an initiative from the Wikimedia Foundation, which would include paying a research vendor to shepherd the project from planning, to launch, to analysis and reporting.

Furthermore, regarding the 2011 reader study that was headed up by Mani Pande, I contacted Ms. Pande on March 7, 2011, offering to help with sample and survey design. She didn't respond. I contacted her again on March 14. Again, no response. So, I left a voicemail for Barry Newstead, and he failed to respond as well. (Ms. Pande left the Wikimedia Foundation around June 2012.)

The work was awarded to Resolve Market Research (which has since been absorbed into Greg Bovitz's research company). I'll bet $20 to any non-WMF charity that there was no competitive bidding associated with the Resolve decision.

thekohser wrote:An annual survey of English Wikipedia readers, randomly sampled, is an excellent idea. Because of it being an excellent idea, I give it little chance of being properly executed (or executed at all) by the Wikipedia "community". It will take an initiative from the Wikimedia Foundation, which would include paying a research vendor to shepherd the project from planning, to launch, to analysis and reporting.

That is why it would be more valuable to repeat the 2011 survey-- which would presumably have determined something about the sample universe (readers of Wikipedia). If it was not a representative sample, what use was it?

An open-to-all online "survey" is just a self-selected "sample" of people with strong opinions. If you know nothing about the demographics of the actual readership:

In the past, we have found that surveys conducted on our website are more biased towards editors, males and heavy readers of Wikipedia. By doing a survey at the household level, we will ensure that we have a good representation of casual and female readers of Wikipedia.

According to Jimmy Wales: link

It should be a fundamental human right that what you read on Wikipedia should be between you and Wikipedia - and Wikipedia hardly keeps any data on what is read, and what it does keep it doesn't keep for long.

I think he knows that Wikipedia does keep data on what is being read; he must be thinking, "Wikipedia does not keep data on the people who are reading it".

If you have no way of knowing anything about who is reading it, you can only survey a general population, of which something is known.

As an opening question, I'd ask what your universe is, i.e. what is the pool from which you sample. If it's everyone who reads Wikipedia, not just those with an account, how do you propose to get hold of them? Do a mail shot to people all over the world, asking if they read Wikipedia?

Poetlister wrote:As an opening question, I'd ask what your universe is, i.e. what is the pool from which you sample. If it's everyone who reads Wikipedia, not just those with an account, how do you propose to get hold of them? Do a mail shot to people all over the world, asking if they read Wikipedia?

Don't know.

However, I'm vaguely thinking of having a little box called "Survey question" at the top of every article. Sometimes it can contain a general question (Should Wikipedia offer you the option to filter potentially offensive images?"). When there's no general survey question it could default to the article feedback tool's "Did you find what you were looking for?" or "Can you suggest an improvement to this article?" As for sample selection, would it make sense to present the question to all readers - logged in or not - and allow people to cut and slice the results any way they want, rather than us choosing the sample a priori?

Poetlister wrote:As an opening question, I'd ask what your universe is, i.e. what is the pool from which you sample. If it's everyone who reads Wikipedia, not just those with an account, how do you propose to get hold of them? Do a mail shot to people all over the world, asking if they read Wikipedia?

This is not difficult at all. You run a pop-up window on Wikipedia that selects every Nth page load, and asks the reader for 5 minutes of their time. About 3% will participate, and that's how you sample the pool. Bonus points if the survey is mobile-reader compliant. It is my lightly-informed opinion that the industry leader in web-user pop-up survey research is Foresee Results (T-H-L). (Yes, the article was created by a single-purpose account, with a conflict of interest as the former social media specialist for Foresee.) (Go Spartans, Daniel!)

Surely many of you have encountered a Foresee pop-up before?

Anthonyhcole wrote:As for sample selection, would it make sense to present the question to all readers - logged in or not - and allow people to cut and slice the results any way they want, rather than us choosing the sample a priori?

If you make the questionnaire, or even bits of the questionnaire, available at all times to all readers, then you invite abuse -- someone with time on their hands could reload and reload the survey, taking it repeatedly to reflect their fringe agenda. (That's what I would do, anyway.)

Obviously, installing a pop-up would require (a) the endorsement of the WMF, so it's a bit counter-productive discussing it here; (b) the services of some of the excellent people now working on th evisual editor and they probably have very little spare time. Anyway, a pop-up give highly biased results; I would never trust any conclusions from one. And you have no idea how biased they are because you don't know who has seen and ignored it.

Poetlister wrote:Anyway, a pop-up give highly biased results; I would never trust any conclusions from one.

I guess we have no choice then, but to insist on house-to-house searches for Wikipedia readers, coupled with compulsory participation in a paper-and-pencil survey about said activity.

thekohser wrote:

Poetlister wrote:Anyway, a pop-up give highly biased results; I would never trust any conclusions from one.

I guess we have no choice then, but to insist on house-to-house searches for Wikipedia readers, coupled with compulsory participation in a paper-and-pencil survey about said activity.

Well, it could be computer-assisted.

Seriously, that's the only way to get gold standard results, and since it's obviously absurd to contemplate such a thing, we must just accept that we're not going to get anything more reliable than Wikipedia itself.

I'd put the survey invitation at the top of every article for all readers (logged-in or -out), and concurrently I'd put the same questions in street, mail and phone surveys in three different English-speaking countries (USA, UK & Australia).

Compare the results. If the difference between en.Wikipedia-hosted results and the others is minimal or large but consistent and predictable, we can rely on en.Wikipedia-hosted questions in future, adjusting for the predictable bias and running the occasional street/mail/phone survey to ensure the difference between article-based and street/mail/phone results hasn't drifted.

Poetlister wrote:Obviously, installing a pop-up would require (a) the endorsement of the WMF, so it's a bit counter-productive discussing it here

MediaWiki already has a tool for this, Wikipedia:Article Feedback Tool. See the bottom of Cancer pain (T-H-L) where the tool is deployed. Changing its shape and moving to the top of the article is simple.

Poetlister, what do you mean by computer-assisted?

Does anyone on this site have expertise and a successful professional record in survey research? (Advice from keen amateurs is of course welcome and appreciated too.)

Anthonyhcole wrote:Poetlister, what do you mean by computer-assisted?

Does anyone on this site have expertise and a successful professional record in survey research? (Advice from keen amateurs is of course welcome and appreciated too.)

Poetlister meant that a door-to-door survey taker can have the questionnaire loaded on a laptop or tablet device, so that administration of the questionnaire is easier and kept uniform. (Skip patterns on a paper questionnaire can get tricky for the interviewer.)

I have been working in survey research for 20 years. I currently head a research department with more than a $2 million annual project budget (not including personnel costs). You can find some of my past, mostly casual musings on the business here.

Poetlister is an accomplished statistician who knows more about the science behind population studies than I ever will. I'm more of a business practitioner of applied research.

Another issue with pop-up surveys is that it's easier to answer a survey a dozen times than to vote a dozen times in the ArbCom elections.

Poetlister wrote:Another issue with pop-up surveys is that it's easier to answer a survey a dozen times than to vote a dozen times in the ArbCom elections.

One could control it a bit by ceasing pop-ups to IP addresses that already completed a survey.

thekohser wrote:

Poetlister wrote:Another issue with pop-up surveys is that it's easier to answer a survey a dozen times than to vote a dozen times in the ArbCom elections.

One could control it a bit by ceasing pop-ups to IP addresses that already completed a survey.

My IP changes most days.