Wikipedia reader survey
-
- Habitué
- Posts: 1120
- kołdry
- Joined: Sat Apr 14, 2012 3:35 am
- Wikipedia User: Anthonyhcole
Wikipedia reader survey
I've thought for a while this would be a good idea. But I know nothing about statistics or survey design. Does anyone think it would be a good idea or have any ideas about structure?
I'm wedded to nothing but would presently prefer to have it as an ongoing thing - just asking readers questions as they arise.
I'm wedded to nothing but would presently prefer to have it as an ongoing thing - just asking readers questions as they arise.
- thekohser
- Majordomo
- Posts: 13410
- Joined: Thu Mar 15, 2012 5:07 pm
- Wikipedia User: Thekohser
- Wikipedia Review Member: thekohser
- Actual Name: Gregory Kohs
- Location: United States
- Contact:
Re: Wikipedia reader survey
Are you aware of this?
"...making nonsensical connections and culminating in feigned surprise, since 2006..."
-
- Habitué
- Posts: 1120
- Joined: Sat Apr 14, 2012 3:35 am
- Wikipedia User: Anthonyhcole
Re: Wikipedia reader survey
Yep. And I started this last May, but it didn't attract much interest.thekohser wrote:Are you aware of this?
Here, I'm suggesting an en.Wikipedia-initiated survey of its readers on the en.Wikipedia website, not telephone survey initiated by the foundation.
Last edited by Anthonyhcole on Thu Dec 19, 2013 4:39 pm, edited 1 time in total.
- thekohser
- Majordomo
- Posts: 13410
- Joined: Thu Mar 15, 2012 5:07 pm
- Wikipedia User: Thekohser
- Wikipedia Review Member: thekohser
- Actual Name: Gregory Kohs
- Location: United States
- Contact:
Re: Wikipedia reader survey
An annual survey of English Wikipedia readers, randomly sampled, is an excellent idea. Because of it being an excellent idea, I give it little chance of being properly executed (or executed at all) by the Wikipedia "community". It will take an initiative from the Wikimedia Foundation, which would include paying a research vendor to shepherd the project from planning, to launch, to analysis and reporting.
Furthermore, regarding the 2011 reader study that was headed up by Mani Pande, I contacted Ms. Pande on March 7, 2011, offering to help with sample and survey design. She didn't respond. I contacted her again on March 14. Again, no response. So, I left a voicemail for Barry Newstead, and he failed to respond as well. (Ms. Pande left the Wikimedia Foundation around June 2012.)
The work was awarded to Resolve Market Research (which has since been absorbed into Greg Bovitz's research company). I'll bet $20 to any non-WMF charity that there was no competitive bidding associated with the Resolve decision.
Furthermore, regarding the 2011 reader study that was headed up by Mani Pande, I contacted Ms. Pande on March 7, 2011, offering to help with sample and survey design. She didn't respond. I contacted her again on March 14. Again, no response. So, I left a voicemail for Barry Newstead, and he failed to respond as well. (Ms. Pande left the Wikimedia Foundation around June 2012.)
The work was awarded to Resolve Market Research (which has since been absorbed into Greg Bovitz's research company). I'll bet $20 to any non-WMF charity that there was no competitive bidding associated with the Resolve decision.
"...making nonsensical connections and culminating in feigned surprise, since 2006..."
Re: Wikipedia reader survey
That is why it would be more valuable to repeat the 2011 survey-- which would presumably have determined something about the sample universe (readers of Wikipedia). If it was not a representative sample, what use was it?thekohser wrote:An annual survey of English Wikipedia readers, randomly sampled, is an excellent idea. Because of it being an excellent idea, I give it little chance of being properly executed (or executed at all) by the Wikipedia "community". It will take an initiative from the Wikimedia Foundation, which would include paying a research vendor to shepherd the project from planning, to launch, to analysis and reporting.
An open-to-all online "survey" is just a self-selected "sample" of people with strong opinions. If you know nothing about the demographics of the actual readership:
According to Jimmy Wales: linkIn the past, we have found that surveys conducted on our website are more biased towards editors, males and heavy readers of Wikipedia. By doing a survey at the household level, we will ensure that we have a good representation of casual and female readers of Wikipedia.
I think he knows that Wikipedia does keep data on what is being read; he must be thinking, "Wikipedia does not keep data on the people who are reading it".It should be a fundamental human right that what you read on Wikipedia should be between you and Wikipedia - and Wikipedia hardly keeps any data on what is read, and what it does keep it doesn't keep for long.
If you have no way of knowing anything about who is reading it, you can only survey a general population, of which something is known.
former Living Person
- Poetlister
- Genius
- Posts: 25599
- Joined: Wed Jan 02, 2013 8:15 pm
- Nom de plume: Poetlister
- Location: London, living in a similar way
- Contact:
Re: Wikipedia reader survey
As an opening question, I'd ask what your universe is, i.e. what is the pool from which you sample. If it's everyone who reads Wikipedia, not just those with an account, how do you propose to get hold of them? Do a mail shot to people all over the world, asking if they read Wikipedia?
"The higher we soar the smaller we appear to those who cannot fly" - Nietzsche
-
- Habitué
- Posts: 1120
- Joined: Sat Apr 14, 2012 3:35 am
- Wikipedia User: Anthonyhcole
Re: Wikipedia reader survey
Don't know.Poetlister wrote:As an opening question, I'd ask what your universe is, i.e. what is the pool from which you sample. If it's everyone who reads Wikipedia, not just those with an account, how do you propose to get hold of them? Do a mail shot to people all over the world, asking if they read Wikipedia?
However, I'm vaguely thinking of having a little box called "Survey question" at the top of every article. Sometimes it can contain a general question (Should Wikipedia offer you the option to filter potentially offensive images?"). When there's no general survey question it could default to the article feedback tool's "Did you find what you were looking for?" or "Can you suggest an improvement to this article?" As for sample selection, would it make sense to present the question to all readers - logged in or not - and allow people to cut and slice the results any way they want, rather than us choosing the sample a priori?
- thekohser
- Majordomo
- Posts: 13410
- Joined: Thu Mar 15, 2012 5:07 pm
- Wikipedia User: Thekohser
- Wikipedia Review Member: thekohser
- Actual Name: Gregory Kohs
- Location: United States
- Contact:
Re: Wikipedia reader survey
This is not difficult at all. You run a pop-up window on Wikipedia that selects every Nth page load, and asks the reader for 5 minutes of their time. About 3% will participate, and that's how you sample the pool. Bonus points if the survey is mobile-reader compliant. It is my lightly-informed opinion that the industry leader in web-user pop-up survey research is Foresee Results (T-H-L). (Yes, the article was created by a single-purpose account, with a conflict of interest as the former social media specialist for Foresee.) (Go Spartans, Daniel!)Poetlister wrote:As an opening question, I'd ask what your universe is, i.e. what is the pool from which you sample. If it's everyone who reads Wikipedia, not just those with an account, how do you propose to get hold of them? Do a mail shot to people all over the world, asking if they read Wikipedia?
Surely many of you have encountered a Foresee pop-up before?
If you make the questionnaire, or even bits of the questionnaire, available at all times to all readers, then you invite abuse -- someone with time on their hands could reload and reload the survey, taking it repeatedly to reflect their fringe agenda. (That's what I would do, anyway.)Anthonyhcole wrote:As for sample selection, would it make sense to present the question to all readers - logged in or not - and allow people to cut and slice the results any way they want, rather than us choosing the sample a priori?
"...making nonsensical connections and culminating in feigned surprise, since 2006..."
- Poetlister
- Genius
- Posts: 25599
- Joined: Wed Jan 02, 2013 8:15 pm
- Nom de plume: Poetlister
- Location: London, living in a similar way
- Contact:
Re: Wikipedia reader survey
Obviously, installing a pop-up would require (a) the endorsement of the WMF, so it's a bit counter-productive discussing it here; (b) the services of some of the excellent people now working on th evisual editor and they probably have very little spare time. Anyway, a pop-up give highly biased results; I would never trust any conclusions from one. And you have no idea how biased they are because you don't know who has seen and ignored it.
"The higher we soar the smaller we appear to those who cannot fly" - Nietzsche
- thekohser
- Majordomo
- Posts: 13410
- Joined: Thu Mar 15, 2012 5:07 pm
- Wikipedia User: Thekohser
- Wikipedia Review Member: thekohser
- Actual Name: Gregory Kohs
- Location: United States
- Contact:
Re: Wikipedia reader survey
I guess we have no choice then, but to insist on house-to-house searches for Wikipedia readers, coupled with compulsory participation in a paper-and-pencil survey about said activity.Poetlister wrote:Anyway, a pop-up give highly biased results; I would never trust any conclusions from one.
"...making nonsensical connections and culminating in feigned surprise, since 2006..."
- Poetlister
- Genius
- Posts: 25599
- Joined: Wed Jan 02, 2013 8:15 pm
- Nom de plume: Poetlister
- Location: London, living in a similar way
- Contact:
Re: Wikipedia reader survey
Well, it could be computer-assisted.thekohser wrote:I guess we have no choice then, but to insist on house-to-house searches for Wikipedia readers, coupled with compulsory participation in a paper-and-pencil survey about said activity.Poetlister wrote:Anyway, a pop-up give highly biased results; I would never trust any conclusions from one.
Seriously, that's the only way to get gold standard results, and since it's obviously absurd to contemplate such a thing, we must just accept that we're not going to get anything more reliable than Wikipedia itself.
"The higher we soar the smaller we appear to those who cannot fly" - Nietzsche
-
- Habitué
- Posts: 1120
- Joined: Sat Apr 14, 2012 3:35 am
- Wikipedia User: Anthonyhcole
Re: Wikipedia reader survey
I'd put the survey invitation at the top of every article for all readers (logged-in or -out), and concurrently I'd put the same questions in street, mail and phone surveys in three different English-speaking countries (USA, UK & Australia).
Compare the results. If the difference between en.Wikipedia-hosted results and the others is minimal or large but consistent and predictable, we can rely on en.Wikipedia-hosted questions in future, adjusting for the predictable bias and running the occasional street/mail/phone survey to ensure the difference between article-based and street/mail/phone results hasn't drifted.
Poetlister, what do you mean by computer-assisted?
Does anyone on this site have expertise and a successful professional record in survey research? (Advice from keen amateurs is of course welcome and appreciated too.)
Compare the results. If the difference between en.Wikipedia-hosted results and the others is minimal or large but consistent and predictable, we can rely on en.Wikipedia-hosted questions in future, adjusting for the predictable bias and running the occasional street/mail/phone survey to ensure the difference between article-based and street/mail/phone results hasn't drifted.
MediaWiki already has a tool for this, Wikipedia:Article Feedback Tool. See the bottom of Cancer pain (T-H-L) where the tool is deployed. Changing its shape and moving to the top of the article is simple.Poetlister wrote:Obviously, installing a pop-up would require (a) the endorsement of the WMF, so it's a bit counter-productive discussing it here
Poetlister, what do you mean by computer-assisted?
Does anyone on this site have expertise and a successful professional record in survey research? (Advice from keen amateurs is of course welcome and appreciated too.)
- thekohser
- Majordomo
- Posts: 13410
- Joined: Thu Mar 15, 2012 5:07 pm
- Wikipedia User: Thekohser
- Wikipedia Review Member: thekohser
- Actual Name: Gregory Kohs
- Location: United States
- Contact:
Re: Wikipedia reader survey
Poetlister meant that a door-to-door survey taker can have the questionnaire loaded on a laptop or tablet device, so that administration of the questionnaire is easier and kept uniform. (Skip patterns on a paper questionnaire can get tricky for the interviewer.)Anthonyhcole wrote:Poetlister, what do you mean by computer-assisted?
Does anyone on this site have expertise and a successful professional record in survey research? (Advice from keen amateurs is of course welcome and appreciated too.)
I have been working in survey research for 20 years. I currently head a research department with more than a $2 million annual project budget (not including personnel costs). You can find some of my past, mostly casual musings on the business here.
Poetlister is an accomplished statistician who knows more about the science behind population studies than I ever will. I'm more of a business practitioner of applied research.
"...making nonsensical connections and culminating in feigned surprise, since 2006..."
- Poetlister
- Genius
- Posts: 25599
- Joined: Wed Jan 02, 2013 8:15 pm
- Nom de plume: Poetlister
- Location: London, living in a similar way
- Contact:
Re: Wikipedia reader survey
Another issue with pop-up surveys is that it's easier to answer a survey a dozen times than to vote a dozen times in the ArbCom elections.
"The higher we soar the smaller we appear to those who cannot fly" - Nietzsche
- thekohser
- Majordomo
- Posts: 13410
- Joined: Thu Mar 15, 2012 5:07 pm
- Wikipedia User: Thekohser
- Wikipedia Review Member: thekohser
- Actual Name: Gregory Kohs
- Location: United States
- Contact:
Re: Wikipedia reader survey
One could control it a bit by ceasing pop-ups to IP addresses that already completed a survey.Poetlister wrote:Another issue with pop-up surveys is that it's easier to answer a survey a dozen times than to vote a dozen times in the ArbCom elections.
"...making nonsensical connections and culminating in feigned surprise, since 2006..."
- Poetlister
- Genius
- Posts: 25599
- Joined: Wed Jan 02, 2013 8:15 pm
- Nom de plume: Poetlister
- Location: London, living in a similar way
- Contact:
Re: Wikipedia reader survey
My IP changes most days.thekohser wrote:One could control it a bit by ceasing pop-ups to IP addresses that already completed a survey.Poetlister wrote:Another issue with pop-up surveys is that it's easier to answer a survey a dozen times than to vote a dozen times in the ArbCom elections.
"The higher we soar the smaller we appear to those who cannot fly" - Nietzsche