Inside the Internet's Hidden Science Factory 31
tcd004 writes Sarah Marshall has completed roughly 20,000 academic surveys. Clay Hamilton has finished about 40,000. Marshall and Hamilton are part of a small but highly-active community of paid online study participants on Amazon's Mechanical Turk who generate data at break-neck speed to fuel modern scientific research. But can a person who's completed thousands of surveys still provide good data? Here's a look at the humans feeding science from inside the machine.
I can see where this is going. (Score:5, Funny)
Name: Sarah Marshall
Age: 18-34
Education: Overqualified.
Employment: FIlling in surveys like this one
Hobbies: Writing about filling in surveys like this one
No (Score:1, Informative)
I do these surveys when I need money and I assure you, my answers are usually not truthful.
Re:No (Score:4, Funny)
At least he's truthful that he's not truthful.
Re: (Score:2)
At least he's truthful that he's not truthful.
Are you sure about that?
Also, I can see great benefit in turning some code like OpenCyc loose on the mechanical turk to generate answers to the survey questions, then reaping the results as OpenCyc answers thousands of surveys a day for you. I'd guess you should probably create multiple identities for that purpose though, as Amazon might get suspicious if someone spends 5,000 hours/day filling out surveys.
Re: (Score:2)
At least he's truthful that he's not truthful.
Telling the truth is not the highest moral quality: acting morally is.
Someone who does good but sometimes lies (eg. telling a white lie about Santa Claus to avoid upsetting a child) is morally superior to someone who does evil but is entirely honest about it.
Re: (Score:1)
Then they are the idiots
Re: (Score:1)
And you are committing fraud. Sure it is not enough money to seek legal recourse, but how does that weigh on your conscious? Obviously not at all and that tells us a lot about you (e.g. you are a sociopath).
...Which makes you a good capitalist businessperson, so carry on!
Re:No (Score:5, Interesting)
I'm a tenured researcher who has used MTurk from time to time for different things. It's a small part of what I've utilized, and I think maybe it's over utilized somewhat, but here's my thoughts in no particular order:
1. We know about people like you. We can't filter you out all the time, but we have ways of reducing your input (e.g., whether or not your response times are psychophysically possible, consistency of response, consistency of responses with known information about you, like general location).
2. There's been tons of research about MTurk as a participant pool. Generally speaking, data from MTurk is comparable to data from other sources, although MTurkers probably are dealing with a teeny bit more more problems in their life than the average person (which can be good or bad from a researcher's perspective). So you might be lying, but there's always people lying, and the question is whether they're lying more on MTurk than anywhere else.
3. Studies have shown that most MTurkers do it for fun rather than money.
4. Having said that, there have been some issues brewing about the original intent of MTurk and how it's sometimes used. I've seen problems related to the fact some MTurkers see the tasks as jobs, whereas researchers see it as volunteer research participation with a extra gift thrown in. It's something that I used to see as being a non-issue, but I've changed in that opinion over time. I do think there's some subtle ethical issues that arise in terms of differences of expectations between researchers and participants going into things (e.g., as a participant am I a volunteer or an employee? What does it mean if I'm participating in research for a job?).
5. MTurk is great for some tasks--those that are short and quick. Things that take more time, not so much.
6. You can reach a very broad audience with MTurk, which can be useful if you're trying to reach people out of your country.
Personally, I'd like to see something like MTurk set up where it's explicitly about the research. You're seeing people do this with FB and whatnot sometimes, but it would be nice to see something more organized. I've seen attempts, but nothing quite the same.
Re: (Score:2)
Thanks for sharing this. really fascinating.
Re: (Score:2)
This is excellent additional perspective -would you be willing to add this to the comments in the article?
Re: (Score:1)
>3. Studies have shown that most MTurkers do it for fun rather than money.
That one made me laugh.
Re: (Score:2)
Those people probably know far more about you than you think they do and that is in your most wild imaginings. Never assume based upon the title of those kinds of surveys and questions being asked, what the real nature of the survey might actually be about and how they choose to interpret and analyse you answers. Be careful how long the answers to your questions last and what future potential employer might gain access to interpretations of your behaviour.
Re: (Score:2)
Been there (Score:3)
After skimming this glad I left it.