The kinds of experiments that work on Mturk

In my never ending attempt to push colleagues into the Mturk world, I often tell them how I have never had a study that worked on another population and not work on the Mturk worker population. However, inevitably my advertisements prove false and I get someone who says that they got some weird results. After studying the types of experiments conducted, I noticed something that should have been obvious from the beginning. Workers seem to be really good at short iterative tasks, or longer tasks that have objective measurements. Thus my decision making work which asks multiple choice questions or asks people to click some buttons  and watch some numbers appear and then judge them works really well. It is akin to tasks like photo tagging. Something short over and over again is easy to pay attention to. Similarly, asking someone to write out how they would solve a problem gets good results. Anyone can look at a paragraph on how to choose between two insurance options and tell if the person put effort into it or not.

However, complex imagined scenarios tend to not work very well. A worker is not used to keeping 2 pages worth of instructions memorized to imagine what it would be like to be a Doctor figuring out whether to prescribe a patient drug X which has a higher cure and premature death rate or drug Y which guarantees immediate safety, but may not cure the disease. In general I think it should be assumed that workers are under high-cognitive load at all times. The worker may be taking your survey at work and while answering a question they may also be trying to gauge if their boss is walking by. If cognitive load factors into your variables or you think a high amount of attention is needed, you will be better to use a different recruitment method or pay a significantly higher amount of money. Last, if you are using Mturk, I would recommend using mechanisms to force attention to instructions like not allowing the Next button to appear for a minute or two in Qualtrics.


Single Post Navigation

3 thoughts on “The kinds of experiments that work on Mturk

  1. BillB on said:

    Any advice for a task asking workers to list cognitions? In my experience workers like to click but don’t like to type at all! This task is only 1:30-2:00 in length, asking Ss to list their thoughts after viewing a message, but in pilot testing I had huge falloffs in participation. Any ideas how to incentivise it with a bonus structure or any other suggestions? Thanks!

    • BillB on said:

      Sorry just adding the comment to get notified of any replies.

    • There are a lot of varieties of thought listing protocols, so depending on what you are doing, you could require certain numbers thoughts (e.g. put 5 text boxes, all boxes have to be filled before moving on). You could require one be filled out and then bonus for each additional one. You could leave it free form, and just add a timer to the page so people can’t move on for a minute or two. Obviously some people will just go check emails for that minute, but it may be better than nothing. I don’t know what you are trying to do experimentally (since the following will be huge prime) but you could make people feel positive affect by telling them in the HIT that the study will take up to 10 minutes, then when they start inform them that they have been placed in a condition that only takes 2 minutes. They are more likely to view extra time spent doing your task as non-wasted time, and may even feel some kind of reciprocity. I’ve thought about other psych-type manipulations that promote engagement with research. For instance, somehow promoting a collaborative environment. Workers, I’m sure, default to viewing requesters as some kind of boss. Somehow presenting the HIT as a collaboration rather than an agent-subject relationship would probably promote engagement. Also, framing the ethics of research participation as something other than economics would be better. I’m thinking of Freakonomics chapter (and I’m sure its some broad area of research, but I only know it from Freakonomics) about moral decisions being replaced by economic decisions when money is involved (e.g. day care parents show up even later when a late fee is applied for picking up children after they are supposed to). So instead of threatening rejection, saying something about how the goal of the research is to find interesting new knowledge about human cognition, and to do so you need the most accurate data possible, and you only ask people to pay attention to instructions and questions, etc, would probably help a little.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: