The death of the confirmation code… hopefully

The confirmation code is an interesting psychological experiment in and of itself. We receive them for every digital transaction that we take part in from purchasing a book on to paying for dinner with a credit card. In our minds that number is our guarantee that what we just experienced actually happened and that we did not merely purchase that copy of 50 Shades of Grey (what??? I mean A Song of Ice and Fire!) in our imaginations. Without one, in some instances we can survive. Who looks that confirmation code on their credit card receipts after all? However, in others we seem to be lost, uncertain of the future and whether we can go on. Or so it seems in Amazon Mechanical Turk.

When I helped kickstart that use of MTurk at the Anderson School of Management at UCLA a few years ago, we needed a way to connect the data file on Mturk with the data file for our experiments. I’m sure I’m not the first person to think of this simple solution, but I suggested we assign every participant a unique code and have the participant enter it in the MTurk HIT before submitting it. It caught on there, and from the looks of the landscape its how everyone else does it too. It’s so ubiquitous, in fact, that when I stopped using them in favor of, in my opinion, a better practice, I started receiving 30 emails per study asking where their confirmation code was.

The problem with the confirmation code in MTurk is the error rate. Sometimes as many as 10% of people enter the code wrong. Also, it’s a pain to give each participant a unique code, and it’s annoying using these extra columns in the data files. I wanted to eliminate human copy & paste error, and utilize data already available. To do this I started storing each participant’s Amazon Worker ID in my experiment data files. The ID is automatically stored in the MTurk data file, and sending it and storing it in Qualtrics or other online survey systems is a breeze. The only problem, other than the uncertain feeling of the missing confirmation code, is that the worker has to accept the hit before you can access the ID. This is only a small problem.

What is below is a simple JavaScript that will create a hyperlink that only sends people to a URL if they have accepted the HIT already. Once accepted and clicked, the worker’s workerId is appended to the URL as a query string variable.  The code can be copy and pasted into the Source window of the Mturk HIT template. Then you simply have to replace the surveyUrl with the URL of your survey. The code is purposefully rudimentary (for JavaScript at least) so anyone can use it. If you know JavaScript, you can edit the code how you like, using buttons and events in place of the anchor, etc.

<script type='text/javascript'>
  var surveyUrl="";//The url you want to send people to.
  function gotoSurvey()
    var href=window.location.href;//Get the url of loading document (in mturk is the iframe the HIT content is in, not the url of the page itself)
    var queryString={};//Create an empty object to dump query string variables in.
    href.replace(new RegExp("([^?=&]+)(=([^&]*))?", "g"),function($0, $1, $2, $3){queryString[$1] = $3;});//Use a regular expression as well the a nifty alternative second parameter for the String.replace() method to dump all query variables into the queryString object
    {,'workerId',queryString['workerId']),'survey_window');//add workerId variable to URL and open popup. Edit the link text in case of popup blockers
      document.getElementById('survey_anchor').innerHTML="If your popup blocker prevented the survey window from opening, disable it and click this link again.";
      //No workerId variable. Worker hasn't accepted HIT -> ask to accept
      alert("You have not accepted the HIT yet. Please do so before clicking this link");
      document.getElementById('survey_anchor').innerHTML="Please accept the HIT, then click this link again";
  function addQueryVar(url, name, value)
    //Find anchor in URL since you can't add the query string after an anchor
    var fragmentStart = url.indexOf('#');
    if (fragmentStart < 0) fragmentStart = url.length;//no anchor so add variable to end
    var urlBeforeFragment = url.substring(0, fragmentStart);
    return urlBeforeFragment+(urlBeforeFragment.indexOf('?') < 0 ? '?' : '&')+encodeURIComponent(name)+'='+encodeURIComponent(value)+url.substring(fragmentStart);//if there is already query string variables add new variable with &, if not, add with ?, making sure to URI encode the content.
  <a id="survey_anchor" name="survey_anchor" href="javascript:gotoSurvey()">Click here to go to the survey</a>

You no longer have to have subjects enter anything in your Mturk HITs before submitting them. Mturk does require you have at least one input in your code, so I usually put a comment box: <textarea id="comments"></textarea>, and let the users tell me about typos and stuff that I missed.

Storing the ID in Qualtrics is easy. All you have to do is create an embedded data element in the Survey Flow menu. Name the embedded data field workerId (all lowercase except for the I). You are done (leave the value unset). It will grab the value from the URL and store it in your Qualtrics file. The workerId field is already stored in the Mturk data file, so simply sort them, or use VLOOKUP in Excel, or however else you transfer who to accept and who to reject to Mturk.

While confirmation codes psychologically tell us that something actually happened and there is a documented way to look it up, in MTurk they are fraught with problems. I can only hope that by showing people some code that will make their research lives easier, it will also help me by diminishing the amount of worried email I receive each time I post a study.


Single Post Navigation

4 thoughts on “The death of the confirmation code… hopefully

  1. Pingback: Drawing Hats on Monkeys with Mechanical Turk | Monkeyologist

  2. Pingback: Excluding Mturk workers from surveys in Qualtrics (and elsewhere) « the:behavioral:lab

  3. Pingback: How to email multiple MTurk workers | the:behavioral:lab

  4. Good post. I will be experiencing some of these issues as well..

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: