The death of the confirmation code… hopefully
The confirmation code is an interesting psychological experiment in and of itself. We receive them for every digital transaction that we take part in from purchasing a book on Amazon.com to paying for dinner with a credit card. In our minds that number is our guarantee that what we just experienced actually happened and that we did not merely purchase that copy of 50 Shades of Grey (what??? I mean A Song of Ice and Fire!) in our imaginations. Without one, in some instances we can survive. Who looks that confirmation code on their credit card receipts after all? However, in others we seem to be lost, uncertain of the future and whether we can go on. Or so it seems in Amazon Mechanical Turk.
When I helped kickstart that use of MTurk at the Anderson School of Management at UCLA a few years ago, we needed a way to connect the data file on Mturk with the data file for our experiments. I’m sure I’m not the first person to think of this simple solution, but I suggested we assign every participant a unique code and have the participant enter it in the MTurk HIT before submitting it. It caught on there, and from the looks of the landscape its how everyone else does it too. It’s so ubiquitous, in fact, that when I stopped using them in favor of, in my opinion, a better practice, I started receiving 30 emails per study asking where their confirmation code was.
The problem with the confirmation code in MTurk is the error rate. Sometimes as many as 10% of people enter the code wrong. Also, it’s a pain to give each participant a unique code, and it’s annoying using these extra columns in the data files. I wanted to eliminate human copy & paste error, and utilize data already available. To do this I started storing each participant’s Amazon Worker ID in my experiment data files. The ID is automatically stored in the MTurk data file, and sending it and storing it in Qualtrics or other online survey systems is a breeze. The only problem, other than the uncertain feeling of the missing confirmation code, is that the worker has to accept the hit before you can access the ID. This is only a small problem.
You no longer have to have subjects enter anything in your Mturk HITs before submitting them. Mturk does require you have at least one input in your code, so I usually put a comment box:
<textarea id="comments"></textarea>, and let the users tell me about typos and stuff that I missed.
Storing the ID in Qualtrics is easy. All you have to do is create an embedded data element in the Survey Flow menu. Name the embedded data field workerId (all lowercase except for the I). You are done (leave the value unset). It will grab the value from the URL and store it in your Qualtrics file. The workerId field is already stored in the Mturk data file, so simply sort them, or use VLOOKUP in Excel, or however else you transfer who to accept and who to reject to Mturk.
While confirmation codes psychologically tell us that something actually happened and there is a documented way to look it up, in MTurk they are fraught with problems. I can only hope that by showing people some code that will make their research lives easier, it will also help me by diminishing the amount of worried email I receive each time I post a study.