the:behavioral:lab

Excluding Mturk workers from your HITs (not just from your Qualtrics surveys)

My method for excluding Mturk workers from Qualtrics surveys (found here) is rather easy to use, but is pretty much limited to Qualtrics and Survey Gizmo. It provides helpful insight into features of the two systems, but I thought I’d develop a method to help the broader Mturk user (and those who use Survey Monkey). This is a rather simple JavaScript based system for telling workers whether or not you have them on a black list. On the black list means they have done your task before (e.g. taken you survey). Off the list means they are AOK.

The system is flexible; there is plenty about it you can change. I put in a complete HTML simply so people who want to can just copy and paste, changing the necessary parameters. There are plenty of instructions (basically everything within double quotation marks) that you can change if you don’t like how it’s worded. It is also designed for survey use where your worker accepts the HIT and is immediately sent to an outside webpage (i.e. the survey). If you need the user to do something else, you’ll have to change what happens in the last conditional of the check() function.

The logic of the system is:
1) User enters
2) System checks if HIT is accepted (by seeing if there is a workerId query string variable in URL)
3) If not accepted
a. Ask for ID input
b. Check given ID against black list
c. If on list, tell not to accept. If not on list, tell to accept
d. Accepting HIT reloads the page, starting program over again (i.e. start back at 1)
4) If accepted
a. Get workerId variable from query string in URL
b. Check actual ID against black list
c. If on list, ask to return HIT (i.e. they lied about their ID before or did not follow instructions). If not on list, give hyperlink and instructions for going to your webpage (e.g. survey).

There are 4 parameters that need to be changed. The first two are simple HTML edits. Just replace the title and instructions with your own title and instructions. This can be done in the rich text editor if you wish after pasting this code into the Source code area of the HIT template creator. The third parameter is the URL to send users to. Just paste your Survey Monkey, Qualtrics, or some other URL between the double quotation marks. The last parameter is a litter more difficult. You need to create a quotation-mark enclosed, comma-separated list of IDs. Typically you do this in Excel by downloading your previous data files, compiling a list of workerId’s in a column then in the column next to it, use the concatenate function to add quotation marks and commas (e.g. putting =concatenate(‘”‘,A1,'”,’) in cell A2 will do what is needed, then copy the formula down for each cell). Copy and paste that list between the parentheses of the workers variable.

Last, note that this code will send the worker ID to your survey. In Qualtrics you can store the ID by creating an embedded data element and naming it workerId

Here is the commented code. Or you can get it from GitHub: https://gist.github.com/TheBehavioralLab/6770997

1:  <h1>TITLE - change this</h1>  
2:  <p>INSTRUCTIONS - change this</p>  
3:  <div id="idcheck">&nbsp;</div>  
4:  <p><textarea cols="80" name="comment" rows="3"></textarea></p>  
5:  <script type='text/javascript'>  
6:  var url="URL GOES HERE";//The URL that the user will go to after they are checked (i.e. the survey url). Change this.  
7:  var workers=new Array("test1","test1","test3");//This is your black list. Change this. All IDs must be enclosed in quoation marks (single or double), and separated by a comma  
8:  //Below is a utility function that checks if a given value is in an array. This will be used to check if the id (given value) is in the black list (an array)  
9:  Array.prototype.contains = function(k) {  
10:    for(var p in this)  
11:      if(this[p] === k)  
12:        return true;  
13:    return false;  
14:  }  
15:  //HTML instructions and form for ID input  
16:  var naCheck="<p>To see if you have completed this survey already, enter worker ID below and click \'Check ID\'</p><p><textarea rows='1' cols='20' id='idinput'></textarea></p><div id='ok' onclick='check(false)' style='border: 3px solid black;cursor: hand; cursor: pointer; background-color:gray;color:white;width:70px;height:20px;text-align:center;margin-left:10px;'>Check ID</div>";  
17:  var href=window.location.href.toString();//Get URL of mturk webpage which may or may not include the worker ID  
18:  var queryString={};//storage variable for query string variables  
19:  href.replace(new RegExp("([^?=&]+)(=([^&]*))?", "g"),function($0, $1, $2, $3){queryString[$1] = $3;});//Populates variable named queryString (created above) with the actual queryString variable names and values  
20:  //If queryString contains a variable names workerId, check if the black list contains that value of that variable, if not, ask the user to input their ID.  
21:  if(queryString['workerId']!=undefined)  
22:  {  
23:       check(queryString['workerId']);//Check ID against black list  
24:  }  
25:  else  
26:  {  
27:       document.getElementById('idcheck').innerHTML=naCheck;//Add HTML necessary to ask for id input  
28:  }  
29:  //This function will check the ID given against the black list, or if not given will check the ID input into the text box against the black list  
30:  function check(id){  
31:       if(id==false){  
32:            if(workers.contains(document.getElementById('idinput').value))  
33:            {  
34:                 //Input ID found on list -- taken the survey before, ask not to accept  
35:                 document.getElementById("idcheck").innerHTML="Your ID is on the list of workers who have completed this survey already. Please do not accept the HIT. Thank you for your previous participation.";  
36:            }  
37:            else  
38:            {  
39:                 //Input ID not found on list -- ask to accept, when this happens it will start the program over again, this time checking their actual ID not their inputted ID, which should be the same  
40:                 document.getElementById("idcheck").innerHTML="Your ID is not on the list of workers who have completed this survey. Please accept the HIT to continue.";  
41:            }  
42:       }  
43:       else  
44:       {  
45:            if(workers.contains(id))  
46:            {  
47:                 //Actual ID found on list -- ask to return HIT (this should rarely happen, if ever... it basically means the person did not follow instructions or tried to lie before hand  
48:                 document.getElementById("idcheck").innerHTML="Your ID is on the list of workers who have completed this survey already. Please return the HIT. If you previously entered your ID and were told to accept the HIT, you likely did not enter the correct worker ID.";  
49:            }  
50:            else  
51:            {  
52:                 //Actual ID not found on list -- send to survey  
53:                 document.getElementById("idcheck").innerHTML="<p>To proceed to the survey <a href='"+url+'&workerId='+id+"' target='mturksurvey'>click here</a>";  
54:            }  
55:       }  
56:  }  
57:  </script>  
Advertisements

Single Post Navigation

10 thoughts on “Excluding Mturk workers from your HITs (not just from your Qualtrics surveys)

  1. Kaitlin on said:

    Thank you so much, this is really helpful!

  2. Pingback: Allow only specified workers to complete your Mturk HITs | the:behavioral:lab

  3. Pingback: Changing Mturk submit button functionality & A new to prevent duplicate workers on separate HITs | the:behavioral:lab

  4. Rachel on said:

    Hi,
    I am trying to use this in my survey. It works when I leave “test1”, “test2” as the blacklisted worker IDs, but when I paste in my list of IDs, it no longer works the same. I followed all instructions and have quotations marks surrounding each ID and a comma in between. I have a list of several hundred people I want to exclude, perhaps the length is the problem?
    Thanks!

  5. Length shouldn’t be a problem. Send your code to andrew.r.long@colorado.edu.

    It could be that one item is missing a quotation mark, or one comma is missing, or a closing parentheses got deleted or something like that. You could check if there is an error in the javascript by opening up the javascript console window (for instuctions based on your browser go here: http://siliconforks.com/doc/debugging-javascript/)

    For instance in Chrome, it may say UNEXPECTED TOKEN var and then give you the line of code. You can then look at that line or the preceeding lines and look for potential problems. If you have no experience with coding this may be difficult though, so feel free to send me what you have.

  6. Hi,
    Thank you very much for your script. I try to use it in order to prevent duplicates. It’s works fine for two workers and then I didn’t get any participation. One of the workers send me an email and says that the script works fine but the link was broken. I checked and the link to the survey works fine for me. Do you have any idea about what is happening?
    (I’m sorry for my bad english I’am not a native speaker).
    Thank you very much for your answers.
    Jennifer

    • Can you test it and send me the URL that the link goes to. In most browsers you can right click a link and select “Copy link address” or something similar in your browser’s language.

      • jb8438 on said:

        Hi,
        Thanks a lot for your quick answers. I tried it again using the sandbox this time, and think that I can see where is the problem but I don’t know how to fix it (I am very bad in java script).
        Here is the URL that the link goes to : https://fr.surveymonkey.com/s/ITRAH&workerId=ASFUQKXST9RS0

        It seems that the script adds “&workerId=ASFUQKXST9RS0” after the link to the survey. Do you know how to fix it? (I’m sorry if it’s a stupid question)
        Thanks a lot in advance

      • That is definitely the problem. The script was written for people who use Qualtrics. To fix it, you can do one of 2 things. 1) Find the line (#53 in the blog post) that starts as document.getElementById(“idcheck”).innerHTML=”

        To proceed to the survey… Shortly after that there is a the following string of letters: ‘&workerId=’+id+ Remove it. You should be left with: document.getElementById(“idcheck”).innerHTML=”

        To proceed to the survey click here“;
        2) Do the same thing, but instead of erasing a whole section, change the & to a ?. So it would look like: document.getElementById(“idcheck”).innerHTML=”

        To proceed to the survey click here“;

        Either works.

      • jb8438 on said:

        Great ! Thanks a lot !

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: