Archive for the category “mturk”

How to email multiple MTurk workers

I frequently want to contact Mechanical Turk workers who have completed studies for me. Sometimes they say they are interested in the study and want more debriefing info. Sometimes I just want to thank them for notifying me of a typo or error that I didn’t catch. Sometimes I want to offer a follow-up opportunity to people who have participated before. It is a violation of Amazon Mechanical Turk policies to require workers to divulge their email address or other identifying information. Some people (including me) have avoided outright breaking that policy by entering the grey area of “asking” for the email address but not making HIT completion contingent on entering a valid address. Another way to contact workers would be to use the Requester User Interface. However, its difficult to A) figure out whether it is possible to contact workers through the RUI, B) figure out how to do it, and C) send emails to multiple workers, since the RUI requires you to do it one at a time.

MTurk does allow you batch process emails to workers, but only through their API. The API allows people to build their own software to utilize Mturk features — more features than what is available on the website. One such feature is sending emails, and allowing a computer to send out 500 emails as opposed to doing it manually saves a lot of time. The API requires using the .NET Framework, which is only implementable in languages I don’t have much experience with. My goal was to create a web service to allow anyone to log in and send emails. However, earlier this summer I came across an R library on GitHub that implemented many, if not all, of the API’s resources in a single package. That library has recently (9/8/14) been published on CRAN (making it easier to find and use). I want to thank Thomas Leeper for saving me the time of both learning Ruby and programming a web service in Ruby. Since Mr. Leeper (Dr. Leeper I believe, actually) implemented it in R, it won’t function as a web service, unless someone feels like setting up an R server with it. However, it is probably better this way. Using the Mturk API requires telling the program your Amazon Web Services (AWS) credentials. With those credentials someone can have full access to your entire AWS account including credit cards on Amazon Payments and thousands of other pieces of information you’d rather not let people have. Thus, security is a concern, and using the Mturk API does carry some risks that you should be willing to accept before using it.

Below is a step-by-step guide to emailing workers. It is written assuming you have no knowledge of the R programming language.

  1. Install R and Rstudio. Rstudio is technically optional, but the rest of the instructions assumes you have it.
  2. Open Rstudio.
  3. In the console (lower-left is the default location), you should see a > on the bottom. Place the cursor there and paste the following:

    then hit Enter. The console should show the steps of installing (downloading, etc.) until it says that the package was sucessfully unpacked.

  4. Create a new R script by clicking File > New File > Rscript (or Ctrl+Shift+N on PCs).
  5. Copy the following code into the script:
     credentials(keypair=c("AccessKeyID goes here","SecretAccessKey goes here"))  
        subjects = "Email subject line goes here",  
        msgs = "Email body goes here",  
        workers = "WorkerId goes here"  

    #Note: The contact function above is setup to contact a single worker. You can send multiple workers the same message or customized messages to each worker in a single function call. To do so, you use vectors. For our purposes, think of a vector as a list. If you have a list of 100 email subject lines, 100 email bodies, and 100 worker IDs, R can send 100 custom emails. If you have a list of 1 email subject line, 1 email body, and 100 worker IDs, R will send the same email to 100 people. It can get more complicated, but I don’t know how it would applicable for this purpose. To create a vector, you use the c() function. Enclose all your text (e.g. worker IDs) in quotation marks and separate them with commas, like below.

        subjects = c("Email subject 1", "Email subject 2", "..."),  
        msgs = c("Email body 1", "Email body 2", "..."),  
        workers = c("WorkerId 1","Worker Id 2", "...")  
  6. In the line that starts with credentials, replace AccessKeyId goes here with your AWS access key and SecretAccessKey with your secret access key. To find your credentials, do the following:
    1.  Log in to the AWS Security Manager with the log in info for your Mturk account.
    2. Open the Access Keys panel.
    3. Click “Create New Access Keys.”
    4. Copy the Access Key and Secret Access Key and paste into the R script
      1. Note to those who know more than I do. It is my understanding that AWS no longer lets you view Secret Access Keys and you have to create a new one in order to see it. If there is a way to view a Secret Access Key without creating a new one, please email me or leave a comment.
  7. Save your R script. You will edit this file each time you want to send an email out, so put it somewhere you remember.
  8. To send an email, edit one of the two “contact” functions. The first one is for sending a single email to single worker (note you can only send emails to workers who have done work for you). The second one is for sending multiple emails (e.g. multiple emails to same worker [not recommended] or the same email to multiple workers or distinct emails to multiple workers). I’ll assume you want to send one email to multiple workers, and the instructions will edit the second contact() function.
    1. The parameter “subjects” is the subject line. You are assigning a list of subjects to that variable. If the list has only one entry, everyone will get the same subject. If you have 500 workers, you can create 500 separate subjects if you want. You just separate them by a comma. For the subject and every other item you will edit, make sure everything is enclosed within quotation marks.
    2. The parameter “msgs” is the body of the email. Similarly, you can create a list of separate email bodies or just put in one.
    3. The parameter “workers” is the WorkerId of the workers you want to contact. You can get worker IDs from the RUI if you don’t collect them yourself. Simply log into the Requester site, and go to Manage, then open a HIT posting, then download the CSV of the data. In the CSV are a lot of columns. One is labeled WorkerID. Entries in that column are what you are looking for. You can also view worker IDs in the main Review Results page. They’re in one of the default columns of results or, if they are not showing, you can click “Customize View” to have them show.
    4. Assuming you want to send one email to a list of workers, the contact function should look something like this
      1.  contact(  
            subjects = c("Email subject"),  
            msgs = c("Email body"),  
            workers = c("WorkerId 1","Worker Id 2", "...")  
  9. Execute the code.
    1. Place the cursor on the first line (starts with library) or highlight the entire line. Hit CTRL+Enter (for PCs) or Command+Enter (for Macs, I think). This loads the MturkR library.
    2. Place the cursor on the line that starts with credentials or highlight the entire line. Hit CTRL+Enter or Command+Enter. This stores your credentials temporarily, to use when the program communicates with Amazon’s servers.
    3. Highlight the entire contact function you wish to execute (i.e. the from the c in contact to final parentheses after all the worker ids). Hit CTRL+Enter.
    4. Look at the console to see if everything executed fine. You should see a list of workers that were notified. If some weren’t it’ll tell you why (e.g. worker ID invalid, worker hasn’t completed work for you, etc.). If you see an error, it may give you a hint as to why. For instance, perhaps you are missing an quotation mark surrounding your entries. Perhaps you are missing a comma between worker IDs.

Once everything has executed, you are finished. You can close the file, then reopen it next time, edit it again (step 8) then execute the new code (step 9). You only need to install the MturkR package once, but need to load it every time (the library command) and enter your credentials every time (the credentials function).

There are A LOT of other things the MTurkR library can do, such as setting up qualifications as well as qualifications tests, publishing and managing HITs, etc. I may tinker with other features and post my findings, but for now, MturkR is a great resource even if you only use it for contacting workers.


Preventing Mturk workers from prematurely submitting HITs (by altering the submit button’s functionality)

If you’ve ever used the Mturk web interface for creating HITs, you know it’s lacking in several areas. One that can be particularly annoying is the submit button that gets added to the end of every template you make. It looks like a plain HTML submit button, and is a half inch away from your normal content. I used to receive emails constantly from people who accidentally submitted their HIT before completing it. Never one to cry over 25 cents, I approved them, but answering those emails took too much time.

My initial solution was simple. I put about 10 blank lines at the end of my HIT template, following by a horizontal bar, followed by instructions to only click the submit button once the HIT is complete. That stopped about 99% of the emails. Which to me was a success. This is still all I do for my HITs, and if you are having this problem, just add the HTML code below to the end of your HITs.

 <p>After you have completed everything in this HIT, click the submit button below.</p>  

However, a friend recently wanted something more foolproof. I can’t remember the exact reasoning, but my friend wanted the submit button gone entirely. If you need 100% assurance (good luck you’ll never get it), you can try the method below. The logic is to first hide the submit button, which is a simple CSS modification. Once you want the user to submit the HIT, you can “click” the submit button for them by using the JavaScript click() function. What’s in the middle is really up to you. The code below replaces the submit button with a different button. That button queries a database which records when workers complete tasks within HITs. If a task is complete but the HIT is not submitted, the script submits the HIT. If the task is not completed, it alerts the user that they have not finished the task yet. This naturally assumes you have a database that records this information. Likely you do not. However, if you use confirmation codes or secret end-of-survey-passwords you can use JavaScript to check if the passcode is correct or not. If it is incorrect, they are either trying to cheat your out of 25 cents, or forgot the passcode (which is, of course, a problem).

 //Requires jquery. If you want to do the selection and the AJAX by hand, feel free, but its much easier for me to just implement Jquery.  
   //Hide submit button  
   //Create new submit button with validator function when clicked  
   $("<button/>").attr({'id':'newSubmit','type':'button'}).text("Check Submission").click(function(){  
    //Get location, which includes the worker ID, create an object to hold query string variables, and parse the query string  
       var href=window.location.href.toString();  
    var qs={};  
    href.replace(new RegExp("([^?=&]+)(=([^&]*))?", "g"),function($0, $1, $2, $3){qs[$1] = $3;});  
    //The url below would be whatever script you use to query the database. The query string includes the worker ID and the surveyId,  
       //so you know who completed what. The callback=? is jquery's method of doing JSON with padding, which is necessary due to the   
       //cross-domain scripting (since this will be implemented in mturk)  
       var url=""+qs['wid']+"&surveyId="+surveyId+"&callback=?";  
                     //Listed as completed in the database  
                     //Listed as something other than completed  
                     alert("Our databases indicate you have not completed the survey yet. Please make sure you have completed every portion of the HIT before submitting.");  

As I mentioned, there are a lot of problems with this kind of implementation. You’re risking screwing over a good worker to save yourself 50 cents, since unless the person is good at manipulating your HTML directly in their browser console (which if you didn’t know is very easy to do) they cannot submit the HIT. One quick fix for that is to replace the line of code toward the end with a confirm() message instead of an alert() message. The confirm message would tell the user that it looks like they haven’t completed the HIT yet, but they can submit anyway if they want to. That code would look like this:

 //Replace the line of code with the alert() function with these two lines of code.  
 var sub=confirm("Our databases indicate you have not completed the survey yet. Please make sure you have completed every portion of the HIT before submitting. If you think this is an error, and you have completed the HIT, click OK to submit the HIT. Click Cancel to continue working on the HIT.");  
 if(sub) $('#submitButton').click();  

That’s about all I can say about preventing workers from submitting HITs before they complete the task. Again, in my experience this is non-problem. Simple HTML formatting fixes it. If someone is nefarious, there is not a lot you can do to stop him or her from stealing your payment other than rejecting workers after the fact. My philosophy is provide the best experience I can for the workers, and I suppose adding the confirm() option isn’t a bad idea. I’ll leave with an alternative function that validates based on a password you give your workers at the end of your survey. It uses the confirm() not the alert() method since I think completely preventing someone from submitting a HIT will do more harm than good.

 //To use this function, copy it and replace the the function that is currently an argument in the click() function in the code above  
      //Someone can look up the password in the console, since they can see the Javascript, but its doubtful. Again, nothings ever foolproof.  
      var password = "PUT YOUR PASSWORD HERE";  
      //Get password entered by user. The code assumes your text box which the user puts the password in has the id: password_input  
      var pass = $('#password_input').val();  
      //If nothing was entered, alert user and exit function  
      if(pass=="") {alert("Please enter the password given to you in the survey before sumbitting the HIT.');return;}  
      //Create Boolean variable to test whether to submit or not  
      var submit;  
      pass==password ? submit=true : submit=confirm('The password you entered does not match the password displayed at the end of the survey. Please make sure the survey is 100% complete before submitting the HIT. If you would like to submit anyway, click OK. However, you run the risk of having your work rejected. If you would like to continue working on the HIT, click Cancel.');  
      if(submit) $('#submitButton').click();  

Post Navigation