Archive for the month “September, 2013”

Excluding Mturk workers from your HITs (not just from your Qualtrics surveys)

My method for excluding Mturk workers from Qualtrics surveys (found here) is rather easy to use, but is pretty much limited to Qualtrics and Survey Gizmo. It provides helpful insight into features of the two systems, but I thought I’d develop a method to help the broader Mturk user (and those who use Survey Monkey). This is a rather simple JavaScript based system for telling workers whether or not you have them on a black list. On the black list means they have done your task before (e.g. taken you survey). Off the list means they are AOK.

The system is flexible; there is plenty about it you can change. I put in a complete HTML simply so people who want to can just copy and paste, changing the necessary parameters. There are plenty of instructions (basically everything within double quotation marks) that you can change if you don’t like how it’s worded. It is also designed for survey use where your worker accepts the HIT and is immediately sent to an outside webpage (i.e. the survey). If you need the user to do something else, you’ll have to change what happens in the last conditional of the check() function.

The logic of the system is:
1) User enters
2) System checks if HIT is accepted (by seeing if there is a workerId query string variable in URL)
3) If not accepted
a. Ask for ID input
b. Check given ID against black list
c. If on list, tell not to accept. If not on list, tell to accept
d. Accepting HIT reloads the page, starting program over again (i.e. start back at 1)
4) If accepted
a. Get workerId variable from query string in URL
b. Check actual ID against black list
c. If on list, ask to return HIT (i.e. they lied about their ID before or did not follow instructions). If not on list, give hyperlink and instructions for going to your webpage (e.g. survey).

There are 4 parameters that need to be changed. The first two are simple HTML edits. Just replace the title and instructions with your own title and instructions. This can be done in the rich text editor if you wish after pasting this code into the Source code area of the HIT template creator. The third parameter is the URL to send users to. Just paste your Survey Monkey, Qualtrics, or some other URL between the double quotation marks. The last parameter is a litter more difficult. You need to create a quotation-mark enclosed, comma-separated list of IDs. Typically you do this in Excel by downloading your previous data files, compiling a list of workerId’s in a column then in the column next to it, use the concatenate function to add quotation marks and commas (e.g. putting =concatenate(‘”‘,A1,'”,’) in cell A2 will do what is needed, then copy the formula down for each cell). Copy and paste that list between the parentheses of the workers variable.

Last, note that this code will send the worker ID to your survey. In Qualtrics you can store the ID by creating an embedded data element and naming it workerId

Here is the commented code. Or you can get it from GitHub:

1:  <h1>TITLE - change this</h1>  
2:  <p>INSTRUCTIONS - change this</p>  
3:  <div id="idcheck">&nbsp;</div>  
4:  <p><textarea cols="80" name="comment" rows="3"></textarea></p>  
5:  <script type='text/javascript'>  
6:  var url="URL GOES HERE";//The URL that the user will go to after they are checked (i.e. the survey url). Change this.  
7:  var workers=new Array("test1","test1","test3");//This is your black list. Change this. All IDs must be enclosed in quoation marks (single or double), and separated by a comma  
8:  //Below is a utility function that checks if a given value is in an array. This will be used to check if the id (given value) is in the black list (an array)  
9:  Array.prototype.contains = function(k) {  
10:    for(var p in this)  
11:      if(this[p] === k)  
12:        return true;  
13:    return false;  
14:  }  
15:  //HTML instructions and form for ID input  
16:  var naCheck="<p>To see if you have completed this survey already, enter worker ID below and click \'Check ID\'</p><p><textarea rows='1' cols='20' id='idinput'></textarea></p><div id='ok' onclick='check(false)' style='border: 3px solid black;cursor: hand; cursor: pointer; background-color:gray;color:white;width:70px;height:20px;text-align:center;margin-left:10px;'>Check ID</div>";  
17:  var href=window.location.href.toString();//Get URL of mturk webpage which may or may not include the worker ID  
18:  var queryString={};//storage variable for query string variables  
19:  href.replace(new RegExp("([^?=&]+)(=([^&]*))?", "g"),function($0, $1, $2, $3){queryString[$1] = $3;});//Populates variable named queryString (created above) with the actual queryString variable names and values  
20:  //If queryString contains a variable names workerId, check if the black list contains that value of that variable, if not, ask the user to input their ID.  
21:  if(queryString['workerId']!=undefined)  
22:  {  
23:       check(queryString['workerId']);//Check ID against black list  
24:  }  
25:  else  
26:  {  
27:       document.getElementById('idcheck').innerHTML=naCheck;//Add HTML necessary to ask for id input  
28:  }  
29:  //This function will check the ID given against the black list, or if not given will check the ID input into the text box against the black list  
30:  function check(id){  
31:       if(id==false){  
32:            if(workers.contains(document.getElementById('idinput').value))  
33:            {  
34:                 //Input ID found on list -- taken the survey before, ask not to accept  
35:                 document.getElementById("idcheck").innerHTML="Your ID is on the list of workers who have completed this survey already. Please do not accept the HIT. Thank you for your previous participation.";  
36:            }  
37:            else  
38:            {  
39:                 //Input ID not found on list -- ask to accept, when this happens it will start the program over again, this time checking their actual ID not their inputted ID, which should be the same  
40:                 document.getElementById("idcheck").innerHTML="Your ID is not on the list of workers who have completed this survey. Please accept the HIT to continue.";  
41:            }  
42:       }  
43:       else  
44:       {  
45:            if(workers.contains(id))  
46:            {  
47:                 //Actual ID found on list -- ask to return HIT (this should rarely happen, if ever... it basically means the person did not follow instructions or tried to lie before hand  
48:                 document.getElementById("idcheck").innerHTML="Your ID is on the list of workers who have completed this survey already. Please return the HIT. If you previously entered your ID and were told to accept the HIT, you likely did not enter the correct worker ID.";  
49:            }  
50:            else  
51:            {  
52:                 //Actual ID not found on list -- send to survey  
53:                 document.getElementById("idcheck").innerHTML="<p>To proceed to the survey <a href='"+url+'&workerId='+id+"' target='mturksurvey'>click here</a>";  
54:            }  
55:       }  
56:  }  
57:  </script>  

Using YouTube for Experimental Video Display

[Headnote: WordPress, geniuses that they are, removes iframes and auto-embeds youtube urls, even if they are escaped and within code tags. Therefore, in my code here I use jframes and link to utube. If you want to just copy and paste code, make sure you search and replace jframe to iframe and utube to youtube and &amp; to &.]

One of the most common problems I help people solve is how to display video for experimental purposes. Youtube contains links to other videos, information about the title and uploader, etc. Experimenters don’t want to show that. There may be a better service for online video display for experiments (please mention them in the comments), but I find manipulating the YouTube player to be easy enough, and serve most of my purposes. Below is an email I wrote to a person who needed help with this very problem. Since I could not help the person in person, I included a lot of additional information to help understand what is really going on. For the advanced reader, ignore what I oversimplify, but feel free to correct what I got completely wrong. If you have a background in HTML you can probably just skip to the end. Also see footnote as to why the URLs don’t work.

Q: How can I display videos in Qualtrics? Youtube includes rewind buttons and other things I don’t want.

A: Your best bet is using YouTube and manipulating the player. If you have no experience with web development, what you need to do is probably completely new to you, but not too difficult to learn. Here’s an overview, skip to the end (the query string section) if you already know HTML and URL structures:

Embedding a youtube player inside Qualtrics is done using an Inline Frame (IFrame for short).

Ex. code

<jframe width="560" height="315" src="//" frameborder="0" allowfullscreen></jframe>

An iframe is an element on a webpage that itself is a webpage (i.e. a webpage within a webpage). That iframe webpage can be something you create, or it can be something from somewhere completely different, like a YouTube video. The webpage you load for YouTube videos is different than the one you normally view videos on. It is a page that only contains the video. Hence, on your page (your qualtrics survey) it looks like all you are doing is putting in a video player. Know that what you are really doing is putting in a separate web page that contains a video player.

That iframe element above has some attributes. The first is the element type, “iframe.” This tells the browser how to display it and how it functions. The second and third are width and height. This is the width and height of your frame. The youtube player in your target webpage is set to take up the entire page. Changing the values after width and height will change the size of your YouTube player (note: if the youtube player was not set to take up the entire page it is on, changing this will not change the youtube player size, just the size of your iframe element on your page, useless info for now, but may be important in the future). Skipping the src attribute for now, the last attribute is allowfullscreen. There is no value associated with it because it is simply a binary property. If it has this attribute, it is true, and each browser that supports HTML5 has a way of allowing your iframe to be made into full screen mode. If you remove this, it can’t be made into full screen mode. For experimental control, I usually remove this.

The src element tells the iframe where to load your target webpage from. If you copy and pasted that url above into your browser you will get a page that is entirely taken up by your youtube video (the URL I chose is for a different Michotte task video I just searched for). It is not a complete URL for a couple reasons. 1) there are some things missing (e.g. http or https)for purposes beyond what we need to talk about today. 2) Some elements of URL’s are not necessary each time. Below is a URL with every possible element. I will go over the elements.

The official syntax is below, and below that is a slightly more readable version




The scheme or protocol is usually http or https but there are others like ftp. Not important, just use whatever youtube spits out (which will match whatever the page you are loading the video on uses).

The domain contains three sub-components sub-domain, domain, and top-level domain. In the example, calendar is a subdomain. Basically, it is a section of the google website. Google is the domain. This is the primary identifier. com is the top-level domain. Used for organizing websites by purpose. Pretty much meaningless now (though it was important in 1992, I’m sure).

Ports you don’t have to worry about, if you are interested just go to Wikipedia.

The filepath is just like a file path on your computer except it is the path on a server. This is oversimplified, but basically how it works. There are folders on the server, to find your file it will go through the path you tell it to, to find the file you are trying to load. The file is usually a web page. The youtube example at the top is in a folder called embed. Within the embed folder is a folder called e_jKNlC2YKo. There is no file name. This is because there is a file inside that folder called index.html (or index.php or any of several file extensions). When no filename is specified it opens the index page. If no index page exists, the server makes one for you. It is just a list of files in the folder.

Skipping one section,the last part is the fragment id. This have many different meaning (again go to wikipedia for all of them). You won’t use it now, but basically what it’ll typically do is take you to a specific part of a website. For instance when you are on a wikipedia article (e.g. Albert Michotte’s,, and click on a section of the table of contents, it takes you to that part of the article. How it does that is by adding a fragment identifier to the URL (e.g. #Early_work), which takes you to the section of the article (the HTML element with the attribute id=”Early_work”).

The query string section is what is important for manipulating the player. A query string contains data that gets passed to the web page that is used to manipulate that page. There are variables and values. The query string starts with a question mark (?), then a variable name, an equals sign and a variable value (technically the equals sign and value are optional). If you have two or more variables, you separate them with a ampersand (&). For instance, the example at the top tells the page that an id which could be the id associated with your account. That tells the page to load content specific to your account. Then there is a variable for whether or not you are logged in. Adding random data doesn’t do anything. The webpage has to be programmed to use data that may be in the query string. YouTube is programmed with certain variables that it can get from the query string. These variables change how the youtube player operates. Here is a page that has all the parameters:

What you want to do is set controls to 0 (which removes them). This allows a person to click the play button to start a video, but not rewind the video (they can rewatch it though after everything is completed). There are others that are good for experimental control purposes. Setting modest branding to 1, uses a small youtube logo instead of baraging the viewer with the knowledge that they are on Youtube and not somewhere else. Setting showinfo to 0 removes the name of the video and who uploaded it. Setting rel to 0 removes suggested videos from the end of your video (you don’t want people viewing a funny cat video after your experimental task.).

So your Youtube url looks like this:

And your code for embedding the video into Qualtrics is this:

<jframe src="//;modestbranding=1&amp;showinfo=0&amp;rel=0" height="315" width="560" allowfullscreen="" frameborder="0"></jframe>

The only way to prevent people from rewatching a video is to have it autoplay (add &autoplay=1 to your url), and tell Quatlrics to autoadvance the page after X second (x being however long your video is). You can easily do this using the Timing question. There is a chance though that it takes a while for a person to load the youtube video. The page will still advance after, say, 5 second, even if it took the computer 3 seconds to load the video, and the person only saw 2 seconds of the video. You can control for this by asking people if they had any technical difficulties and just removing there rows of data if they did.

Pausing is still possible. If you want to prevent pausing, that only way I currently know how to do is put an invisible div html element on top of your youtube player. You also have to edit the player to change what is called the wmode. This isn’t in the docs, but sets the Adobe Flash version of the player to act like any other HTML element. Normally flash videos appear on top of anything else (we need a div on top of everything else). The wmode should always be set first in your query string. Not sure why, but all the forums I found said to do it that way. The div element needs to have some css properties set (done in the style attribute of the html tag). As you’ll see below you need to set the positioning type (either relative or absolute), the actual positioning (top and left), the z-index (telling it to display on top of elements with a higher z-index value) and the transparency (using both the internet explorer and the every other browser transparency style).

So your embedded player code looks like this:

<jframe src="//;controls=0&amp;modestbranding=1&amp;showinfo=0&amp;rel=0" height="315" width="560" allowfullscreen="" frameborder="0"></jframe>

Your div looks like this (and should be pasted exactly below your youtube player

<div style="position: relative;top:-315px;left:0px;width: 560px;height:315px;background-color: white;z-index:2;opacity:0.0;filter: alpha(opacity = 0)"></div>


<div style="position: absolute;top:0px;left:0px;width: 560px;height:315px;background-color: white;z-index:2;opacity:0.0;filter: alpha(opacity = 0)"></div>

The difference between absolute and relative is that relative will by default go whereever it is placed then you tell it to go up 315 pixels (top= -315 or -315 pixels from where the top normally is). Absolute starts at the 0 point of the parent element (in this case another div element that contains both your youtube player and your cover div). Since the youtube player displays at the 0 point, you don’t need to change the top or left style of your div. You can use either, in some scenarios, one might make more sense than the other. Width and height have to be set to exactly the width and height of your iframe. Background-color doesn’t really matter since it’s transparent. Z-index have to higher than the z-index of your iframe. By default it is 1 so 2 works here. If you want to be sure, set it to 50000. Opacity is the opacity for current versions of all browsers. Filter: alpha(opacity) is for IE 8 and earlier which a lot of people still use. Opacity goes from 0-1, filter:alpha(opacity) goes from 0-100.

Post Navigation