the:behavioral:lab

Archive for the tag “qualtrics”

A better way to ask for currency responses in Qualtrics

Willingness to Pay questions (WTP) are ubiquitous in marketing research (my field), as well as many others. When asking for WTP in Qualtrics, you normally have to settle for some sub-par work-arounds. When asking in an open-answer text box, you can leave no validation, but you risk getting a lot of gibberish that both isn’t quite missing data, but isn’t quite usable data either. If you put a numerical validator participants may get annoyed at the inevitable error messages. They try to type something like $300, but to a computer, that’s not a number (it contains the dollar sign so it gets interpreted as a character string). You could also do things like slider bars, but that sets artificial anchors that could bias results.

I solve this problem with regular expressions. If you are not familiar with regular expression, they are a way of programming a pattern within a string of characters. For instance, an email address is any number letters, numbers, and certain special characters, followed by the @ symbol, followed any number of strings of letter, numbers and hyphens, that are optionally separated by periods, followed by a letter string with between 2 and 4 characters (its actually a little more complicated, but that covers 99% of email addresses). In regular expression coding, that is represented by the following: ^[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}$

If you think it looks complicated, its because it is. After 7 years of programming, when I write regular expressions, I still use cheat sheets. Recently, I decided I was sick of using Qualtrics’ numeric validator and inserting the instruction “Only use numbers and optionally a decimal point. Do not use dollar signs or commas.” every time I asked for a response in currency format. Qualtrics allow you to create your own validations, and one option is to match the inputted text to a regular expression. With some writing, some tinkering, some forum searches, some more tinkering, and a lot testing, I settled on the following expression to validate US currency (it is not difficult to change this to other formats).

^[+-]?\$?[0-9]{1,3}(?:,?[0-9]{3})*(?:\.[0-9]{2})?$

That regular expression optionally allows the person to specify positive or negative values ([+-]?), optionally followed by a dollar sign (\$?). The next part is a little complicated. The point of it is to optionally allow a thousands separator (a comma in the US, typically). I defined that as 1 to 3 numbers ([0-9]{1,3}). Optionally followed by a comma, and if it is followed by a comma, there must be 3 numbers after the comma ( (?:,?[0-9]{3}) ). That previous pattern can repeat infinitely ( * ). Optionally, it allows decimals by requiring a period followed by exactly 2 numbers ( (?:\.[0-9]{2})? ). The ^ at the beginning says that whatever is typed has to start there. The $ at the end says whatever is typed has to end there. Both together just means that the entirety of what they type has to match what is between the two symbols. Otherwise someone could type “asdlk;fjasdf$300.00” and it would validate, because it can find a valid string within the whole of the text.

I explained it out in detail in case you are in a different country with different currency symbols and formats. To change the dollar sign, just change the first dollar sign to your currency symbol (other dollar signs have special meaning and have nothing to do with currency, so leave them). If you don’t use commas as thousands separators, but use something else, change the comma within ?:,? to your thousands separator. If you don’t use thousands separators at all, you could just leave it the way it is assuming no one would use them if they don’t know they exist, or you could remove the ?:,? entirely. If you use something besides a period as a decimal indicator (e.g. a comma) replace the \. in ?:\. with your decimal indicator (e.g. ?:,).

The regular expression above matches as many formats that I could think of.

30
$30
$30.00
3030
3,030
$3,030.00
etc.

It isn’t perfect, however. Someone could put in a value like $3000000,000, and it would match, even though with only 1 thousands separator it hard to know what the person meant. I couldn’t figure out a way to require proper thousands separators if they are used. However, this kind of problem would be so rare, that I can’t imagine ever seeing it.

To use it, click on your text entry question. In the Validation Type area of the menu on the right, select Custom Validation. The logic should read IF [your question] [theres only 1 option for the second drop down menu] [MATCHES REGEX] [^[+-]?\$?[0-9]{1,3}(?:,?[0-9]{3})*(?:\.[0-9]{2})?$]. Note the brackets are only there to separate out the different drop down and text entry menus. Do not leave the brackets in place when pasting in the regular expression.

The final piece of the puzzle is how the data is stored. You could do some javascript work to have Qualtrics save the currency as a number. However, Excel is good at noticing currency and changing it to a number, so I figured I’d do the conversion in the data cleaning phase rather than adding complicated javascript to every question.

The death of the confirmation code… hopefully

The confirmation code is an interesting psychological experiment in and of itself. We receive them for every digital transaction that we take part in from purchasing a book on Amazon.com to paying for dinner with a credit card. In our minds that number is our guarantee that what we just experienced actually happened and that we did not merely purchase that copy of 50 Shades of Grey (what??? I mean A Song of Ice and Fire!) in our imaginations. Without one, in some instances we can survive. Who looks that confirmation code on their credit card receipts after all? However, in others we seem to be lost, uncertain of the future and whether we can go on. Or so it seems in Amazon Mechanical Turk.

When I helped kickstart that use of MTurk at the Anderson School of Management at UCLA a few years ago, we needed a way to connect the data file on Mturk with the data file for our experiments. I’m sure I’m not the first person to think of this simple solution, but I suggested we assign every participant a unique code and have the participant enter it in the MTurk HIT before submitting it. It caught on there, and from the looks of the landscape its how everyone else does it too. It’s so ubiquitous, in fact, that when I stopped using them in favor of, in my opinion, a better practice, I started receiving 30 emails per study asking where their confirmation code was.

The problem with the confirmation code in MTurk is the error rate. Sometimes as many as 10% of people enter the code wrong. Also, it’s a pain to give each participant a unique code, and it’s annoying using these extra columns in the data files. I wanted to eliminate human copy & paste error, and utilize data already available. To do this I started storing each participant’s Amazon Worker ID in my experiment data files. The ID is automatically stored in the MTurk data file, and sending it and storing it in Qualtrics or other online survey systems is a breeze. The only problem, other than the uncertain feeling of the missing confirmation code, is that the worker has to accept the hit before you can access the ID. This is only a small problem.

What is below is a simple JavaScript that will create a hyperlink that only sends people to a URL if they have accepted the HIT already. Once accepted and clicked, the worker’s workerId is appended to the URL as a query string variable.  The code can be copy and pasted into the Source window of the Mturk HIT template. Then you simply have to replace the surveyUrl with the URL of your survey. The code is purposefully rudimentary (for JavaScript at least) so anyone can use it. If you know JavaScript, you can edit the code how you like, using buttons and events in place of the anchor, etc.

<script type='text/javascript'>
  var surveyUrl="https://thebehaviorallab.wordpress.com";//The url you want to send people to.
  function gotoSurvey()
  {
    var href=window.location.href;//Get the url of loading document (in mturk is the iframe the HIT content is in, not the url of the page itself)
    var queryString={};//Create an empty object to dump query string variables in.
    href.replace(new RegExp("([^?=&]+)(=([^&]*))?", "g"),function($0, $1, $2, $3){queryString[$1] = $3;});//Use a regular expression as well the a nifty alternative second parameter for the String.replace() method to dump all query variables into the queryString object
    if(queryString['workerId']!=undefined)
    {
      window.open(addQueryVar(surveyUrl,'workerId',queryString['workerId']),'survey_window');//add workerId variable to URL and open popup. Edit the link text in case of popup blockers
      document.getElementById('survey_anchor').innerHTML="If your popup blocker prevented the survey window from opening, disable it and click this link again.";
    }
    else
    {
      //No workerId variable. Worker hasn't accepted HIT -> ask to accept
      alert("You have not accepted the HIT yet. Please do so before clicking this link");
      document.getElementById('survey_anchor').innerHTML="Please accept the HIT, then click this link again";
    }
  }
  function addQueryVar(url, name, value)
  {
    //Find anchor in URL since you can't add the query string after an anchor
    var fragmentStart = url.indexOf('#');
    if (fragmentStart < 0) fragmentStart = url.length;//no anchor so add variable to end
    var urlBeforeFragment = url.substring(0, fragmentStart);
    return urlBeforeFragment+(urlBeforeFragment.indexOf('?') < 0 ? '?' : '&')+encodeURIComponent(name)+'='+encodeURIComponent(value)+url.substring(fragmentStart);//if there is already query string variables add new variable with &, if not, add with ?, making sure to URI encode the content.
  }
</script>
<p>
  <a id="survey_anchor" name="survey_anchor" href="javascript:gotoSurvey()">Click here to go to the survey</a>
</p>

You no longer have to have subjects enter anything in your Mturk HITs before submitting them. Mturk does require you have at least one input in your code, so I usually put a comment box: <textarea id="comments"></textarea>, and let the users tell me about typos and stuff that I missed.

Storing the ID in Qualtrics is easy. All you have to do is create an embedded data element in the Survey Flow menu. Name the embedded data field workerId (all lowercase except for the I). You are done (leave the value unset). It will grab the value from the URL and store it in your Qualtrics file. The workerId field is already stored in the Mturk data file, so simply sort them, or use VLOOKUP in Excel, or however else you transfer who to accept and who to reject to Mturk.

While confirmation codes psychologically tell us that something actually happened and there is a documented way to look it up, in MTurk they are fraught with problems. I can only hope that by showing people some code that will make their research lives easier, it will also help me by diminishing the amount of worried email I receive each time I post a study.

Post Navigation