How attention filter format affects responses on Mturk
Attention filters or known-answers are the most common way to gauge the responders level of attention or effort. If a question where the answer is given to the participant or easily known (e.g. 2+2 = ?; or telling the answer in instructions before the question) is answered incorrectly, you have a good idea that the person was breezing through and they either should not be paid or you should not use their data.
When I first started using Mturk, the attention filter fail rate was about 5% in my surveys, which is phenomenal for the cost. I attributed this to the fact that people on Mturk are worried about not getting paid, so more attention is put into work. However then I had a string of surveys where the fail rate was around 50%. This is terrible and justification to not publish results even if they are good. I decided to try to figure out the cause.
A few weeks ago I wrote about how workers sometimes don’t attend to instructions in order to make the survey go faster. Workers are used to quick tasks, and repetitive multiple choice questions are therefore better than long instruction sets. I ran two studies and each contained the same attention filter. The filter was one that gave the answer in a paragraph before the question. Then asked a simple question where the correct answer (the one given) would not be the correct answer to the question. Like “Answer ‘8’ to the following question. 2+2=___.” The question was always bolded, but the instructions were bold in one study and not bold in another. When bold the fail rate was 7%. When not bold the fail rate was 45%. The simple act of making the instructions look like instructions made people not read them.
Because of this, it is important both to construct attention filters in a way that they measure what you want to measure (e.g. do you want to know if instructions are being read or if question text is being read) and it is important to format instructions in a way that they look like questions.
Questions get read, instructions don’t.