Online surveys made empirical research much easier. However, they also resulted in new ways in which the quality of the data can be impaired - in particular, when careless people come across your survey. That is why, while creating your questionnaire, you must take into account those human errors and prevent them as early as possible. 

‍

Here, we explain the main possible biases and techniques, which help to avoid them.


Response bias

Response bias describes the behaviour when the participant has a tendency to always answer in a similar direction. This especially happens in rating scale questions. For example: 


  • "Towards the middle", in which the participants always tick a middle position, or
  • “Yes-saying” (Acquiescence bias), when the participant agrees to every question,
  • “No-saying”, when the participant denies everything.


The problem with this bias is, it leads to systemic errors in the survey data.


Speeders/Lurkers

Speeders or lurkers are the survey participants, who click through the survey regardless of its content. It’s often because they simply want to finish it quickly in order to receive a promised prize, or they want you to answer their survey in return.


Speeders can also lead to data distortions. Sometimes they are committing response bias (look above) in the process, but not always - that is why it’s so hard to spot them.

‍

Possible solutions:


  • Choosing a survey scale consciously

In general, 1-5 and 1-7 scales are considered to be the most effective. But they are not free of flaws.


First of all, the 1-7 scale allows a more diverse outlook, which will make it easier to spot any inconsistencies, such as response bias. However, the question occurs: will the well-meant participants consistently be able to differentiate between moderately agree and somewhat agree? Or between moderately disagree and somewhat disagree?

On the other hand, the 1-5 scale, although easier to comprehend, it’s also a paradise for speeders. It’s easy for them to choose a random option and avoid being inconsistent. But there are options to prevent that - keep reading to learn them.


Studies show that providing a numeric scale with verbal descriptions for endpoints only causes more extreme answers. So you also should be mindful about naming your scale.


Again - avoid scales without a neutral option, such as a 1-4 scale. Although participants opting for a neutral answer may annoy you, it’s better than them skipping a question, or dropping out. And for participants choosing neutral options only, you can discard them as a part of response bias.


  • Rotating the scales (not recommended)

Another approach is, to rotate the negative / positive poles of a scale randomly in order to keep participants on their toes. 

It may help to avoid bias to a certain extent - e.g. to spot inconsistent participants, who didn’t pay attention to a rotating scale. However, you also risk losing genuine participants, who simply didn’t notice a change and provided inaccurate answers due to it.

‍

If you want to use rotating scales, be sure to not overdo it. Ideally, use a computer program to randomise a direction of a scale for every participant.


  • Control and trap questions

A similar and far more effective approach to getting the participant's attention is to include control questions. Control questions are irrelevant for your research, but they are used to compare their responses to relevant questions. They might be a rephrased question asked before e.g. ‘Our website was easy to use’ vs. ‘It was easy to navigate our website’. It is a very good way to spot inconsistent users.


Another idea is to have so-called ‘trap’ questions. Trap question makes no sense in regards to a survey, yet they are put there to monitor the reliability of your participants.

‍

E.g. questions such as: “What’s your favourite TV show”, or even “Are you paying attention?”.


But be careful about asking too many control or trap questions: they can compromise the reliability of your survey. 


  • Time control

Probably the most useful tool to spot speeders. They will answer your survey suspiciously quickly in comparison to reliable users. Be sure to collect a data record of the time participants needed to complete your survey.  

‍

As a site, PollPool offers automatic time control for every survey on our platform. Speeders are immediately spotted and sanctioned by us. 


Click for the next part of the guide --> Your survey is ready. But what about participants?

‍