Why are survey questions randomized?

With the new UserReport survey release the concept of a serialized surveys is broken down.

A problem we have been facing is that webmasters have a hard time redefining surveys when they need to gather new information. After a certain number of questions, the survey gets to long. And every time you want to add something, you need to remove something to keep the length down. And removing something breaks historical data series which also must be considered.

In other words, making changes to an already running survey requires thought and attention. That is why we have introduced random rotation of questions in the new online survey.

Try to forget the concept of a survey as you know it!

Surveys are still suffering from the constraints imposed when a survey was something printed on a piece of paper. Instead, try to imagine a survey as a long list of questions you would like to know the answers to. Sort of a 'Questions and Answers'. Is the sky blue? How high is a mountain? etc.

With the new UserReport online survey, you don't really build a survey as a traditional serialized sequence of questions.

You just define questions you would like answered - Nothing else!

The UserReport survey system dynamically builds a custom survey for each and every visitor that takes the survey. We use a random selection of questions to make sure you get enough answers to each question. This means that no single survey will be longer than 20 questions.

When you edit your survey in the survey editor, you may preview your survey - either in full length or ramdomized like a respondent will experience it.

But how does it affect my data?

Let's say you have 20 questions in your survey and you collect 2000 interviews/responses. In this case, you got less than the max limit of 20 questions pr. survey, so you will get 2000 answers to each and every question.

If you increase the number of questions in your survey to 40 questions - and conduct another 2000 interviews - you will get 1000 answers to each of the 40 questions. But each participant will still experience the survey as being only 20 questions long.

Clever right? No matter how many questions you add, the survey experience will be the same. And all 40 questions can be crosstabulated with each other due to the fact that the order of each survey is randomized.

The randomization is a bit more advanced than just 'randomizing'.

The randomization algorithm will always try to split the 20 questions between 50% personal questions and 50% opinion questions. Why? Because if you ask to many personal questions, the participants will be bored and leave the survey. So we need to strike a nice balance between personal and opinion questions in the online survey.

Another important detail is that personal questions should alway be asked at the end of the survey. That will ensure higher completion rates. Opinions first - then personal questions. But don't worry - we automated that behaviour as well.

Finally - Questions about the overall satisfaction score + gender and age will always be asked in your survey. It would not be a user satisfaction survey without those questions. So they are (still) mandatory.

Attachments

Comments

  • Avatar
    nefeli85

    What if i need a standar order of questions? Sometimes this is necessary....

Powered by Zendesk