Experimental questionnaire design
This tutorial is the first in a series of tutorials in which I show how you can use Limesurvey to design experimental questionnaire designs. In this first tutorial we’ll deal with how we can randomly assign respondents to different questionnaires.
A while ago my research group wanted to carry out a questionnaire study in which we randomly assigned respondents to one of two conditions. In both conditions, respondents listened to small audio excerpts. However, respondents read different introductory texts about the clips they were about to listen to. Finally, we had three different audio clips, which each was manipulated prosodically in 5 different ways.
Including the originals we had 18 different audio clips. However, respondents only listened to 3 videos clips each. To make things a bit more complicated respondents listened to three different clips (and not just 3 versions of the same clip), and they listened to three different kinds of manipulations. So, if a respondent listened to manipulation 1 of clip 1, he or she did not listen to any version of clip 1 again, or listen to manipulation 1 of any of the subsequent clips. Just to top it off, the order of all questions and audio clips were randomized, and everything was contained within a single questionnaire. This rather complex design is visualized below:
Every red circle represents steps that one particular respondent went through. That is, they answered questions about their, age, education, gender, etc. They were then met by one of the two introductory texts. They then listened to one of the six manipulations of clip 1 (in this case, manipulated with method number 2), answered some questions related to that clip, and repeated this process for clips 2 and 3 as well. Over the course of three posts we try to replicate this research design. For this post we’ll focus on how we randomly assign respondents to one of two introductory framing texts.
First create a new questionnaire with 2 new question groups. The first of these we will call Demographics, the second group we’ll call Intro. Alternatively, you can also download this survey structure, which contains the complete questionnaire.
Lesson 1: Randomly assign respondents to different questionnaires
First we want to randomly assign respondents to framing text 1 or 2. To do just that we’ll use a function called rand, which will output a number randomly, within a range we define. We’ll then match each of the framing texts with a number from the range we defined earlier, with the condition designer.
Step 1: Create equation
First you go into the question group that comes right before the items you want to randomize. For my example, this would be the Demographics question group. Here you make a new question of the question type Equation (find it under category Mask questions):
Tick also the Mandatory box, so that it’s green like in the screenshot above. Then click on the Advanced settings and set the Always hide this equation to Yes:By doing this we make sure that respondents won’t be able to see the manipulations we make “behind the screen”. For the question itself we click on source and then write:
The function will return a number between 1 and 2 each time the function runs, that is for each new respondent. If we instead of two conditions (or framing texts) we need 3 or 4 we simply need to replace number 2 with the the number of conditions.
Step 2: Filter questions displayed based on equation
Next we need to go to the Intro question group and make sure it’s placed after the group containing the rand function. We then create a text display for each condition. Add a new question and find the text display in the category mask questions. Next we click on the first question in the Question explorer and click on Set conditions in the horizontal top menu:
Scroll down to heading Add condition, click on the equation containing the rand function, make sure the Comparison operator is set to equals, click on Constant, insert 1 into the textfield and click Add condition:
Now you only need to repeat this step for the other text display (remember, we added 2). However, for the second display we will replace the 1 with 2 as the constant. Then when the rand function randomly returns a 1 respondents will see framing text 1, and see framing text 2 if the functions return a 2.
You can also use the condition manager to screen out respondents our show different things dependent on age, gender, previously answered questions, etc. There is a guide here that explains how to do just that.
Now when respondents enter the questionnaire, they will get to read either intro1 or intro2. We can also take this even further and ask participants completely different questions using the same technique.
Stay tuned for the next tutorial in which we’ll work with we can conditionally display a set of questions from a pool of questions.
Note 1: It’s important that any question affected by the rand function is not mandatory. If that is the case LimeSurvey will throw and error when completing the questionnaire saying that not all questions are answered. This happens because only 1 of 2 questions were answered (when randomizing between to questions).