How do you randomize within a Qualtrics form without breaking timing?

Question

My current survey has a number of blocks. Each of those blocks contains a bunch of questions, each on their own page with a page break, and each with an associated timing question to limit time spent filling it out. I'd like to randomize the order of these questions, but not randomize the order of the timing questions. Right now, if I just randomize the entire block, the timing questions will get shuffled too.

Essentially my blocks look like this:

---- (page break)

1. timing question

2. quiz question

---

3. timing question

4. quiz question

---

5. timing question

6. quiz question

---- etc..

In the above example, I'd like to randomize the order of questions 2, 4, and 6.

Any suggestions on how to do this? Previously when this has been asked it has been suggested to put the questions into blocks and then randomize the order of the blocks, as in here: How to randomise order of survey items in Qualtrics where some items have to appear together?.

This is not really a workable solution for me - it would result in a really unwieldy number of blocks and a very hard to modify and upkeep survey.

Answer 1

I've since completed this survey, so I thought that I would update with what I ended up doing.

So, it turns out to not be possible to do randomization in an easy way when timing questions are involved, based on a review of online resources and talking to Qualtrics support. Timing questions simply don't support Qualtrics randomization, and there is not an advanced option to get around this.

In the end, I used the following solution:

1 - I divided each of my questions into a separate block which consisted of the timing question, the multiple choice question, and a feedback display.

2 - Instead of randomizing fully, I created 10 manual counterbalance conditions. I determined 10 different arrangements of my 8 questions that it seemed very unlikely that order effects would hurt my survey. For example, questions 1-8 in order, reverse order, and then shuffled randomly 8 additional times.

3- In my case, with 8 questions to randomize, I used the web services to randomly generate a number from 1-10, and then used a series of "if" statements which would trigger a conditional branch based on the random number, and then display the questions in a set order.

For my use case this ended up being the best solution I could find. Manually and fully randomizing all blocks would have required 8! = 40,000+ different conditions. Even using a script to generate a .qsf file with this survey design would not work -- in my testing Qualtrics crashed whenever I tried to run a survey of this size.

For those with fewer questions, programmatically creating a text file (either qsf or advanced text) may be a viable option. I found this link describing how to read qsf files in R and this explanation of import and exporting surveys helpful.

(I'm going to accept this answer as I believe it's the most comprehensive response to my question possible, but if an editor disagrees and would prefer it remain open feel free to revert)

Answer 2

I recently needed to crate very similar survey, and managed to get better randomisation:

1. As @verybadatthis each question went to its own block;
2. Go to Survey Flow, created a Randomizer Flow element;
3. Added blocks to the randomizer flow.

Providing introduction panels and outro panels for each logical survey section was also straightforward --- I created a group flow element that contained intro, outro and beforementioned randomiser. So in the end it everything looked like that:

Aforementioned appreach allowed me to easily randomize blocks of about 30 questions.

If you want more detalis here is my blog post on the matter: https://blog.askesis.pl/post/2019/04/qualtrics-random.html