Best practices: developing a questionnaire

A good evaluation should meet the following criteria:

Reliability – it should accurately measure the knowledge acquired

Efficiency – it should make good use of every user’s time and only ask questions influencing the final score

Understandability – a good evaluation should be easily understood, and the purpose of every question very clear. Ambiguity must be avoided.

The number of questions is one of the most important feature to take in account. To be reliable, a test should have at least 25 questions.

There exists a mathematical formula to accurately determine the right number of questions based on learners’ previous results.

Putting aside the details of the formula, it is important to keep in mind the following point:

A high number of questions brings reliabilty, allows a better coverage of a specific topic, and diminishes the memorisation bias when retesting.

Once answered, quizzes often do not offer explanation. They sometimes do not even display the right answer.

In theses cases, the questionnaire is limited to its function of checking knowledge, and gives up on the training aspect. This logic makes sense in a very specific case: when you only want to evaluate your learners’ knowledge.

In every other case, it has been shown that the learner would benefit from a feedback, i.e. an explanation and possibly an analysis of his answer.

In particular, after having analysed the question, thought about the options and answered, the user is in a favourable intellectual condition to learn something.

Moreover, if the learner failed the question, he will certainly feel the need to know the right answer. This state of mind should be exploited to acquire knowledge.

A basic principle to know if a questionnaire is relevant is to make sure that an expert would know every answer.

If he would fail a question, this is not a good one. These “bad” questions often rely on details or rote knowledge.

If the expert has to do research to write a question, it is a sign that this question is likely to be irrelevant.

There exist many studies on good practice in designing questionnaires. Here is a list of the main features you should consider:

Every question should assess a specific point, a particular knowledge

The set of questions should cover the topic you want to evaluate

The phrasing of questions must avoid ambiguity (i.e. confusing words such as “sometimes”, “often”, etc.) and focus on a single notion

The phrasing of questions must perfectly coincide with the answers

The phrasing of answers should be similar to the one of questions (same length, same lexical field, etc.)

The correct answer(s) must be unambiguous