Question Appraisal System QAS-99 By: Gordon B.Willis and Judith T. Lessler Research Triangle Institute Suite 420 6110 Executive Blvd. Rockville, MD 20852 August, 1999 Acknowledgments This version of the RTI Questionnaire Appraisal System (QAS-99) is based on a system that we developed for Behavioral Surveillance Branch of the Centers for Disease Control and Prevention (Contract No.: 200-98-0103 TO#3) for use in evaluating questions for the Behavioral Risk Factor Surveillance System (BRFSS). The BRFSS-QAS can be obtained by contacting: Centers for Disease Control and Prevention National Center for Chronic Disease Prevention and Health Promotion Division of Adult and Community Health Behavioral Surveillance Branch 4770 Buford Highway, NE, MS K47 Atlanta GA 30341-3717 770.488.2455 http://www.cdc.gov/nccdphp/brfss Questionnaire Appraisal System QAS-99: -ii- Research Triangle Institute Table of Contents Section Page Section 1. Background and Instructions 1-1 Overview of the QAS 1-1 Learning to Use the QAS-99 1-2 Applying the QAS-99 in Practice 1-3 Section 2. QAS-99 Coding Form 2-1 Section 3. QAS-99 Manual 3-1 Instructions on Using the Manual 3-1 Purpose of the QAS-99 Manual 3-1 How to Read the Manual, the First Time Through 3-1 Two Issues to Consider, Up Front. 3-1 Finally, a Note About Terminology 3-3 STEP 1 - READING 3-5 Problem Type 1a - What to Read 3-5 Problem Type 1b - Missing Information 3-6 Problem Type 1c - How to Read 3-7 STEP 2 - INSTRUCTIONS 3-8 Problem Type 2a - Conflicting or Inaccurate Instructions . 3-8 Problem Type 2b - Complicated Instructions . 3-9 STEP 3 - CLARITY 3-10 Problem Type 3a - Wording 3-10 Problem Type 3b - Technical Terms 3-11 Problem Type 3c - Vague 3-12 Problem Type 3d - Reference Periods 3-16 STEP 4 - ASSUMPTIONS 3-17 Problem Type 4a - Inappropriate Assumptions . 3-17 Problem Type 4b - Assumes Constant Behavior . 3-19 Problem Type 4c - Double-Barreled 3-20 STEP 5 - KNOWLEDGE/MEMORY 3-21 Problem Type 5a - Knowledge 3-21 Problem Category 5b - An Attitude 3-23 Problem Type 5c - Recall 3-24 Problem Type 5d - Computation 3-25 Questionnaire Appraisal System QAS-99: -iii- Research Triangle Institute Table of Contents (continued) Section Page STEP 6 - SENSITIVITY/BIAS 3-26 Problem Type 6a - Sensitive Content 3-26 Problem Type 6b - Sensitive Wording 3-26 Problem Type 6c - Socially Acceptable 3-27 STEP 7 - RESPONSE CATEGORIES 3-28 Problem Type 7a - Open-Ended Question 3-28 Problem Type 7b - Mismatch 3-30 Problem Type 7c - Technical Terms 3-32 Problem Type 7d - Vague 3-33 Problem Type 7e - Overlapping 3-34 Problem Type 7f - Missing 3-34 Problem Type 7g - Illogical Order 3-35 STEP 8 - OTHER 3-36 Section 4. Example Questions 4-1 How to Code Problem Types: Two Detailed Example Questions . 4-1 Coding Form for Example Question 1 4-2 Coding Form for Example Question 2 4-10 Section 5. Practice Questions 5-1 Four Practice Questions 5-2 Answers to the Four Practice Questions 5-3 Questionnaire Appraisal System QAS-99: -iv- Research Triangle Institute Section 1. Background and Instructions The questionnaire appraisal system is designed to assist questionnaire designers in evaluating survey questions, and in finding and fixing problems, before the questions “go into the field.” We strongly believe that many improvements to questions can be made through the process of appraisal— the systematic review of a questionnaire, using a system like the QAS-99 as a guide. Our goal is to improve the efficiency of the questionnaire review process by providing developers with an easy-to-use method for identifying and correcting potential problems with draft questionnaire items. QAS-99 was based on several previous question appraisal systems and, in part, from a method developed to examine and classify the cognitive processes inherent in the question-answering process (Lessler and Forsyth, 1996). 1 This version is different from ones that we have used in other studies. In this version, we have eliminated codes that characterize the nature of the response task, that focus on mnemonic and judgment processes, and have, instead, focused on question characteristics that are likely to present problems when administered in the field. Overview of the QAS The QAS guides users through a systematic appraisal of survey questions and helps them to spot potential problems in the wording or structure of the questions that may lead to difficulties in question administration, miscommunication, or other failings. The user examines proposed questions by considering specific categories of question characteristics in a step-wise fashion and, at each step, decides whether the question exhibits features that are likely to cause problems. In completing the appraisal, the user indicates whether the problem is present by circling YES or NO on an accompanying coding form (see Section 2) and, for each YES circled, notes the reason a YES code was assigned. In brief, the steps are as follows, with a full discussion of each step given in the QAS-99 Manual (Section 3): 1 Lessler, Judith T., and Forsyth, Barbara H. (1996) “A Coding System for Appraising Questionnaires.” In Schwarz, Norbert, and Seymour Sudman (Eds.), Answering Questions: Methodology for Determining Cognitive and Communicative Processes in Survey Research. San Francisco: Jossey- Bass Publishers. Questionnaire Appraisal System QAS-99 1-1 Research Triangle Institute STEP 1: READING: Determine if it is difficult for the interviewers to read the question uniformly to all respondents. STEP 2: INSTRUCTIONS: Look for problems with any introductions, instructions, or explanations from the respondent’s point of view. STEP 3: CLARITY: Identify problems related to communicating the intent or meaning of the question to the respondent. STEP 4: ASSUMPTIONS: Determine if there are problems with assumptions made or the underlying logic. STEP 5: KNOWLEDGE/MEMORY: Check whether respondents are likely to not know or have trouble remembering information. STEP 6: SENSITIVITY/BIAS: Assess questions for sensitive nature or wording, and for bias. STEP 7: RESPONSE CATEGORIES: Assess the adequacy of the range of responses to be recorded. STEP 8: OTHER: Look for problems not identified in Steps 1 - 7. Learning to Use the QAS-99 You can learn to use the QAS-99 on your own by following these steps: 1. Review the QAS-99 Coding Form (Section 2). Although the way in which to use this form will not be immediately apparent, it provides a good overview of the system. 2. Read the QAS Manual (Section 3). The manual contains detailed discussion and examples of the various problems encountered in survey questions, instructions on how to “code” these problems on the coding form, and suggestions for correcting problems. As you read the QAS-99 manual, look at the corresponding part of the coding form, so you can associate the detailed discussion with the form. As you Questionnaire Appraisal System QAS-99 1-2 Research Triangle Institute begin using the coding form to appraise questions, it will be helpful to return to the manual until you are throughly familiar with how the codes are applied. 3. Examine the Example Questions (Section 4). Following the manual are two example questions that have been appraised using the QAS-99, with detailed explanations of why particular codes were assigned. Review these two question appraisals in detail before beginning to work with the system. 4. Practice using the QAS-99 by coding the four practice questions in Section 5. 5. Check your work by comparing your codes to those assigned by expert appraisers. The results from expert coders are given for all of the questions contained in Section 5. If you find substantial differences between the codes that you assigned and those of the expert coders, we recommend that you re-read the QAS-99 manual and attempt to understand why the coders assigned each particular code. Applying the QAS-99 in Practice QAS-99 is a tool for you to use to evaluate and revise draft questions. If one diligently applies the QAS-99, few questions may escape being assigned a problem code because no question is perfect for every respondent that will be encountered in a survey. Your goal is to understand the potential problems and to reduce them as much as possible. Thus, we recommend that you be strict in assigning codes and err on the side of finding a problem rather than not finding any. Otherwise, you will miss opportunities to consider more effective wordings and structures for the questions that you are drafting. You can use the QAS-99 in several ways: 1. Improving questions: You can use the QAS-99 to guide the revision of your own questions after you write them. Keep in mind that even after learning the system, and considering how to avoid various pitfalls, it is still important to apply the QAS-99, and to do the question coding, rather than just concluding that “now that I know what to avoid, I’ll write good questions.” Even very expert questionnaire designers benefit from considering a number of separate aspects of a question— systematically, and one at a time— and the QAS-99 forces one to do this. So, after you have drafted questions, use the QAS-99 to identify potential problems with them, then revise the questionnaire to remove these potential problems. Questionnaire Appraisal System QAS-99 1-3 Research Triangle Institute 2. “Flagging” questions for further testing: Because many questions may be subsequently evaluated through cognitive testing or field pretests, you can use your findings to highlight issues that you believe should be investigated in this subsequent testing. 3. Stimulating collaborative review: One particularly helpful way to use the system is to have one or more of your colleagues also evaluate the questions, and then discuss with you the findings and ways to improve the questions. Remember to focus on the respondent’s question-answering task when appraising the questions. In particular, think about the different types of respondents who may be administered the question: Think about how people of varying ages and different levels of education, income, and life experience may react to the questions. For example, if the question focuses on a particular health condition, remember that it will be asked of both people with and without the condition. Also, it is important to keep in mind the general conditions of the survey. If respondents are answering questions over the telephone, they may not have been expecting a call, and that there may be distractions in their environment that make it difficult for them to concentrate on their answers. Respondents may be reluctant to provide truthful answers to sensitive questions because they are concerned that the interviewer will judge them or that members of their household may overhear their responses. They are often in a hurry and may not have thought about the topics in the way that is demanded by the questionnaire. Your goal is to develop questions that are robust within these general survey conditions. Questionnaire Appraisal System QAS-99 1-4 Research Triangle Institute Section 2. QAS-99 Coding Form This section contains a sample QAS-99 coding form. Although the details of how to use this form are contained in the section that follows (Section 3), take a quick look at this form first to get an idea of the basic steps and problem types that are involved. Questionnaire Appraisal System QAS-99 2-1 Research Triangle Institute QUESTION APPRAISAL SYSTEM (QAS-99): CODING FORM INSTRUCTIONS. Use one form for EACH question to be reviewed. In reviewing each question: 1) WRITE OR TYPE IN QUESTION NUMBER. ATTACH QUESTION. Question number or question here: 2) Proceed through the form - Circle or highlight YES or NO for each Problem Type (1a 8). 3) Whenever a YES is circled, write detailed notes on this form that describe the problem. STEP 1 - READING: Determine if it is difficult for the interviewers to read the question uniformly to all respondents. 1a. WHAT TO READ: Interviewer may have difficulty determining what parts of the question should be read. YES NO 1b. MISSING INFORMATION: Information the interviewer needs to administer the question is not contained in the question. YES NO 1c. HOW TO READ: Question is not fully scripted and therefore difficult to read. YES NO STEP 2 - INSTRUCTIONS: Look for problems with any introductions, instructions, or explanations from the respondent’s point of view. 2a. CONFLICTING OR INACCURATE INSTRUCTIONS, introductions, or explanations. YES NO 2b. COMPLICATED INSTRUCTIONS, introductions, or explanations. YES NO Questionnaire Appraisal System QAS-99 2-2 Research Triangle Institute [...]... response categories that are read and those that are not 3-3 Questionnaire Appraisal System QAS-99 Research Triangle Institute The next pages detail all of the QAS-99 steps and problem types 3-4 Questionnaire Appraisal System QAS-99 Research Triangle Institute STEP 1 - READING: Determine if it is difficult for the interviewers to read the question uniformly to all respondents Problem Type 1a - WHAT... subjective judgments, and to create the best questions that are possible at this point The QAS-99 is a tool that guides us through this process In summary, remember that the overall purpose of the QAS-99 is not to find ways to criticize every survey question (or questionnaire designer!), but rather, to help the designer to find 3-2 Questionnaire Appraisal System QAS-99 Research Triangle Institute basic... previously identified 2-5 Questionnaire Appraisal System QAS-99 Research Triangle Institute Section 3 QAS-99 Manual Instructions on Using the Manual Purpose of the QAS-99 Manual The manual provides detailed directions on how to fill out the coding form (in Section 2) by following the eight basic steps of the QAS-99 These steps are arranged in an order that loosely mirrors the question asking-and-answering... of the QAS-99 as a series of fishing nets— if one net misses, another one may “make the catch.” 2 Because this version is derived from the project that was done for the Behavioral Surveillance Branch of CDC, most of the example questions are health questions that commonly occur in the BRFSS 3-1 Questionnaire Appraisal System QAS-99 Research Triangle Institute 2 “There are problems with every question. ”... 3-5 Questionnaire Appraisal System QAS-99 Research Triangle Institute STEP 1 - READING (continued) Problem Type 1b - MISSING INFORMATION: Information the interviewer needs to administer the question is not contained in the question ! Circle YES for 1b if … 1) The interviewer must remember something about the respondent, consult other materials, or return to another question to determine if the question. .. relates to variation in some attribute or behavior that the question does not take account of, then select 4b 3-19 Questionnaire Appraisal System QAS-99 Research Triangle Institute STEP 4 - ASSUMPTIONS (continued) Problem Type 4c - DOUBLE-BARRELED question that contains multiple implicit questions ! Circle YES for 4c if there are two or more questions in one Example 1 Are you covered by Medicaid or any... perfect survey question, and the QAS-99 is designed to find a lot of different types of potential flaws So, instead of just concluding that it is futile to script good questions, it may help to consider a few points that put the appraisal process in the appropriate context: ! Using a system such as this to critique the question in effect forces the user to consider many aspects of the question and to... sensitivity YES NO 6c SOCIALLY ACCEPTABLE response is implied by the question YES NO 2-4 Questionnaire Appraisal System QAS-99 Research Triangle Institute STEP 7 - RESPONSE CATEGORIES: Assess the adequacy of the range of responses to be recorded 7a OPEN-ENDED QUESTION that is inappropriate or difficult YES NO 7b MISMATCH between question and response categories YES NO 7c TECHNICAL TERM(S) are undefined,... (here, age) will have been collected previously 3-6 Questionnaire Appraisal System QAS-99 Research Triangle Institute STEP 1 - READING (continued) Problem Type 1c - HOW TO READ: Question is not fully scripted and therefore difficult to read ! Circle YES for 1c if the wording necessary to read the question is not provided This frequently occurs when questions are converted from a self-administered method... “arthritis,” whereas the question asks about a number of items that will not be recognized by most respondents as types of arthritis The question would be clearer without the introduction 3-8 Questionnaire Appraisal System QAS-99 Research Triangle Institute STEP 2 - INSTRUCTIONS (continued) Problem Type 2b - COMPLICATED INSTRUCTIONS, introductions, or explanations ! Circle YES for 2b if the question has extensive . involved. Questionnaire Appraisal System QAS-99 2-1 Research Triangle Institute QUESTION APPRAISAL SYSTEM (QAS-99) : CODING FORM INSTRUCTIONS. Use one form for EACH question. are not. Questionnaire Appraisal System QAS-99 3-3 Research Triangle Institute The next pages detail all of the QAS-99 steps and problem types. Questionnaire Appraisal System QAS-99 3-4. have drafted questions, use the QAS-99 to identify potential problems with them, then revise the questionnaire to remove these potential problems. Questionnaire Appraisal System QAS-99 1-3