Conducting Statewide Conducting Statewide Customer Satisfaction Customer Satisfaction Surveys Surveys Brad Sickles U.S. Department of Labor Employment and Training Administration Measuring Customer Measuring Customer Satisfaction Satisfaction T he T he Measures Measures  Participant Customer Satisfaction Participant Customer Satisfaction  Employer Customer Satisfaction Employer Customer Satisfaction Customer ratings on each of the three core Customer ratings on each of the three core questions regarding satisfaction are questions regarding satisfaction are weighted to account for regional weighted to account for regional differences and reported on a 1 to 100 differences and reported on a 1 to 100 scale. Computation of the ACSI is scale. Computation of the ACSI is discussed in the WIA reporting guidance. discussed in the WIA reporting guidance. The American Customer Satisfaction The American Customer Satisfaction Index (ACSI) is used to report statewide Index (ACSI) is used to report statewide results to the Department. results to the Department. The Core ACSI Questions The Core ACSI Questions 1. Utilizing a scale of 1 to 10 where “1” means “Very 1. Utilizing a scale of 1 to 10 where “1” means “Very Dissatisfied” and “10” means “Very Satisfied” what is your Dissatisfied” and “10” means “Very Satisfied” what is your overall satisfaction with the services provided from overall satisfaction with the services provided from ________? ________? 2. Considering all of the expectations you may have had 2. Considering all of the expectations you may have had about the services, to what extent have the services met about the services, to what extent have the services met your expectations? “1” now means “Falls Short of Your your expectations? “1” now means “Falls Short of Your Expectations” and “10” means “Exceeds Your Expectations” and “10” means “Exceeds Your Expectations.” Expectations.” 3. Now think of the ideal program for people in your 3. Now think of the ideal program for people in your circumstances. How well do you think the services you circumstances. How well do you think the services you received compare with the ideal set of services? “1” now received compare with the ideal set of services? “1” now means “Not very close to the Ideal” and “10" means “Very means “Not very close to the Ideal” and “10" means “Very Close to the Ideal.” Close to the Ideal.” Participant Customer Participant Customer Satisfaction Satisfaction T he T he Measures Measures The weighted average of participant ratings on The weighted average of participant ratings on each of the three questions regarding overall each of the three questions regarding overall satisfaction reported on a 1 to 100 scale. satisfaction reported on a 1 to 100 scale.  Who: Who: Participant with a case closure Participant with a case closure  When: When: After case closure and no later than After case closure and no later than 60 days after date of case closure 60 days after date of case closure  How: How: Telephone survey Telephone survey  Required completions: Required completions: 500 statewide 500 statewide  Required response rate: Required response rate: 70% 70% Employer Customer Satisfaction Employer Customer Satisfaction T he T he Measures Measures The weighted average of employer ratings on The weighted average of employer ratings on each of the three questions regarding overall each of the three questions regarding overall satisfaction reported on a 1 to 100 scale. satisfaction reported on a 1 to 100 scale.  Who: Who: Employers who receive a substantial Employers who receive a substantial service. service.  When: When: 1to 60 days after date of the 1to 60 days after date of the completion of a service. For employers who completion of a service. For employers who listed a job order where no referrals were made, listed a job order where no referrals were made, contact should occur 30 to 60 days after a job contact should occur 30 to 60 days after a job order was listed. order was listed.  How: How: Telephone survey Telephone survey  Required completions: Required completions: 500 statewide 500 statewide  Required response rate: Required response rate: 70% 70% Capturing Customer Satisfaction Capturing Customer Satisfaction Results to Assess Local WIBs Results to Assess Local WIBs  States may use the ACSI questions and approach to measure States may use the ACSI questions and approach to measure satisfaction at the local level satisfaction at the local level  States may also develop their own customer satisfaction States may also develop their own customer satisfaction instruments and approach for assessing local WIBs instruments and approach for assessing local WIBs  Approaches must be applied Approaches must be applied uniformly and consistently to all uniformly and consistently to all WIBs WIBs  WIA reporting guidance outlines WIA reporting guidance outlines parameters for collecting customer satisfaction results parameters for collecting customer satisfaction results  Response rates: 70%. Response rates: 70%.  Response levels below the specified minimums Response levels below the specified minimums will will invalidate the results. invalidate the results.  States are required to determine the appropriate sample States are required to determine the appropriate sample sizes. States are no longer required to draw minimum sizes. States are no longer required to draw minimum sample sizes of 1,000. sample sizes of 1,000.  Respondents Respondents must must be told responding to the survey is be told responding to the survey is voluntary and the information will be kept confidential. voluntary and the information will be kept confidential.  States States must must create a process for creating an up-to-date create a process for creating an up-to-date customer list, capturing information on each customer’s customer list, capturing information on each customer’s address and telephone number. address and telephone number. Surveys to Obtain State Level Surveys to Obtain State Level Results Results  A survey is c A survey is c omplete omplete when valid answers are provided when valid answers are provided by the respondents for each of the core ACSI questions. by the respondents for each of the core ACSI questions. Valid answers are responses 1 through 10. Valid answers are responses 1 through 10.  The calculation of the ACSI score includes only the The calculation of the ACSI score includes only the results from results from complete complete surveys. surveys.  The introductory statements for both the participant The introductory statements for both the participant survey and the employer survey were revised to better survey and the employer survey were revised to better set the context for the interviews. set the context for the interviews. Surveys to Obtain State Level Surveys to Obtain State Level Results Results  Local program staff must inform the customer at the Local program staff must inform the customer at the time of participation about the importance of satisfying time of participation about the importance of satisfying customers and the possibility of being contacted for customers and the possibility of being contacted for information on his or her experience with the services. information on his or her experience with the services.  The interview should be limited to 15 minutes or less. The interview should be limited to 15 minutes or less.  A minimum of five follow-up attempts is required, A minimum of five follow-up attempts is required, involving various times of the day before closing the involving various times of the day before closing the record. record. Surveys to Obtain State Level Surveys to Obtain State Level Results Results [...].. .Surveys to Obtain State Level Results   Local programs should collect alternate contact information A letter in advance of the survey should be sent out informing the customer that he or she can expect to be contacted about his or her satisfaction with the services Creating Questions to Supplement the ACSI Customer Satisfaction Not the Same as Customer Service   Customer service... Candid Characteristics of Firms With High Customer Satisfaction        Customers define quality Variety of services Customization of services Convenience of services Timeliness of services Continually identify factors that influence satisfaction Continually identify expectations and set customer service standards well above these expectations (to “delight” customers) Writing Effective Questions... Not the Same as Customer Service   Customer service is measured by standards set by you or your agency Customer satisfaction is measured by the customer s standards for the services, regardless of whether they make sense Drivers of Satisfaction Service Quality Measures • • • • • • • • • • • Customer Service Characteristic • Convenience • Accessibility • Ease of use • Timeliness • Safety • Reliability... Hayes, Bob E Measuring Customer Satisfaction: Development and Use of Questionnaires Milwaukee: ASQC Quality Press,1993 Payne, S L The Art of Asking Questions Princeton, NJ: Princeton University Press, 1951 Schuman, H and Presser, S Questions and Answers in Attitude Surveys: Experiments in Question Form, Wording, and Context New York: Academic Press,1981 Simply Better! The Voice of the Customer, Second Edition... Training Administration, 1998 Sudman, S and Bradburn, N Asking Questions: A Practical Guide to Questionnaire Design San Francisco, CA: Jossey‑Bass Inc., 1982 Vavra, T G Improving Your Measurement of Customer Satisfaction Milwaukee: ASQC Quality Press, 1997 ... set customer service standards well above these expectations (to “delight” customers) Writing Effective Questions         Remember your survey's purpose: to provide you feedback on the level of satisfaction If you have doubts about a question, discard it State your questions simply Stay focused when writing questions; avoid vague areas If a question can be misinterpreted by the respondent, chances... question Avoid questions that lead respondents Consider optional ways to ask questions that deal with sensitive areas Writing Effective Questions  Create questionnaire items to determine the key drivers of satisfaction Constructing Effective Response Items      Answer options need to be mutually exclusive and exhaustive Keep open‑ended questions to a minimum Respondents interpret time-oriented responses . Conducting Statewide Conducting Statewide Customer Satisfaction Customer Satisfaction Surveys Surveys Brad Sickles U.S Measuring Customer Measuring Customer Satisfaction Satisfaction T he T he Measures Measures  Participant Customer Satisfaction Participant Customer Satisfaction  Employer