418 Teil C 1. Gesamteindruck: 2. Die Ausdrucksweise des Systems war 3. Das System reagierte 4. Sie hätten eine bessere Hilfestellung vom System erwartet: 5. Das System konnte alle meine Fragen beantworten: 6. Missverständnisse konnten leicht ausgeräumt werden: 7. Das System hat den Gesprächsverlauf bestimmt: 8. Sie konnten ohne Probleme mit dem System umgehen: 9. Sie sind von den Gesprächen mit diesem System 10. Die Gespräche haben Spaß gemacht: 11. Sie fühlen sich über die Fähigkeiten des Systems ausreichend informiert: 12. Der Anruf beim System hat sich Gelohnt: extrem schlecht schlecht dürftig ordenttich gut ausgezeichnet ideal klar unklar höflich unhöflich trifft zu tritft nicht zu trifft zu trifft nicht zu trifft zu trifft nicht zu trifft zu trifft nicht zu trifft zu trifft nicht zu beeindruckt enttäuscht trifft zu trifft nicht zu trifft zu trifft nicht zu trifft zu trifft nicht zu Questionnaires 419 13. Sie empfanden die Auskunftsmöglichkeit als 14. Sie schätzen das System ein als 15. Sie verwenden lieber eine andere Auskunftsquelle: 16. Die Bedienung des Systems war 17. Sie bevorzugen eine mit einem Menschen besetzte Auskunfts- stelle: 18. Sie würden dieses System noch einmal Benutzen: 19. Was hat Ihnen am System gefallen? 20. Was hat Sie am System gestört? 21. Haben Sie Verbesserungsvorschläge für das System? hilfreich nicht hilfreich vertrauenswürdig zweifelhaft trifft zu trifft nicht zu einfach kompliziert trifft zu trifft nicht zu trifft zu trifft nicht zu 420 Nachdem Sie nun Erfahrungen mit BoRIS gemacht haben, bitten wir Sie, die folgenden Fragen noch einmal zu beantworten. Wenn Sie bei einem Restaurant-Auskunftssystem anrufen, wie wichtig ist es Ihnen 22.1 ein normales Gespräch zu führen wie mit einem Menschen? 22.2 von einer freundlichen Stimme bedient zu werden? 22.3 dem System Fragen stellen zu können? 22.4 vom System Fragen (Scar übre Vorlieben gestellt zu bekommen? 22.5 schnell die gewünschte Information zu bekommen? 22.6 das System leicht bedienen zu können? 22.7 vom System erklärt zu bekommen, wie es Ihnen helfen kann? Viele n Dank für Ihre Mühe! wichtig nicht wichtig wichtig nicht wichtig wichtig nicht wichtig wichtig nicht wichtig wichtig nicht wichtig wichtig nicht wichtig wichtig nicht wichtig Questionnaires 421 English Translation Part A Personal data Gender: Age: Profession / education: Region / city of birth: Current residence: 1.0 2.0 3.0 4.0 5.0 How often do you eat out on an average? How would you search for a restaurant when you are in a foreign place (multiple choices possible)? What is important for you when you decide on a restaurant (multiple choices possible)? 1.1. 1.2 1.3 1.4 1.5 3.1 3.2 3.3 3.4 3.5 Have you ever used an automatic speech-based information system? yes 3.6 3.7 3.8 3.9 3.10 Other: no 4.1. If yes, on which occasion? 4.1.1 How would you characterize your experience with it? Do you have experience with a speech understanding system? yes no female male years times a week times a month times a year 1.6 1.7 1.8 Other: Magazines Commercial flyers City guide Telephone book Internet Tips from friends Calling a automatic speech-based system Price Food type Food quality Variety of food offered Location Ambience Opening hours Service speed Service friendliness extremely bad bad poor fair good excellent ideal 422 5.1 . If yes, what kind of system? 6.0 7.0 8.0 Do you have experience with synthesized speech? 6.1. If yes, on which occasion? What information about a restaurant do you want to get from an information system? If you would call a restaurant information system, how important is it 8.1 to have a normal conversation just like with a human? 8.2 to be served by a friendly voice? 8.3 that you can ask questions to the system? 8.4 that the system asks questions to you about your preferences? 8.5 to get the desired information quickly? 8.6 that the system can be used easily? 8.7 to get help from the system? yes no important unimportant important unimportant important unimportant important unimportant important unimportant important unimportant important unimportant Questionnaires 423 Par t B Overall impression: Informatio n obtained from the system 1. The system provided the desired information: 2. The provided information was 3. The information was 4. you would rate the information as Communication with the system 5. How well did you feel understood by the system? 6. you had to concentrate in order to understand what the system expected from you: 7. How well was the system acoustically intelligible? System behavior 8. You knew at each point of the dialogue what the system expected from you: 9. In your opinion, the system processed your specifications correctly: 10. The system’s behavior was always as you expected: extremely bad bad poor Mr good excellent ideal yes no complete incomplete clear unclear wrong true extremely bad bad poor fair good excellent ideal yes no extremely bad bad poor fair good excellent ideal yes no yes no yes no 424 11. How often did the system make mistakes? 12. The system reacted in the same way as humans do: 13. The system reacted 14. You were able to control the dialogue in the desired way: 15. The system reacted 16. The system reacted in a way: Dialogue 17. The system utterances were 18. You perceived the dialogue as 19. The course of the dialogue was 20. The dialogue was 21. The course of the dialogue was Your impression of the system 22. The system’s voice was 23 . Overall, you ar e satisfied with the dialogue: frequently rarely yes no flexibly inflexibly yes no too fast adequatly too slowly friendly unfriendly short long natural unnatural clear confusing too short adequate too long smooth bumpy natural unnatural yes no Questionnaires 425 Personal impression 24. You perceived the dialogu e as 25. During the dialogue, you felt pleasant unpleasant relaxed stressed 426 Part C 1. Overall impression: 2. The system’s way of expression was 3. The system reacted 4. You would have expected more help from the system: 5. The system was able to answer all of your questions: 6. Misunderstandings could be cleared easily: 7. The system controlled the flow of the dialogue: 8. You were able to handle the system without any problems: 9. Regarding the dialogues, you are 10. You enjoyed the dialogues: 11. You feel adequately informed about the system’s possibilities: 12 . The telephone calls with the system were worthwhile: extremely bad bad poor fair good excellent ideal clear unclear politely impolitely yes no yes no yes yes no no yes no impressed disappointed yes no yes no yes no Questionnaires 427 13. You perceived this possibility for obtaining information as 14. You rate the system as 15. You prefer to use another source of information: 16. The handling of the system was 17. You prefer a human operator: 18. In the future, you would use the system again? 19. Which characteristics of the system did you like best? 20. Which characteristics of the system disturbed you mostly? 21. Do you have any proposals for system improvements? helpful not helpful reliable unreliable yes no easy complicated yes no yes no [...]... B., Offersgaard, L., and Povlsen, C (1995) The Danish Spoken Language Dialogue Project: A General Overview In: Proc ESCA Workshop on Spoken Dialogue Systems, P Dalsgaard, L.B Larsen, L Boves, and I Thomsen, eds., 89–92, DK–Vigsø Baekgaard, A (1995) A Platform for Spoken Dialogue Systems In: Proc ESCA Workshop on Spoken Dialogue Systems, P Dalsgaard, L.B Larsen, L Boves, and I Thomsen, eds., 105 108 ,... for Spoken Language Dialogue Systems Development In: Proc ESCA Workshop on Spoken Dialogue Systems, P Dalsgaard, L.B Larsen, L Boves, and I Thomsen, eds., 93–96, DK–Vigsø Dybkjær, L., Bernsen, N O., and Dybkjær, H (1996) Evaluation of Spoken Dialogue Systems In: Dialogue Management in Natural Language Systems Proc 11th Twente Workshop on Language Technology (TWLT 11), University of Twente, S LuperFoy,... USA–Boston MA 438 Fraser, N (1997) Assessment of Interactive Systems In: Handbook on Standards and Resources for Spoken Language Systems, D Gibbon, R Moore, and R Winski, eds., 564–615, Mouton de Gruyter, D–Berlin Fraser, N M (1995) Quality Standards for Spoken Language Dialogue Systems: A Report on Progress in EAGLES In: Proc ESCA Workshop on Spoken Dialogue Systems, P Dalsgaard, L.B Larsen, L Boves,... pages, NL–Enschede Dybkjær, L., Bernsen, N O., and Minker, W (2004) Evaluation and Usability of Multimodal Spoken Language Dialogue Systems Speech Communication, 43:33–54 Dybkjær, L and Dybkjær, H (1993) Wizard of Oz Experiments in the Development of the Dialogue Module for P1 Report 3a, Spoken Language Dialogue Systems Program, Centre for Cognitive Informatics, Roskilde University, DK-Roskilde Ehrette,... A Figure of Merit for the Analysis of Spoken Dialogue Systems In: Proc 7th Int Conf on Spoken Language Processing (ICSLP-2002), 2:877– 880, USA–Denver CO Hansen, M (1998) Assessment and Prediction of Speech Transmission Quality with an Auditory Processing Model Doctoral dissertation, Carl-von-Ossietzky-Universität, D-Oldenburg Hansen, M and Kollmeier, B (2000) Objective Modeling of Speech Quality with... (1998) Robust Speech Recognition Spoken Dialogues with Computers, 405–460, Academic Press, UK–London Möller, S (2000) Assessment and Prediction of Speech Quality in Telecommunications Kluwer Academic Publ., USA-Boston MA 450 Möller, S (2002a) A New Taxonomy for the Quality of Telephone Services Based on Spoken Dialogue Systems In: Proc Third SIGdial Workshop on Discourse and Dialogue, 142–153, USA–Philadelphia... and I Thomsen, eds., 157–160, DK–Vigsø Fraser, N M and Dalsgaard, P (1996) Spoken Dialogue Systems: A European Perspective In: Proc 1996 Int Symp on Spoken Dialogue (ISSD 96), H Fujisaki, ed., 25–36, USA– Philadelphia Fraser, N M and Gilbert, G N (1991a) Effects of System Voice Quality on User Utterances in Speech Dialogue Systems In: Proc 2nd Europ Conf on Speech Communication and Technology (EUROSPEECH’91),... 35–42, UK–Sheffield 432 Bernsen, N O and Dybkjær, L (1999) A Theory of Speech in Multimodal Systems In: Proc ESCA Workshop on Interactive Dialogue in Multi-Modal Systems, P Dalsgaard, C.–H Lee, P Heisterkamp, and R Cole, eds., 105 108 , D–Kloster Irsee Bernsen, N O and Dybkjær, L (2000) A Methodology for Evaluating Spoken Language Dialogue Systems and Their Components In: Proc 2nd Int Conf on Language Resources... for Spoken Language Systems In: Proc DARPA Speech and Natural Language Workshop, 102 108 , USA–Hidden Valley PA References 431 Baum, L F (1900) The Wonderful Wizard of Oz Kansas Centennial Ed (1999), University Press of Kansas, USA-Lawrence KS Beerends, J G., Hekstra, A P., Rix, A W., and Hollier, M P (2002) Perceptual Evaluation of Speech Quality (PESQ) –The New ITU Standard for End-to-End Speech Quality. .. Ahrenberg, L (1993) Wizard of Oz Studies – Why and How Knowledge-Based Systems, 6(4):258–266 Dalsgaard, P and Baekgaard, A (1994) Spoken Language Dialogue Systems In: Progress and Prospects of Speech Research and Technology, H Niemann, R de Mori, and G Hanrieder, eds., 178–191, Infix, D–Sankt Augustin Danieli, M and Gerbino, E (1995) Metrics for Evaluating Dialogue Strategies in a Spoken Language System . Platform for Spoken Dialogue Systems. In: Proc. ESCA Workshop on Spoken Dialogue Systems, P. Dalsgaard, L.B. Larsen, L. Boves, and I. Thomsen, eds., 105 108 , DK–Vigsø. Baekgaard, A. (1996). Dialogue. way: Dialogue 17. The system utterances were 18. You perceived the dialogue as 19. The course of the dialogue was 20. The dialogue was 21. The course of the dialogue was Your impression of. on Spoken Dialogue Systems, P. Dalsgaard, L.B. Larsen, L. Boves, and I. Thomsen, eds., 93–96, DK–Vigsø. Dybkjær, L., Bernsen, N. O., and Dybkjær, H. (1996). Evaluation of Spoken Dialogue Systems. In: