MINISTRY OF EDUCATION AND TRAINING VINH UNIVERSITY Phan chÝ nghÜa DESIGNING AN ENGLISH ACHIEVEMENT TEST FOR THE ENGLISH MAJORS AT PHAN BOI CHAU SPECIALISING HIGH SCHOOL IN NGHE AN (ThiÕt kế kiểm tra học kỳ cho học sinh chuyên Anh trờng THPT chuyên Phan Bội Châu, Tỉnh Nghệ An) Master Thesis in Education Vinh, 2011 MINISTRY OF EDUCATION AND TRAINING VINH UNIVERSITY Phan chÝ nghÜa DESIGNING AN ENGLISH ACHIEVEMENT TEST TABLE OF CONTENTS CHAPTER 1: INTRODUCTION Rationale of the Study Aims of the study Scope of the study Methods of study Organization of the study CHAPTER 2: LITERATURE REVIEW 11 2.1 Basic concepts of testing 11 2.2 Roles of testing 12 2.2.1 The relationship between testing and teaching 12 2.2.2 The relationship between testing and learners 13 2.3 Types of tests and testing 14 2.3.1 Proficiency Tests 14 2.3.2 Achievement Tests 15 2.3.3 Diagnostic Tests 16 2.3.4 Placement tests 17 2.3.5 Direct versus Indirect Tests 17 2.3.6 Discrete point verse Integrative Testing 19 2.3.7 Norm-Referenced versus Criterion-Referenced Testing 19 2.3.8 Objective Testing versus Subjective Testing 20 2.3.9 Communicative Language Testing 20 2.4 Characteristics of a good test 21 2.4.1 Validity 21 2.4.1.1 Construct validity 21 2.4.1.2 Content validity 22 2.4.1.3 Face validity 22 2.4.1.4 Backwash validity 23 2.4.1.5 Criterion-related validity 24 2.4.2 Reliability 24 2.4.3 Discrimination 25 2.4.4 Practicability 25 2.5 Test items for reading skill, writing skill, grammar, and vocabulary 26 2.5.1 Test items 26 2.5.2 Language components and language skills 27 2.5.3 The test item types used to evaluate languages components and 28 language skills CHAPTER 3: METHODOLOGY 30 3.1 Setting of the study 30 3.1.1 The subjects and the current English teaching, learning and 30 testing situations at Phan Boi Chau Speacialising High School 3.1.1.1 The situation of teaching English to English major pupils at Phan 30 Boi Chau Specialising High School 3.1.1.1.1 The community 3.1.1.1.2 The school 3.1.1.1.3 The course 3.1.1.1.4 The teachers 3.1.1.1.5 The pupils 3.1.1.1.6 The teaching materials 3.1.1.1.6.1 The general description of the teaching materials used for 30 32 34 35 35 37 37 English major pupils at Phan Boi Chau Specialising High School 38 3.1.1.6.2 The good points of of teaching materials when used for English major pupils at Phan Boi Chau Specialising High School 3.1.1.6.3 The drawbacks of of teaching materials when used for English 39 major pupils at Phan Boi Chau Specialising High School 3.1.2 The current testing situation at Phan Boi Chau Specialising High 42 School 3.2 Methods of the study 44 3.3 Research design 45 3.3.1 Sample and sampling 45 3.3.2 Instruments 45 3.3.3 Data collection 48 3.3.4 Data analysis 49 CHAPTER 4: FINDINGS AND DISCUSSIONS 50 4.1 Data collection and interpreting test scores and items 50 4.1.1 Interpretation and test score analysis 50 4.1.1.1 The frequency distribution 50 4.1.1.2 The central tendency 54 4.1.1.3 The dispersion 55 4.1.2 Test items evaluation 57 4.1.2.1 The item difficulty 57 4.1.2.2 The item discrimination 60 4.1.3 Estimating reliability 63 4.2 Evaluation of validity 64 4.2.1 Face validity 64 4.2.2 Suggestions by teachers to improve the tests’ face validity 65 CHAPTER 5: CONCLUSION 68 5.1 Summary of the study 68 5.2 Suggestions on improvements of designing the final achievment test 69 for English major pupils at Phan Boi Chau Specialising High School 69 5.2.1 Reliability 69 5.2.1.1 Test length 70 5.2.1.2 Item qualities 71 4.2.2 Face validity 71 5.2.3 Some other suggestions 74 5.3 Limitations 74 5.4 Suggestions for further study 76 BIBLIOGRAPHY APPENDIX APPENDIX APPENDIX DECLARATION I certify that the thesis entitled “Designing an English achievement test for the English majors at Phan Boi Chau Specialising High School in Nghe An ” is submitted in partial fulfillment of the requirements for the degree of Master of Arts is the result of my own work, except where otherwise acknowledge and that this thesis or any part of the same had not been submitted for a higher degree to any other universities or institution Vinh, November 2011 Phan Chi Nghia ACKNOWLEDGEMENTS To carry out this MA coursework I am indebted to many people for their encouragement, cooperation, and advice First and foremost, I would like to express my deepest gratitude to Assoc Prof & Ph.d Ngo Dinh Phuong, my supervisor for his useful advice, insightful ideas, and dutiful supervision I also would like to take this opportunity to express my thanks to all the colleagues in the English section in Phan Boi Chau Specialising High School for their help in answering questions in surveys, direct interviews, for their constructive suggestions about this research I would like to give my special thanks to the English major pupils at Phan Boi Chau Specialising High School who have actively participated in doing the sample test, the surveys and responding to my interviews Last but not least, my sincere thanks go to my family, my classmates, my friends, especially my wife who encouraged me to carry out the thesis ABSTRACT Testing plays a very important role in teaching process which helps teachers to assess their teaching as well as their students’ learning Evaluating a test in terms of its qualities such as reliability and validity is very necessary as to ensure the usefulness of this assessment instrument However, this issue receives little considerations from teachers at Phan Boi Chau Specialising High School Thus, this study was designed to evaluate two qualities, reliability and validity of the sample achievement tests for the English major pupils at Phan Boi Chau Specialising High School The MA thesis entitled “Designing an English achievement test for the English majors at Phan Boi Chau Specialising High School in Nghe An” deals with the problems of the current testing situation in the school On doing the study the author attempts to study the possible reasons for the low reliability and validity,… of the achievement test for the English major pupils and propose some ways to improve them The study is based on both the analysis of the English course including the subjects, the syllabus and the sample test and the analysis and interpretation of the test scores and test items of the suggested test As a result, major findings were drawn Firstly, the reliability of the test was low Secondly, the test had incompletely face validity On the basis of these findings and valuable ideas from teachers of English section, some suggestions for improvements were proposed to better the test’s quality It is hoped that this research study will help teachers at Phan Boi Chau Specialising High School have an insight into their test’s quality from which an awareness of the test design will be raised and better and more accurate assessment devices will be produced LIST OF TABLES AND FIGURES TABLE Table 2.5.3.1: Test item types for Reading and Writing skills and Grammar Vocabulary Table 3.3.2.1: The Paper Specification Grids for The nd Term Achievement Test for Grades 10, 11, 12 of English Major Pupils at Phan Boi Chau Specialising High School Table 4.1.1.1.1: Frequency distribution in the final achievement test for English major pupils of grade 10 Table 4.1.1.1.1: Frequency distribution in the final achievement test for English major pupils of grade 10 Table 4.1.1.1.2: Frequency distribution in the final achievement test for English major pupils of grade 11 Table 4.1.1.1.3: Frequency distribution in the final achievement test for English major pupils of grade 12 Table 4.1.2.1.1: The interpretation of the item difficulty of the final achievement test for English major pupils of grade 10 Table 4.1.2.1.2: The interpretation of the item difficulty of the final achievement test for English major pupils of grade 11 Table 4.1.2.1.3: The interpretation of the item difficulty of the final achievement test for English major pupils of grade 12 Table 4.1.2.2.1: The result of item discrimination of the final achievement test for English major pupils of grade 10 Table 4.1.2.2.2: The result of item discrimination of the final achievement test for English major pupils of grade 11 Table 4.1.2.2.3: The result of item discrimination of the final achievement test for English major pupils of grade 12 FIGURES Figure 4.1.1.1.1: Histogram of score distribution in the final achievement test for English major pupils of grade 10 Figure 4.1.1.1.2: Histogram of score distribution in the final achievement test for English major pupils of grade 11 Figure 4.1.1.1.3: Histogram of score distribution in the final achievement test for English major pupils of grade 12 CHAPTER 1: INTRODUCTION RATIONALE OF THE STUDY The importance of language testing is recognized by virtually all professionals in the field of language education It is of special importance in educational system that is highly competitive It is because testing is not only an indirect stimulus to learning, but also plays a crucial role in determining the success or failure of an individual's career with direct implications for his future earning power “Thus, testing is an important tool in educational research and for programme evaluation, 10 contrary be passed allowed/ permit 10 set Part 2: Read the following passage and choose the correct answer for each of the questions correct answer gets point (total: points) A A C C A D WRITING Rewrite the following sentences, using the word given in capital Do NOT change this word in any way correct answer gets point (total: points) She was lost for words./ she had lost her tongue You’ll have to turn over a new leaf if … I’m a sociable rather than a ……… person./ Rather than being a ………, I’m sociable He was unable to cope with the … because of … The news about the …… spread through the …… like wildfire TOTAL MARK: 50 points CONVERTED MARK: 50x2/10 = 10 (The mark will be then converted to round nunmber, ie ending in and ) - Answer key and marking scale for the sample of The nd Term Achievement Test for grade 11 major English pupils A LISTENING You will hear an interview with someone who is involved in the music business For questions 1-5, choose the best answer (A, B or C) correct answer gets point (total: points) B A A C A B LEXICO-GRAMMAR Part 1: Choose the word or phrase which best completes each sentence 201 correct answer gets point (total: 10 points) C D A C C B D C A 10 C Part 2: Supply correct tense or form of the verbs correct answer gets point (total: points) had discovered never existed/had never existed had hidden was looking forward were resting Part 3: Use the correct form of the word given to fill in each blank correct answer gets point (total: points) revelation unprecedented understaffed overdose trisyllabic Part 4: Use the the correct forms of the words in the box to fill in the blanks to make a meaningful passage correct answer gets point (total: points) circulation headlines reviews censorship editorials C READING Part 1: Fill each of the blank spaces with one suitable word to finish the passage correct answer gets point (total: 10 points) missing but baby world found through how holding vibrations 10 own Part 2: Read the following passage and choose the correct answer for each of the questions correct answer gets point (total: points) C 2.B D C A D WRITING Rewrite the following sentences, using the word given in capital Do NOT change this word in any way 202 correct answer gets point (total: points) They had to take the vehicles to pieces to get is getting me down can be put down to has put our city on the map Trade has gone from bad to worse and staff are TOTAL MARK: 50 points CONVERTED MARK: 50x2/10 = 10 (The mark will be then converted to round nunmber, ie ending in and ) - Answer key and marking scale for the sample of The nd Term Achievement Test for grade 12 major English pupils A LISTENING You will hear an interview with someone who is involved in the music business For questions 1-5, choose the best answer (A, B or C) correct answer gets point (total: points) B A A C A B LEXICO-GRAMMAR Part 1: Choose the word or phrase which best completes each sentence correct answer gets point (total: 10 points) D B B D C D B A C 10 D Part 2: Supply correct tense or form of the verbs correct answer gets point (total: points) has conquered has decreased will be issued is being studied have been developed Part 3: Use the correct form of the word given to fill in each blank 203 correct answer gets point (total: points) better-off insensitive mal-nourished/ under-nourished enliven inexplicable Part 4: Insert A, AN, THE or Φ (zero article) where necessary Write your answer in the numbered box correct answer gets points (total: 10 points) Q# Key Ø the C READING the a the Part 1: Fill each of the blank spaces with one suitable word to finish the passage correct answer gets point (total: points) self human function reduction pain shed serve manly intense 10 harmful Part 2: Read the following passage and choose the correct answer for each of the questions correct answer gets point (total: points) C B B D B D WRITING Rewrite the following sentences, using the word given in capital Do NOT change this word in any way correct answer gets point (total: points) David was instrumental in the drafting of/in drafting the proposal Had you stuck to/by what we originally agreed, everything would have been fine Or: If you had stuck to … I think you should spare a thought for those/(the) people whose lives aren’t as privileged as yours Or: … those who don’t have lives at privilege as yours We decided to extend our stay as we were so thrilled by the place 204 Her speech had/made/created/exerted/achieved a profound/great/huge/powerful/ tremendous impact on me TOTAL MARK: 50 points CONVERTED MARK: 50x2/10 = 10 (The mark will be then converted to round nunmber, ie ending in and ) APPENDIX A: THE SURVEY QUESTIONNAIRE (For the English major pupils at Phan Boi Chau Specialising High School) This questionnaire asks for your comments and ideas about the final written achievement test for the English major pupils at Phan Boi Chau Specialising High School The purpose of this survey questionnaire is to find out how you assess the present final achievement tests based on the criteria of a good test Your responses are important as they will enable improvement on test design All information you provide will be kept confidential Thank you very much for your valuable contribution I General questions Please tick (√) the column you think the most suitable Questions Strongly Disagree Don’t disagree know 205 Agree Strongly agree The test measures what the students have been taught The task types given in the text are familiar to the students Time allowance for this test is enough The weighting demonstrated on the marking scale of the test is appropriate In terms of test item format, this English test mainly intends to measure the students’ grammar and vocabulary knowledge The Grammar and Vocabulary part is long enough and related to what students have been taught A student who is given a high score in the grammar and vocabulary section of the final achievement test is the one who has good grammar and lexical knowledge The topics of the reading passages are familiar to students The length of the reading passages is long enough 206 The reading passages are of the right level in terms of linguistic difficulty The reading section of the final achievement test tells us accurately about the students’ reading ability The grammar structures and expressions given in writing tasks are familiar to students Sentence building and transformation measure students’ grammar and vocabulary knowledge A student who is given a high score in the writing section of the final achievement test is the one who can write English as well as the course requires II Questions for improvements of the final achievement tests In your opinion, should we change the construction of the grammar and vocabulary part of the current achievement test in terms of format? a Yes b No c If “yes”, please specify how ……………………………………………………… 207 ……………………………………………………………………………………… In your opinion, should we change the construction of the reading part of the current achievement test in terms of format? a Yes b No c If “yes”, please specify how ……………………………………………………… ……………………………………………………………………………………… In your opinion, should we change the construction of the writing part of the current achievement test in terms of format? a Yes b No c If “yes”, please specify how In your opinion, should we change the construction of the current achievement test in terms of the total points for each part? a Yes b No c If “yes”, please specify how ……………………………………………………… ……………………………………………………………………………………… If you choose YES, please tick (√) to indicate the appropriate language skills that should be tested for the English major pupils at Phan Boi Chau Specialising High School 208 a Listening comprehension b Speaking c Reading comprehension d Writing In your opinion, which of the following language elements should be tested for the English major pupils at Phan Boi Chau Specialising High School a phonetics b Grammar usage c Vocabulary d Others (Please specify) ……………………………………………………………………………………… Please give comments or suggestions towards the improvement of the existing final achievement test for the English major pupils at Phan Boi Chau Specialising High School ……………………………………………………………………………………… …… ……………………………………………………………………………………… …… ……………………………………………………………………………………… …… ……………… ………………………… 209 B: THE SURVEY QUESTIONNAIRE (For Teachers of English at Phan Boi Chau Specialising High School) This questionnaire asks for ideas about the final written achievement tests for the English major pupils at Phan Boi Chau Specialising High School The purpose of this survey questionnaire is to find out how you feel about the present final achievement tests based on the criteria of a good test Your responses are important as they will enable improvement on test design All information you provide will be kept confidential Thank you very much for your valuable contribution I General questions Please tick (√) the column you think the most suitable Questions Strongly Disagree Don’t disagree know The test measures what the students have been taught The task types given in the text are familiar to the students Time allowance for this test is enough The weighting demonstrated on the marking scale of the test is appropriate 210 Agree Strongly agree In terms of test item format, this English test mainly intends to measure the students’ grammar and vocabulary knowledge The Grammar and Vocabulary part is long enough and related to what students have been taught A student who is given a high score in the grammar and vocabulary section of the final achievement test is the one who has good grammar and lexical knowledge The topics of the reading passages are familiar to students The length of the reading passages is long enough The reading passages are of the right level in terms of linguistic difficulty The reading section of the final achievement test tells us accurately about the students’ reading ability The grammar structures and expressions given in writing tasks are familiar to students 211 Sentence building transformation students’ and measure grammar and vocabulary knowledge A student who is given a high score in the writing section of the final achievement test is the one who can write English as well as the course requires II Questions for improvements of the final achievement tests In your opinion, should we change the construction of the grammar and vocabulary part of the current achievement test in terms of format? a Yes b No c If “yes”, please specify how ……………………………………………………… ……………………………………………………………………………………… In your opinion, should we change the construction of the reading part of the current achievement test in terms of format? a Yes b No c If “yes”, please specify how ……………………………………………………… ……………………………………………………………………………………… 212 In your opinion, should we change the construction of the writing part of the current achievement test in terms of format? a Yes b No c If “yes”, please specify how ……………………………………………………… ……………………………………………………………………………………… In your opinion, should we change the construction of the current achievement test in terms of the total points for each part? a Yes b No c If “yes”, please specify how If you choose YES, please tick (√) to indicate the appropriate language skills that should be tested for the English major pupils at Phan Boi Chau Specialising High School a Listening comprehension b Speaking c Reading comprehension d Writing In your opinion, which of the following language elements should be tested for the English major pupils at Phan Boi Chau Specialising High School a phonetics b Grammar usage 213 c Vocabulary d Others (Please specify) ……………………………………………………………………………………… Please give comments or suggestions towards the improvement of the existing final achievement test for the the English major pupils at Phan Boi Chau Specialising High School ……………………………………………………………………………………… …… ……………………………………………………………………………………… …… ……………………………………………………………………………………… …… ……………… ………………………… C: INTERVIEW QUESTIONS I For teachers 214 How long have you been teaching English? Have you ever designed an achievement test of your own? Have you been trained in test writing? How long you spend writing a test paper? What, in your opinion, is the purpose of an achievement test? What content you think should be included in the achievement test? What you think about the marking scales for each part of the current achievement test? What about the time allowance? In general, how appropriate is the achievement test we use to our students? 10 In general, how appropriate is the achievement test we use to the objective of our course? 11 What improvements you think are needed for the achievement test we use in our school? II For students Do you satisfy with the mark you get for the current achievement test? Is the test closely related to what you have been taught? Does the test reflect your real ability? Which part of the test you find the most difficult? How fair you think the test is? What you think about the time allowance? 215 ... pupils at Phan Boi Chau Specialising High School The MA thesis entitled ? ?Designing an English achievement test for the English majors at Phan Boi Chau Specialising High School in Nghe An? ?? deals... of teaching materials 3.1.2 The current testing situation at Phan Boi Chau Specialising High School Based on the experience in teaching English at Phan Boi Chau Specialising High School for nearly... for improvement of testing at Phan Boi Chau Specialising High School SCOPE OF THE STUDY 13 The scope of the study focuses on the existing situation at Phan Boi Chau Specialising High School in