Đang tải... (xem toàn văn)
A STUDY ON THE RELIABILITY OF THE FINAL ACHIEVEMENT COMPUTER-BASED MCQS TEST 1 FOR THE 4TH SEMESTER NON - ENGLISH MAJORS AT HANOI UNIVERSITY OF BUSINESS AND TECHNOLOGY
VIETNAM NATIONAL UNIVERSITY, HANOI COLLEGE OF FOREIGN LANGUAGES DEPARTMENT OF POSTGRADUATE STUDIES NGUYEN THI VIET HA A STUDY ON THE RELIABILITY OF THE FINAL ACHIEVEMENT COMPUTER-BASED MCQS TEST 1 FOR THE 4 TH SEMESTER NON - ENGLISH MAJORS AT HANOI UNIVERSITY OF BUSINESS AND TECHNOLOGY (đánh giá độ tin cậy của bài thi trắc nghiệm THứ NHấT TRÊN MáY TíNH cuối kỳ 4 dành cho sinh viên năm thứ hai không chuyên ngành tiếng anh trờng đại học kinh doanh và công nghệ hà nội) Minor Programme Thesis Field: Methodology Code: 601410 HANOI, 2008 1 VIETNAM NATIONAL UNIVERSITY, HANOI COLLEGE OF FOREIGN LANGUAGES DEPARTMENT OF POSTGRADUATE STUDIES NGUYễN THị VIệT Hà A STUDY ON THE RELIABILITY OF THE FINAL ACHIEVEMENT COMPUTER-BASED MCQS TEST 1 FOR THE 4 TH SEMESTER NON - ENGLISH MAJORS AT HANOI UNIVERSITY OF BUSINESS AND TECHNOLOGY (đánh giá độ tin cậy của bài thi trắc nghiệm THứ NHấT TrÊN MáY TíNH cuối kỳ 4 dành cho sinh viên năm thứ hai không chuyên ngành tiếng anh trờng đại học kinh doanh và công nghệ hà nội) Minor Programme Thesis Field: Methodology Code: 601410 Supervisor: Nguyễn Thu Hiền. M.A HANOI, 2008 2 VIETNAM NATIONAL UNIVERSITY, HANOI COLLEGE OF FOREIGN LANGUAGES DEPARTMENT OF POSTGRADUATE STUDIES CANDIDATE S STATEMENT’ I hereby state that I: Nguyen Thi Viet Ha, Class 14A, being a candidate for the degree of Master of Arts (TEFL) accept the requirements of the College relating to the retention and use of Master of Arts Thesis deposited in the library. In terms of these conditions, I agree that the origin of my thesis deposited in the library should be accessible for the purposes of study and research, in accordance with the normal conditions established by the librarian for the care, loan or reproduction of the thesis. Signature Date i ACKNOWLEDGMENTS In the completion of this thesis, I have received a great deal of backup. Of primary importance has been the role of my supervisor, Ms. Nguyen Thu Hien, M.A, Teacher of Department of English and American Languages & Cultures, College of Foreign Language, Vietnam National University, Hanoi. I am deeply grateful to her for her precious guidance, enthusiastic encouragement and invaluable critical feedback. Without her dedicated support and correction, this thesis could not have been completed. I am deeply indebted to my dear teacher, Mr. Vu Van Phuc, M.A, Head of Testing Center, College of Foreign Languages, VNU, who provided me with a lot of useful suggestion and assistance towards my study. I would also like to express my sincere thanks to all teachers and colleagues in English Department, HUBT, for their help in conducting the survey, sharing opinions and making suggestions to the study. Especially, my thanks go to Ms. Le Thi Kieu Oanh, Assistant of English Department, HUBT for her willingness to offer test score data. I wish to show my special thanks to the students of K11 at Hanoi University of Business and Technology who have actively participated in the survey Finally, it is my great pleasure to acknowledge my gratitude to beloved members of my family, especially my husband who constantly encouraged and helped me with my thesis. ii ABSTRACT The main aim of this minor thesis is to evaluate the reliability of the final Achievement Computer-based MCQs Test 1 for the 4th semester non-English majors at Hanoi University of Business and Technology. In order to achieve this aim, a combination of both qualitative and quantitative research methods were adopted. The findings indicate that there is a certain degree of unreliability in the final achievement computer-based MCQs test1 and there are two main factors that cause the unreliability including test item quality and test- takers performance. ’ Having carefully considered a thorough analysis of the collected data, the author made some suggestions in order to improve the quality of the final achievement test and the MCQs test 1 for the non-majors of English in the 4 th semester in Hanoi University of Business and Technology. Firstly, the test objectives, sections and skill weight should be adjusted to be more compatible with the course objectives and the syllabus. Secondly, a testing committee should be set up for the construction and development of a multi choice item bank including test items which are of good p-value and discrimination value. iii LIST OF ABBRIVIATIONS 1. CBT: Computer-based testing 2. HUBT: Hanoi University of Business and Technology 3. MC: Multi choice 4. MCQs: Multi choice questions 5. ML Pre- : Market Leader Pre-intermediate 6. KD: Kuder- Richardson 7. SD: Standard deviation iv LIST OF TABLES AND CHARTS 1. Table 1 Types of tests 2. Table 2 Scoring format for each semester 3. Table 3 The syllabus for 4 th semester (for non English majors– ) 4. Table 4 Time allocation for language skills and sections 5. Table 5 Specification grid for the final computer-based MCQs test 1 6. Table 6 Main points in the grammar section 7. Table 7 Main points in the vocabulary section 8. Table 8 Topics in reading section 9. Table 9 Items in the functional language sections 10. Table 10: Test reliability coefficient 10. Table 11: p-value of items in 4 sections 11. Table 12: Discrimination value of items in 4 sections 12. Table 13: Number of test items with acceptable p-value and discrimination value in 4 sections 13. Table 14: Suggested scoring format 14. Table 15: Proposed test specifications 12. Chart 1 Students response on test content’ 13. Chart 2 Students response on item discrimination value’ 14. Chart 3 Students response on time length ’ 15. Chart 4 Students response arbitrariness ’ 16. Chart 5 Students response on relation between test score and their ’ achievement v TABLE OF CONTENT CANDIDATE S STATEMENT’ i ACKNOWLEDGEMENT ii ABSTRACT iii LIST OF ABBREVIATION iv LIST OF TABLES AND CHARTS v TABLE OF CONTENT vi Chapter 1: INTRODUCTION 1 1.1. Rationale for the study 1 1.2. Aims and research questions 2 1.3. Theoretical and practical significance of the study 2 1.4. Scope of the study 2 1.5. Method of the study 2 1.6. Organization of the paper 3 Chapter 2: LITERATURE REVIEW 4 2.1. Language testing 4 2.1.1. What is a language test? 4 2.1.2. The purposes of language tests 4 2.1.3. Types of language tests 5 2.1.4. Criteria of a good language test 5 2.2. Achievement test 6 2.2.1. Definition 6 2.2.2. Types of achievement test 6 2.2.3. Considerations in final achievement test construction 7 2.3. MCQs test 7 2.3.1. Definition 7 2.3.2. Benefits of MCQs test 8 2.3.3. Limitations of MCQs test 10 2.3.4. Principles on designing a good MCQs test 11 2.4. Reliability of a test 11 2.4.1. Definition 11 2.4.2. Methods for test reliability estimate 12 2.4.3. Measures to improve test reliability 15 2.5. Summary 15 Chapter 3: The Context of the Study 16 3.1. The current English learning, teaching and testing situation at HUBT 16 3.2. The course objectives, syllabus and materials used for the second non- majors of English in Semester 4. 17 3.2.1. The course objectives 17 3.2.2. Business English syllabus 17 3.2.3. The course book 19 3.2.4. Specification grid for the final achievement Computer-based MCQs test in Semester 4. 19 Chapter 4: Methodology 21 vi 4.1. Participants 21 4.2. Data collection instruments 21 4.3. Data collection procedure 21 4.4. Data analysis procedure 22 Chapter 5: RESULTS AND DISCUSSIONS 23 5.1. The compatibility of the objectives, content and skill weight format of the final achievement computer-based MCQ test 1 for 4 th semester with the course objectives and the syllabus 23 5.1.1 The test objectives and the course objectives 23 5.1.2. The test item content in four sections and the syllabus content 24 5.1.3. The skill weight format in the test and the syllabus 26 5.2. The reliability of the final achievement test 27 5.2.1. Reliability coefficient 27 5.2.2. Item difficulty and discrimination value 27 5.3. The attitude of students towards the MCQs test 1 29 5.4. Pedagogical implications and suggestions on improvements of the existing final achievement computer-based MCQs test 1 for the non- English majors at HUBT. 34 5.5. Summary 38 Chapter 6: CONCLUSION 39 6.1. Summary of the findings 39 6.2. Limitations of the study 40 6.3. Suggestions for further study 40 REFERENCES 41 APPENDICES I APPENDIX 1 Grammar, Reading, Vocabulary and Functional language check list II APPENDIX 2 Survey questionnaire (for students at HUBT) IV APPENDIX 3 Students test scores’ VII APPENDIX 4 Item analysis of the final achievement computer-based MCQs test 1- 150 items, 349 examinees XII APPENDIX 5 Item indices of the final achievement computer-based MCQs test 1 XVII vii Chapter 1: Introduction 1.1. Rationale of the study Testing plays a very important role in teaching and learning process. Testing is one form of measurement which is used to point out strengths and weaknesses in the learned abilities of the students. Through testing, especially tests scores we may discover the performance of given students and of teachers. As far as students are concerned, test scores reveal what they have achieved after a learning period. As for teachers, test scores indicate what they have taught to their students. Based on test results, we may make improvement in teaching, learning and testing for better instructional effectiveness. Another reason for the selection of testing a matter of study lies in the fact that the current language testing at Hanoi University of Business and Technology (HUBT) has been under a lot of controversy among students and teachers. Testing is mainly carried out in the form of two objective tests on computers (named test 1 and test 2) which are administered at the end of each semester. The scores that a student gets on these tests are the main indicators of his or her performance during the whole semester. There are different comments on the results of these tests, especially the test 1 for the second-year non-English majors. Some subject teachers claim that these tests do not truly reflect the students’ language competence. Others say that these tests are appropriate to what students have learnt in class and compatible with the course objectives and therefore reliable. Also, among the students, do opposite ideas exist. Many think that these tests are more difficult than what they have learnt and studied for the exam, others say that these test items are easy and relevant to what they have been taught. Therefore finding out whether the tests are closely related with what the students have been learnt and what the teachers have taught, also, whether these tests are of reliability is indispensable. For the two reasons mentioned above, the author would like to undertake this study entitled A study on the reliability of the final achievement Computer-based MCQs Test“ 1 for the 4th semester non-English majors at Hanoi University of Business and Technology” with the intention to examine rumors about this test. In addition, the author hopes that the study results help to raise awareness among teachers as well as those who are interested in this field. At the same time, study results, in some extent, can be applied to improve the current testing situation in HUBT. 1.2. Aims and research questions 1 [...]... objectives, content and skill weight format of the final achievement computer- based MCQs test 1 compatible with the course objectives, the syllabus content and skill weight format ? 2 To what extend is the test 1 reliable? 3 What is the student’s attitude towards the final achievement Computer- based MCQs test 1? 1.3 Scope of the study The existing final achievement Computer- based MCQs test 1 in the 4 th semester... especially in evaluating students’ acquired language knowledge and skills during a given course 2.2.2 Types of achievement test Achievement tests can be subdivided into the final achievement and progress achievement according to the time of administration and the desired objectives (Henton, 1990) Final achievement tests are usually given at the end of the school year or at the end of the course to measure... examining the task and content in the course book used for construction of the achievement computer- based MCQs test 1 is in Appendix 1 18 3.2.4 Specification grid and scoring scale for the final achievement Computer- based MCQs test 1 in Semester 4 In order to evaluate students’ achievement, the following grid is used to design achievement test 1 Part Main skill Item type Number of Skill Incomplete 50 x;... order to determine the extent to which the final achievement test 1 is reliable Finally, analysis of students’ responses on the questionnaire was made in order to find out students’ attitude towards the MCQs test given to them 22 Chapter 5: Results and Discussions 5.1 The compatibility of the objectives, content and skill weight format of the final achievement computer- based MCQ test 1 for 4th semester... (6 credits) The second score (6 credits ) Daily assessment 1 20% Daily assessment 2 20% Paper test 1 10% Paper test 2 10% Computer- based 70% Computer- based 70% MCQs test 1 MCQs test 2 Table 2: Scoring format for each semester This study is only focused on the final achievement computer- based MCQs 1 3.2 The course objectives, syllabus and materials used for the second non-majors of English in Semester... The main different is that the test is delivered on computers and students simply click mouse for their chosen response among A, B, C, D This kind of test is different from computer adaptive tests which are tailored to the particular abilities of the candidate In 16 other words, a Computer- based MCQs test at HUBT is in fact an MCQs test delivered on computers The following chart illustrates testing guideline... different students and to reflect individuals’ performance of the same group 2.2 Achievement test 2.2.1 Definition Achievement tests are of extensive use at different levels of education due to their distinguished characteristics Researchers define the notion of achievement tests in various ways Henning (1987, p.6) held that: Achievement tests are used to measure the extent of learning in a prescribed... period of learning and it should attempt to cover as much of syllabus as possible 2.2.3 Considerations in final achievement test construction On the basis of its characteristics, Heaton (1990) put that covering much the contents of a syllabus or a course book is a requirement for designing a final achievement test Testers should avoid basing the test on their own teaching rather than on the syllabus... language tests and criteria of a good test are also discussed Section 2.2 classifies achievement tests into two types and mentions consideration in designing final achievement tests The definition, benefits and limitations of MCQs tests and principles of this type of test construction are dealt with in section 2.3 The final Section - 2.4 is concerned with test reliability, methods for estimating test... noted that the proponents of computer- based testing can point to a number of advantages First, just as paper-done MCQs tests, scoring of fixed response items can be done automatically and the candidate can be given a score immediately Second, the computer can deliver tests that are tailored to the particular abilities of the candidate This type of test, as also called computer- adaptive test, can provide . analysis of the final achievement computer- based MCQs test 1- 150 items, 349 examinees XII APPENDIX 5 Item indices of the final achievement computer- based. student’s attitude towards the final achievement Computer- based MCQs test 1? 1.3. Scope of the study The existing final achievement Computer- based MCQs test