Joyce Nutta Is Computer-Based Grammar Instruction as Effective as TeacherDirected Grammar Instruction for Teaching L2 Structures? Joyce Nutta University of South Florida ABSTRACT The study described here compared postsecondary English as a Second Language (ESL) students’ acquisition of selected English structures based on the method of instruction—computer-based instruction versus teacherdirected instruction.1 The results showed that for all levels of English proficiency, the computer-based students scored significantly higher on openended tests covering the structures in question than the teacher-directed students No significant differences were found between the computerbased and teacher-directed students’ scores on multiple choice or fill-inthe-blank tests The results indicate that computer-based instruction can be an effective method of teaching L2 grammar KEYWORDS Computer Assisted Language Learning (CALL), Grammar Instruction, Focus on Form, Structural Syllabus, English as a Second Language (ESL), Computer-Based Instruction INTRODUCTION As the use of computers in language teaching increases, it is essential to establish research-based indications of the appropriate roles of Computer Assisted Language Learning (CALL) in curriculum and instruction One of the most important questions focuses on whether technology should attempt to emulate the characteristics of a communicative classroom, engaging students in real and meaningful communication, or provide the types of tutorials and drills that tend to be de-emphasized in current teaching practice Numerous Second Language Acquisition researchers have Volume 16 Number 49 Computer-Based and Teacher-Driven Instruction asserted that the computer should be used to replicate what they believe ought to occur in the classroom Many proponents of CALL have advocated the development of communicative computer programs that provide opportunities for meaningful communication (Garrett, 1991; Underwood, 1993; Quinn, 1990; Lavine, 1992) Although some educators have decried the use of computers as electronic workbooks for drilland-practice exercises (Chun & Brandl, 1992; Underwood, 1993), others have advocated their use for tutorials and drills to free up more classroom time for real communication (Gilby, 1996; Hoffman, 1996) The teaching context often determines the role of CALL In an English as a Second Language (ESL) environment, communicative CALL programs often supplement and augment classroom activities by providing games for practice or word processing applications for compositions In some English as a Foreign Language (EFL) programs, computer programs may complement or completely supplant classroom instruction by providing instruction in a subject or skill not taught in the classroom (Soo & Ngeow, 1996) In the latter case, computers and other technologies are relied upon to provide a model of native speech that the instructors, many of whom are nonnative speakers of English, cannot offer During the past decade, numerous studies have examined different approaches to teaching grammar (Doughty, 1991; Ellis,1985, 1993; Fotos, 1993; Green & Hecht, 1992; Tomasello & Herron, 1988) These studies have shown that many effective means of teaching L2 grammar are available—from teacher-directed Constructivist tutorials (Adair-Hauck & Donato, 1994), to cooperative group work (Fotos, 1994) and individual study with textbooks (Scott & Randall, 1992) Because these studies indicate that various effective ways to learn grammar exist, teachers have an ever increasing array of options with which to meet the needs of students The use of computers to teach grammar has not received the same amount of attention as communicative CALL, but computer-based grammar instruction offers many potential benefits Although it is currently impossible for the computer to engage learners in authentic two-way communication, it is, in fact, possible for CALL to provide rich input in the form of integrated multimedia programs and to provide explicit grammar explanations that can be viewed and reviewed at the learner’s own pace In a metaanalysis of research on the use of multimedia to teach a variety of subjects, Ragan, Boyce, Redwine, Savenye, and McMichael (1993) found that, in general, multimedia instruction reduces learning time by 30% compared to traditional instruction They further demonstrated that features of multimedia instruction such as learner interactivity and learner control over programs produce improved outcomes in achievement Results of studies on computer assisted instruction have been generally positive For over two decades, researchers have examined the effectiveness of computer assisted instruction to teach many different subjects In 50 CALICO Journal Joyce Nutta 1987, Kulik and Kulik published a ground-breaking metaanalysis of comparative studies on computer assisted and classroom instruction in which they calculated an overall effect size of 31 Other studies by Niemiec and Walberg (1987) and McNeil and Nelson (1991) uncovered similar results In spite of the abundance of comparative research on computer assisted instruction in other academic fields such as reading (Rachal, 1995) and the growing body of research on methods of teaching grammar, only recently have researchers investigated the use of computer-based L2 grammar instruction (McEnery, Baker & Wilson, 1995; Nagata, 1996) The results of these studies seem to indicate that computer-based grammar instruction can be as effective or more effective than more traditional instruction (e.g., workbooks and lectures) If subsequent research bears out this initial indication of the benefits of computer-based grammar instruction, this kind of instruction could become an important complement to communicatively oriented language classes By using the computer for the presentation, explanation, and application of grammatical structures, more classroom time could be dedicated to real communication that focuses on expressing meaning and using appropriate grammatical structures to express that meaning Shifting the systematic study of grammar points from the realm of the classroom to the domain of the computer laboratory would enable instructors to take advantage of classroom interpersonal dynamics and allow them to take into account differences in background knowledge and learning styles The use of computer-based grammar instruction would also support more individualized instruction in programs that have open-entry/openexit enrollment such as adult and vocational education courses Students would not enter the class in the middle of a linear instructional sequence based on a grammatical syllabus Instead, the syllabus could be based on themes of interest and relevance to students, and individual students could follow a grammatical syllabus in computer-based instruction outside the classroom PURPOSE OF THIS STUDY This study examined whether computer-based grammar instruction is as effective as teacher-directed grammar instruction for postsecondary students at multiple levels of proficiency in an intensive ESL program Two experiments were conducted at different levels of English language proficiency, one at the first and second level of instruction and the other at the third and fourth level The primary research question posed in the study was whether there was a difference in the acquisition of a specific grammar point for students taught in a teacher-directed class versus those taught in computer-based instruction Volume 16 Number 51 Computer-Based and Teacher-Driven Instruction Method The first experiment compared the performance of level-one and leveltwo students in a computer-based group versus those in a teacher-directed group The second experiment compared the performance of level-three and level-four students in a computer-based group versus those in a teacherdirected group The purpose of conducting the two experiments was to examine the acquisition of discrete structures at different levels of proficiency in order to increase the generalizability of the results Sample The population of the study consisted of 53 students (24 females and 29 males) enrolled in an intensive academic ESL institute at a major university in Florida The ESL institute offers four levels of instruction and uses the Comprehensive English Language Test (CELT) to place in-coming students Ten students were enrolled in level one, in level two, 20 in level three (in two sections), and 14 in level four Students were matched for native language (Japanese, Thai, Korean, Arabic, Spanish, Italian, French, and Portuguese) and pretest scores on the structures in question and then randomly assigned to computer-based or teacher-directed sections Although the groups were not matched for gender, the breakdown was fairly even, each group differing in gender makeup by no more than two students Independent Variable The independent variable was the method of grammar instruction, either teacher-directed or computer-based Because the participants in each experiment consisted of two levels of English proficiency (first and second levels combined, third and fourth levels combined), grammar points were introduced at the lower level of the combined groups and then reviewed and expanded at the higher level This cycle held true for the teacherdirected as well as computer-based groups The treatment consisted of one hour of instruction per day for seven days Because studies have indicated that certain grammar points are more easily acquired than others (Green & Hecht, 1992; Scott & Randall, 1992; Krashen, 1981), this study examined the acquisition of verb tenses, which included elements of aspect, one of the more difficult forms for ESL students to master (Green & Hecht, 1992) 52 CALICO Journal Joyce Nutta Teacher-Directed Group Students in the teacher-directed groups were taught by five different teachers of varying degrees of experience Classes were held in rooms without computer equipment or other instructional technology Class size ranged from four to seven students Students at all levels used the Focus on Grammar (1994) textbook series and engaged in a variety of types of activities emphasizing interactive, meaningful, and creative expression Computer-Based Groups Students in the computer-based groups received computer-based instruction outside the classroom The students who participated in Experiment One used ELLIS Middle Mastery (1996), and the students who participated in Experiment Two used ELLIS Senior Mastery (1996) ELLIS was selected because of its multimedia delivery (audio and video, recording capability, etc.), its modeling of natural and contextualized language, its interactivity, and its clear grammar explanations and practice activities Although ELLIS lacks a mechanism for sophisticated learner feedback (see Nagata & Swisher, 1995), its exercises provide simple corrections Because the ELLIS program is organized around communicative needs rather than grammatical structures, students followed a sequential checklist developed by the researcher to direct their navigation through the program (See the sample navigational guide in Appendix A.) Dependent Variables In Experiment One, the grammar structure of interest was the past tense, and in Experiment Two, the conditional tense The dependent variables were students’ achievement scores on three separate criterion-referenced tests over the selected structures The tests on the past tense included items covering the simple past in regular and irregular forms, the past continuous, and the present perfect (regular form) The tests on the conditional tense included items covering the factual, unreal, and unlikely conditional The three tests consisted of (1) a discrete-point multiple-choice test, (2) a fill-in-the-blank test, and (3) an open-ended test The researcher, together with the ESL institute’s curriculum specialist and the grammar teachers, decided which grammar points were appropriate to teach and assess The researcher then developed test instruments based on the content of the instruction in the teacher-directed and computer-based groups The students’ scores on the three tests were retained as separate meaVolume 16 Number 53 Computer-Based and Teacher-Driven Instruction surements, not combined into a single dependent variable, since there is a theoretical basis for considering them as measures of different constructs For example, multiple choice tests not measure linguistic production (Hughes, 1989) and therefore can be appropriately used to assess students’ knowledge of grammar within the framework of grammatical consciousness-raising (Sharwood-Smith, 1981) Since open-ended tests measure students’ linguistic production, they represent more demanding tasks of application of knowledge than multiple choice tests The same battery of tests were used for pretests, immediate posttests, and delayed posttests A pool of test items was field tested at two adult education schools From the initial item pool, ten items were selected for the fill-in-the-blank tests (scale of zero to five), ten items for the multiple choice tests, and five items for the open-ended tests (scale of zero to one) Teachers at both adult education schools, as well as the grammar teachers at the institute, examined the tests for content validity and made suggestions for changes that were eventually incorporated into the final version of the tests In addition to analyzing students’ performance as represented by their values for the three dependent variables, the researcher conducted native language interviews with students in the computer-based group of Experiment One and administered questionnaires to students in the computerbased group of Experiment Two The purpose of these interviews and questionnaires was to solicit students’ views of the computer program and computer-based learning Data Collection and Analysis The pretests were administered three days prior to the beginning of the treatment, the immediate posttests were administered on the last day of the treatment, and the delayed posttests were administered two weeks after the posttests After having been trained, the classroom teachers administered the pretests and delayed posttests to all students and the immediate posttests to the students in the teacher-directed groups The researcher administered the immediate posttests to the students in the computer-based group The researcher scored all the tests blindly with no knowledge of students’ identity or group status After the researcher scored the tests, a certified ESL teacher reviewed students’ answers For all potentially problematic answers and scores, the researcher and the teacher reached consensus on the basis of the scoring sheet (See sample questions and scoring criteria in Appendix B.) For the pretests, immediate posttests, and delayed posttests in Experiment One, the calculation of internal consistency was 90 for the fill-inthe-blank test, 58 for the multiple choice test, and 63 for the open-ended 54 CALICO Journal Joyce Nutta test For the pretests, immediate posttests, and delayed posttests in Experiment Two, the calculation of internal consistency was 67 for the fillin-the-blank test, 40 for the multiple choice test, and 43 for the openended test Analysis of Covariance (ANCOVA) was used to analyze the data Because of the small sample size, the alpha level was set at 10 for tests of significance Results As in most comparative CALL studies, it was difficult to distinguish which features of the computer-based instruction led to the outcomes found in the study and whether these factors were inherent to computer-based instruction or simply an instructional strategy that could have been employed with a variety of media under different circumstances (see Williams & Brown, 1991) Nonetheless, the results showed that computerbased grammar instruction is at least as effective as, and in some cases more effective than, teacher-directed grammar instruction Experiment One Table summarizes the results of the descriptive analysis of students’ test scores in Experiment One Table Mean Test Scores by Group in Experiment One Type Time Computer-Based Group Teacher-Directed Group of Test of Test (N = 9) (N = 10) Mean Standard Mean Standard Deviation Deviation OpenPretest 0.44 0.53 0.30 0.68 Ended Immediate 3.10 1.76 1.20 1.62 Posttest Delayed 3.44 1.60 1.80 1.75 Posttest Fill-inPretest 13.67 11.14 8.20 9.27 the-blank Immediate 31.11 14.62 14.90 14.01 Posttest Delayed 22.67 17.95 16.80 12.04 Posttest Multiple Pretest 5.67 2.45 5.00 1.89 Choice Immediate 6.22 2.17 6.80 1.87 Posttest Delayed 6.56 2.46 5.70 1.77 Posttest Volume 16 Number 55 Computer-Based and Teacher-Driven Instruction Analysis revealed a significant difference between the test scores of the students in the computer-based and those in the teacher-directed groups on the open-ended immediate posttest (p < 10) The computer-based group’s mean score on this test was 3.10, and the teacher-directed group’s, 1.20 Using adjusted means, the effect size for the difference was 1.56 Analysis revealed a similar significant difference between the computerbased and the teacher-directed groups on the open-ended delayed posttest (p < 10) The computer-based group’s mean score was 3.44, and the teacher-directed group’s, 1.80 Again using adjusted means, the effect size for this difference was 1.73 Finally, analysis also showed a significant difference between the computer-based and teacher-directed groups’ mean scores on the fill-in-the-blank immediate posttest, 31.11 versus 14.90 (p < 05) This difference disappeared, however, by the time of the delayed posttest No significant differences were found between the groups for the multiple choice immediate posttest or delayed posttest Experiment Two Table summarizes the results of the descriptive analysis of students’ test scores in Experiment Two Table Mean Test Scores by Group in Experiment Two Type Time Computer-Based Group Teacher-Directed Group of Test of Test (N = 14) (N = 10) Mean Standard Mean Standard Deviation Deviation OpenPretest 3.14 1.56 2.80 0.92 Ended Immediate 4.50 1.02 4.20 0.79 Posttest Delayed 4.79 0.43 4.00 0.67 Posttest Fill-inPretest 37.29 2.89 38.90 5.02 the-blank Immediate 45.64 5.18 43.90 4.11 Posttest Delayed 46.29 3.73 45.90 4.61 Posttest Multiple Pretest 7.00 2.08 6.40 2.32 Choice Immediate 8.43 1.28 8.10 1.52 Posttest Delayed 8.71 1.33 8.90 1.10 Posttest 56 CALICO Journal Joyce Nutta Analysis showed a significant difference between the computer-based and the teacher-directed groups’ mean scores on the open-ended delayed posttest The computer-based group’s mean score was 4.79, and the teacherdirected group’s, 4.00 The effect size, once more using adjusted means, was 1.19 No significant differences were found on any of the other posttests Interviews and Questionnaires Insights into students’ experience with computer-based instruction emerged from the student interviews and questionnaires By and large, students were satisfied with the computer-based instruction and expressed a desire to spend more time per day using it Students indicated that the features of computer-based instruction that were most useful were the computer’s capacity that allowed them to review the tutorial as many times as they wished, to proceed at their own learning pace, to record their voices and compare them against the model, and to get immediate feedback on the exercises The Asian students especially appreciated not being “singled out” to speak in class, while some of the Latin students indicated that they would have preferred more human interaction Discussion Although the sample size of this study was too small to draw definitive conclusions, the study does present evidence of meaningful differences in the computer-based and teacher-directed groups’ achievement scores on the open-ended tests If open-ended tests measure students’ ability to use grammatical structures creatively, it would seem that some elements of computer-based study support the development of this skill more effectively than traditional classroom instruction Surprisingly, the scores of the students in the computer-based groups rose on the open-ended delayed posttests This result contrasts with those of other studies of classroom grammar instruction in which students’ production of grammatical structures was accurate on immediate posttests but fell on the delayed posttests Perhaps the interim period between the immediate posttest and the delayed posttest described here allowed the students in the computer-based group time to apply the structure in question in real communicative situations, providing them with an additional opportunity to negotiate meaning and monitor their own output This finding may indicate, as Ellis (1993) has suggested, that the use of computerbased grammar instruction can complement individualized structural syllabi in communicative classrooms and more effectively enable students to Volume 16 Number 57 Computer-Based and Teacher-Driven Instruction use the newly acquired structures to negotiate meaning The potential of the computer laboratory to go beyond providing simple practice and reinforcement of grammar points taught in the classroom is only beginning to be explored Research should be conducted with different populations (e.g., elementary and secondary students) and different types of courses (e.g., Vocational ESL and English for Specific Purposes) Moreover, additional research is needed to ascertain which features of computer-based grammar programs promote the acquisition of L2 structures (e.g., degree of learner control, feedback strategies, etc.) As the body of knowledge on the application of computer-based grammar instruction increases, so will the flexibility and number of options available to teachers and students 58 CALICO Journal Joyce Nutta Appendix A Sample ELLIS Navigational Guide Conversations by Grammar Topic Calling to set up a job interview—See your selection now Could I make an appointment for an interview? Click on each paragraph Culture Vocabulary Grammar Video (Script, Keywords, Role-Play—Record) Phrases Exit—Return to Conversations by Grammar Topic Date Conversations by Grammar Topic Calling for information about a job—See your selection now Could I make an appointment for an interview? Grammar Guide Conditional Sentences Easy—page view/hear Easy—page view/hear Easy—page view/hear Medium—page view/hear Difficult—page view/hear Review—page hear Date Main Activity Menu Practice Activities Practice Items Easy Grammar Practice Activities Practice Items Medium Grammar Practice Activities Practice Items Difficult Grammar Main Activity Menu Listening Activities Listening Comprehension Easy Medium Difficult Skills Check Volume 16 Number Date Date 59 Computer-Based and Teacher-Driven Instruction Appendix B Sample Test Items and Scoring Criteria Open-Ended If I didn’t have to study to get good grades, _ Scored as correct (1 point) or incorrect (0 points) Answers scored as correct must make sense and be correctly formed Multiple Choice _ the movie good last night? A Is B Did C Was D Were One correct answer per item Fill-in-the-Blank About two hours ago, a very unusual thing A student from the English Language Institute a large, round UFO… points—correct formal use of grammar and semantically correct points—correct use of formal grammar and semantically incorrect points—correct use of informal grammar and semantically correct points—correct use of informal grammar and semantically incorrect point—incorrect use of grammar and semantically correct Answers were judged as informal grammar use if they were variations of the structure that were not taught and that are not part of standard, written usage (e.g., Did you ever go to Europe?) NOTE This study was funded, in part, by a Florida Department of Education Section 353 Grant REFERENCES Adair-Hauck, B & Donato, R (1994) Foreign language explanations within the Zone of Proximal Development The Canadian Modern Language Review, 50, 532-557 Chun, D M., & Brandl, K K (1992) Beyond form-based drill and practice: Meaning-enhanced CALL on the Macintosh Foreign Language Annals, 25, 255-267 Doughty, C (1991) Second language instruction does make a difference Studies in Second Language Acquisition, 13, 431-469 60 CALICO Journal Joyce Nutta ELLIS [Computer software] (1996) Salt Lake City, UT: CALI Ellis, R (1985) Understanding second language acquisition Oxford: Oxford University Press Ellis, R (1993) The structural syllabus and second language acquisition TESOL Quarterly, 27, 91-113 Focus on Grammar: Longman Grammar Series (1994) New York: Addison Wesley Fotos, S S (1993) Consciousness raising and noticing through focus on form: Grammar task performance versus formal instruction Applied Linguistics, 14, 385-407 Fotos, S S (1994) Integrating grammar instruction and communicative language use through grammar consciousness-raising tasks TESOL Quarterly, 28, 323-351 Gilby, W (1996) Irrwege des Zweitsprachenerwerbs: Gehort auch das Computerlabor dazu? [False directions in second language acquisition: Does the computer laboratory also count as one?] Unterrichtspraxis/ Teaching German, 29, 87-91 Garrett, N (1991) Technology in the service of language learning: Trends and issues Modern Language Journal, 75, 74-101 Green, P & Hecht, K (1992) Implicit and explicit grammar: An empirical study Applied Linguistics, 13, 168-84 Hoffman, S (1996) Computers and instructional design in foreign language/ESL instruction TESOL Journal, (2), 24-29 Hughes, A (1989) Testing for language teachers New York: Cambridge Krashen, S (1981) Second language learning and second language acquisition Oxford: Pergamon Kulik, J & Kulik, C (1987) Review of recent research literature on computerbased instruction Contemporary Educational Psychology, 12, 222-30 Lavine, R Z (1992) Rediscovering the audio language laboratory: Learning through communicative tasks Hispania, 75, 1360-1367 McEnery, T., Baker, J P., & Wilson, A (1995) A statistical analysis of corpus based computer versus traditional human teaching methods of part of speech analysis Computer Assisted Language Learning, 8, 259-274 McNeil, B J & Nelson, K R (1991) Metaanalysis of interactive video instruction: A 10 year review of achievement effects Journal of Computer-Based Instruction, 18 (1), 1-6 Nagata, N (1996) Computer vs workbook instruction in second language acquisition CALICO Journal, 14, 53-75 Nagata, N & Swisher, M V (1995) A study of consciousness-raising by computer: The effect of metalinguistic feedback on second language learning Foreign Language Annals, 28, 337-347 Niemiec, R & Walberg, H (1987) Comparative effects of computer-assisted instruction: A synthesis of reviews Journal of Educational Computing Research, 3, 19-37 Volume 16 Number 61 Computer-Based and Teacher-Driven Instruction Quinn, R A (1990) Our progress in integrating modern methods and computercontrolled learning for successful language study Hispania, 73, 297-311 Rachal, J R (1995) Adult reading achievement comparing computer-assisted and traditional approaches: A comprehensive review of the experimental literature Reading Research and Instruction, 34, 239-258 Ragan, T., Boyce, M., Redwine, D., Savenye, W C., & McMichael, J (1993, January) Is multimedia worth it?: A review of the effectiveness of individualized multimedia instruction Paper presented at the Association for Educational Communications and Technology Convention, New Orleans, LA Scott, V M & Randall, S A (1992) Can students apply grammar rules after reading textbook explanations? Foreign Language Annals, 25, 357-367 Sharwood-Smith, M (1981) Consciousness-raising and the second language learner Applied Linguistics, 11, 159-168 Soo, K S & Ngeow, Y H (1996) The English teacher vs the multimedia computer: The UNIMAS experience Manuscript submitted for publication Tomasello, M & Herron, C (1988) Down the garden path: Inducing and correcting overgeneralization errors in the foreign language classroom Applied Psycholinguistics, 9, 237-246 Underwood, J H (1993, August/September) The lab of the future: Using technology to teach foreign language American Association of Community Colleges Journal, 33-39 Williams, C & Brown, S (1991) A review of the research issues in the use of computer-related technologies for instruction: An agenda for research Educational Media and Technology Yearbook, 17, 26-46 AUTHOR’S BIODATA Joyce W Nutta is an assistant professor of Foreign Language Education/ TESOL at the University of South Florida She holds a Master’s Degree in Applied Linguistics and a Ph.D in Second Language Acquisition and Instructional Technology Her research interests include CALL and distance learning She has co-authored Virtual Instruction: Issues and Insights from an International Perspective, Libraries Unlimited (in press) and is currently completing The Teacher’s Guide to Dynamic Distance Learning AUTHOR’S ADDRESS Joyce W Nutta, Ph.D Assistant Professor of Foreign/Second Language Education Department of Secondary Education—EDU 208B University of South Florida 4202 East Fowler Avenue Tampa, FL 33620-5650 Phone: 813/974-3759; 813/974-3533 Fax: 813/974-3837 E-Mail: jnutta@typhoon.coedu.usf.edu 62 CALICO Journal