Effects of Check-in/checkout on Behavioral Indices and Mathematics Generalization Michael D Mong University of Southern Mississippi Kristin N Johnson Eastern Illinois University Kristi W Mong Mong Psychological Associates ABSTRACT: Check-in/checkout (CICO) is a behavioral intervention that is used to provide systematic feedback about a student's behavior at the beginning and end of each school day The purpose of the present study was to evaluate the effectiveness of CICO on decreasing problem behaviors and the collateral effects on mathematics performance for at-risk elementary school students A multiple-baseline across-participants design using dyads was used to analyze problem behavior collected through direct classroom observations As ancillary measures, office discipline referrals per week as well as mathematic performance (i.e., digits correct per minute [DCPMj) were collected for each student Treatment integrity and acceptability and social validity were also measured Results indicate a decrease in problem behaviors as well as an increase in DCPM for each participant • When children receive office discipline referrals (ODRs), they can often simultaneously exhibit a multitude of issues including academic and behavioral problems These problems rarely exist in isolation, and in combination they put students in more dramatic risk of school failure (Mclntosh, Flannery, Sugai, Braun, & Cochrane, 2008) Thus, the relationship between academic performance and problem behaviors provides concern because of their documented interaction (Maguin & Loeber, 1995; Roeser & Eccles, 2000) Students with early behavior difficulties are at greater risk for developing academic problems (Fleming, Harachi, Cortes, Abbott, & Catalano, 2004), and students with early academic difficulties are at greater risk for developing problems in social behavior (Mclntosh, Horner, Chard, Boland, & Good, 2006; Morrison, Anthony, Storino, & Dillon, 2001) Given the documented relationship between academic and behavior problems, positive behavior support (PBS) experts recommend a three-tiered model of behavior supports to prevent and intervene with problem behavior (Sugai & Horner, 2001) As students progress through the tiers, the intensity of the intervention increases, as does the Behavioral Disorders, 36 (4), 225-240 cost of resources The purpose of the first tier is to provide primary prevention for all students Those students whose behaviors continue to be discrepant from their peers are identified for additional support at Tier Approximately 15% to 20% of the population will benefit from this level of support (Walker & Shinn, 2002) Tier interventions offer at-risk students additional opportunities to learn expected behaviors that lead to educational success (Lee, Sugai, & Horner, 1999) These services are provided in addition to the core instruction (National Association of State Directors of Special Education, 2005) Tier interventions are for those students who are highly discrepant from their peers either in behavioral excesses or deficits In addition, these students referred to Tier may be nonresponsive to Tier interventions For behavior Filter and colleagues (2007) recommended interventions be implemented with students who have two or more ODRs but whose problem behaviors not pose an immediate danger to self or others These interventions should be de!signed to be quickly accessed, flexible, and to bring about improvement (Hawken & Horner, 2003) One intervention that may meet these requirements is the check-in/checkout (CICO) program August 2011 / 225 The CICO program was developed as an efficient intervention for reducing problem behavior The CICO program was designed to increase feedback and positive adult attention (Crone, Horner, & Hawken, 2004) Previous research has shown that CICO is an easy-to-implement behavioral strategy and is effective at decreasing ODRs (Hawken & Horner, 2003; March & Horner, 2002), decreasing observed problem behaviors (Fairbanks, Sugai, Guardino, & Lathrop, 2007), and increasing appropriate behavior (Todd, Campbell, Meyer, & Horner, 2008) The CICO program has been shown to be a relatively simple and inexpensive intervention (Hawken, MacLeod, & Rawlings, 2007) The CICO program has also been shown to have relatively high levels of teacher and staff acceptability Additionally, previous research (Todd et al., 2008) has shown that typical school personnel were able to effectively implement the CICO intervention If implementing CICO demonstrates students engaging in fewer problem behaviors and spending less time in the school office, improved academic achievement may likely follow (Hawken et al., 2007) Indeed, Hawken and Horner (2003) documented increases in academic engagement following CICO implementation by assessing if the student was (a) looking at the teacher while the teacher was giving instructions/directions, (b) working with a peer when instructed to so, (c) reading silently or completing a writing assignment, (d) participating in a teacher-approved activity if work was completed, or (e) talking about academic material with the teacher or aide for at least s Although the primary focus of the study was on classroom problem behaviors, a secondary analysis indicated that the CICO intervention was associated with increases in mean level of academic performance in mathematics for all students While the previous research is indeed promising, few studies have examined CICO's effect on outcome variables such as academic performance and achievement (Hawken et al., 2007) The main idea is that although CICO may or may not have direct effects on academic achievement, students who spend less time engaging in inappropriate behaviors may replace those behaviors with appropriate behavior Time on task or engagement leads to better academic performance Furthermore, few studies have provided measures of CICO effect size Reporting effect sizes is considered best 226 / August 2011 practice when presenting empirical research findings in many fields (Wilkinson, 1999) Another limitation of previous CICO research involves the manner in which social validity was assessed To date, few studies have measured the social validity of CICO using a measure with validated psychometric properties Thus, the purpose of the present study was to (a) extend the literature by examining CICO's effect on classroom problem behaviors as measured by direct behavioral observations and ODRs, (b) determine whether the effects of a behavioral intervention (CICO) affected student mathematics performance, and (c) assess the social validity of CICO using a measure with validated psychometric properties Method Participants and Setting The study was conducted in a suburban elementary school located in the southeast United States with approximately 415 students (Grades 3-5), 65% of whom qualified for free or reduced lunch and 36% of whom were from ethnic minority backgrounds The school had been implementing schoolwide PBS for more than years The Schoolwide Evaluation Tool (Horner, Sugai, Todd, & Lewis-Palmer, 2005) results indicated that the school was implementing its schoolwide behavior support plan with 82% treatment integrity Students were selected for participation in the study if (a) they received five ODRs in a single month, (b) their problem behavior occurred frequently across multiple settings throughout the school day as noted from ODR analysis and teacher interviews, and (c) the results of a functional behavioral assessment (FBA) indicated that the function of the students' problem behavior was hypothesized to be attention seeking The problem behaviors exhibited by these students are particularly alarming given that the long-term outcomes for students who exhibit early patterns of maladaptive behavior are continued poor academic performance, referral for special education identification and placement, social rejection, low self-esteem, potential for developing a more chronic psychological disorder, and increased school dropout rates (Walker, Ramsey, & Gresham, 2004) Informed consent to participate was obtained for each student from his or her parent/ Behavioral Disorders, 36 (4), 225-240 legal guardian Of the students selected for intervention, were boys and were girls Students ranged in age from 8.4 to 9.1 years, with a mean age of 8.7 years All students were enrolled in the third grade In terms of race, there were Caucasian students and African American students Lauren Lauren was an 8-year-old Caucasian girl who has never received special education services, has no previous diagnoses, and has no history of retention The majority (87%) of Lauren's ODRs resulted from inappropriate behavior in the classroom She was referred by her classroom teacher for talking out during classroom instruction, which accounted for 18% of her total ODRs; noncompliance with teacher demands (40%); disrupting peers during independent seat work (29%); and off-task behavior across settings (13%) including the classroom and cafeteria Based on the results of the FBA, Lauren's problem behaviors were hypothesized to be maintained primarily by adult attention with peer attention as a secondary function During the immediate months before the study, Lauren averaged 2.1 (range, 1-3) ODRs per week Andrew Andrew was an 8-year-old African American boy who has never received special education services, has no previous diagnoses, and has no history of retentions Andrew was receiving Title I services for reading and math The majority of Andrew's ODRs (93%) resulted from inappropriate behavior in the regular classroom or music classroom during instruction He was referred by his classroom teacher or music teacher for noncompliance with teacher demands (37%), oft'-task behavior (22%), and talking out (41%) in the general education class as well as music and physical education classes Based on the results of the FBA, Andrew's problem behaviors were hypothesized to be maintained by adult attention During the immediate months before the study, Andrew averaged 4.1 (range, 2-5) ODRs per week Pam Pam was a 9-year-old African American girl who has never received special education services, has no previous diagnoses, and has no history of retentions The majority of Pam's ODRs (92%) occurred in the classroom during instruction and independent seat work She was referred by her classroom teacher for refusing to complete assignments (14%), noncompliance with teacher demands (54%), talking out in class (24%), and noncompliance with bus driver demands (8%) Based on the results of the FBA, Pam's problem Behavioral Disorders, 36 (4), 225-240 behaviors were hypothesized to be maintained primarily by adult attention with peer attention as a secondary function During the immediate months before the study, Pam averaged 1.9 (range, 1-3) ODRs per week Stanley Stanley was an 8-year-old Caucasian boy who has never received special education services, has no previous diagnoses, and has no history of retentions The majority of Stanley's ODRs (92%) occurred in the classroom during independent seat work He was referred by his classroom teacher for offtask behavior (34%), noncompliance with teacher demands (17%), and talking out in class (41%) as well as off-task behavior in physical education (4%) and in schoolwide specials (4%) Based on the results of his FBA, Stanley's problem behaviors were hypothesized to be maintained primarily by adult attention with peer attention as a secondary function During the immediate months before the study, Stanley averaged 2.0 (range, 1-3) ODRs per week In addition, two guidance counselors and the students' classroom teachers agreed to participate in the study, with the counselors identified as the CICO specialists Prior to the initiation of each phase, the primary investigator trained the teachers and CICO specialists on the required procedures associated with each phase of the study The training consisted of the primary investigator describing the procedures, modeling the procedures, and having the staff member practice the procedures with the primary investigator providing feedback This format was implemented until the individuals were able to implement the procedures independently The primary investigator was available for any questions or concerns from the teachers and CICO specialists as they arose FBA Procedures Prior to initiation of the study, an FBA was conducted for each student The assessment process involved a 20- to 40-min interview conducted by the primary author with each participant's teachers using the Functional Analysis Informant Record for Teachers (FAIR-T; Edwards, 2002) The purpose of the EAIR-T was to determine and help define the problem behaviors, to determine appropriate replacement behaviors, to determine potential reinforcers for appropriate behavior, and also to help formulate the hypothesized function of the presenting problem behavior To date, studies August 2011 / 227 have supported the use of the FAIR-T in such a manner (Doggett, Edwards, Moore, Tingstrom, & Wilczynski 2001; Dufrene, Doggett, Henington, & Watson, 2007; Edwards, 2002) Following the FAIR-T, three 20-min conditional probability observations were conducted in the student's natural classroom setting prior to intervention A FAIR-T hypothesis statement was judged to be supported if the direct observation data provided similar information to the antecedent and consequent events defined in the FAIR-T hypothesis statement As CICO was designed to increase feedback and positive adult attention (Crone et al., 2004), it was important to match the behavioral intervention to the function of the participant's problem behaviors Measurement Direct observation of problem behavior Problem behavior was observed days per week using a 20-min, 10-s partial interval recording system For each participant, observations took place during the same academic class period each day The specific class period for each student was determined by teachers' reports of the most problematic time of day based on the FAIR-T interview Problem behaviors included (a) noncompliance, (b) talking out, and (c) off-task behavior Noncompliance was defined as failure to complete assigned instructions or failure to initiate commands within s Talking out was defined as the student engaging in vocalizations that were not preceded by a raised hand and/or were not initiated by an adult Off-task behavior was defined as the student oriented away (e.g., face or body) from the teacher or materials during instruction or oriented toward irrelevant persons or objects, manipulating materials or objects inappropriately that are relevant to the assigned task or activity, or doing other behaviors that are not related to his or her assignment or task for a period of 10 s or more For all students, the observations occurred in mathematics as it was considered by the teachers as one of the most problematic times of the day Direct observation of problem behavior was calculated by dividing the number of intervals with observed problem behavior by the total number of intervals and multiplying this ratio by 100 to obtain a percentage Office discipline referrals The ODRs were collected to compare the rates of problem behaviors before and during the CICO program and reported as a weekly measure 228 / August 2011 As a measure of behavior, ODRs possess sufficient construct validity and adequate concurrent validity with a number of standardized measures of individual behavior (as cited in Irvin, Tobin, Sprague, Sugai, & Vincent, 2004) as well as predictive validity for negative school outcomes, including physical assaults and dropout (Tobin & Sugai, 1999) Office discipline referrals have also been identified as an effective and efficient measure for decision making in schools (Irvin et al., 2006) Percentage of daily progress reports points earned Each student's percentage of daily progress report (DPR) points earned was examined as a measure of appropriate behavior Evaluations of the DPR suggest that it possesses internal consistency, temporal stability, and concurrent validity and that its sensitivity allows for detection of treatment effects (Pelham, Fabiano, & Massetti, 2005) Treatment Integrity The CICO treatment integrity checklist Treatment integrity was measured in a similar manner used by Hawken et al (2007) For each integrity assessment, the first author examined the CICO daily student intervention protocol to determine the degree with which the CICO intervention was implemented as prescribed Treatment integrity was evaluated during 33% of the sessions evenly distributed across all phases of the study based on completion of a checklist during the session Treatment integrity was calculated by the number of items on the checklist completed divided by the total number of items on the checklist and multiplied by 100 Academic Pretreatment Assessment A Web-based computer program Math Worksheet Generator (interventioncentral.org), was used to generate curriculum-based mathematics worksheets The program allows the user to design worksheets requiring the use of specific skills State benchmarks were used to determine which skills were representative of each grade level The program was then used to create a worksheet specific to a particular grade level and state benchmark The computer program randomized the order of the (a) problems within a worksheet and (b) factors within each problem Each worksheet listed problems in six rowsof four problems in portrait orientation on a regular 'A- by 11 -inch sheet of Behavioral Disorders, 36 (4), 225-240 white paper Each worksheet contained the 24 problems Prior to beginning the study, the third author used curriculum-based assessment (CBA) procedures to identify each student's instructional level Multiple skill CBA probes were administered to determine the current grade-level performance of each of the students The student was given one worksheet at his or her current grade placement in school For example, a third-grade student was given an opportunity to complete a third-grade-level probe Each student was given 60 s to complete each worksheet The digits correct per minute (DCPM) on each worksheet was determined by the number of digits written correctly during a Interval, divided by the number of seconds worked and multiplied by 60 Errors per minute (EPM) served as a secondary dependent variable Responses were scored as errors if incorrect digits were written below the line or if digits were written in the wrong place or omitted If performance was determined to be in the instructional-level range, a worksheet at the same grade level was administered If the student performed at the frustrational level (less than 14 DCPM), a worksheet at a lower grade level was administered These procedures were repeated until a median instructional level performance was obtained across three worksheets within the same grade level to establish baseline According to Burns, VanDerHeyden, and Jiban (2006), a student's independent instructional level was the point in the curriculum at which he or she could complete math problems with 14 to 31 digits correct if enrolled in first through third grade The independent instructional level was the point in the curriculum at which a student could complete math problems obtaining 24 to 49 digits if enrolled in Grades and higher During the weeks of CICO intervention, all students received math instruction once per day and did not receive any math interventions Furthermore, analysis of the classroom teachers' weekly lesson plans indicated that mathematics instruction focused on skills (i.e., estimating and rounding, analyzing graphs) other than mathematics computation Thus, the instruction in the classroom did not necessarily directly address the skills on the mathematic computation probes Academic Progress Monitoring To determine student progress, each student completed three multiple skill grade-level Behavioral Disorders, 36 (4), 225-240 probes at the predetermined instructional level to determine the student's median DCPM Problems included addition with two-digit numbers with regrouping (i.e., 23 -i-18), addition with two three-digit numbers with regrouping (i.e., 156 -H 379), subtraction with two two-digit numbers with regrouping (i.e., 48 — 19), subtraction with two three-digit numbers without regrouping (i.e., 275 - 130), multiplication facts to 10 (i.e., X 5), and division facts to 10 (i.e., -H 2) The multiple skill instructional-level probes were administered once per week The use of the multiple skill probes allowed the evaluation of level and rate of progress changes on instructional-level material Social Validity The Behavior Intervention Rating Scale (BIRS; Treuting & Elliott, 1991) is a 24-item scale that employs a 6-point Likert-type format (1 = strongly disagree to = strongly agree) to measure teachers' perceptions of treatment acceptability and the perceived efficacy of classroom interventions The BIRS is composed of three factors (Acceptability, Effectiveness, and Time of Effect) and a Cronbach's alpha of 97 for the total scale (Elliott & Treuting, 1991) The Acceptability factor is composed of 15 items with a Cronbach's alpha of 97 The Effectiveness factor consists of seven items with a Cronbach's alpha of 92 The Time of Effect factor is composed of two items with a Cronbach's alpha of 87 (Elliot, 1998) The BIRS was completed by both guidance counselors and the participant's classroom teacher at the end of each week for a total of eight measurements per respondent The mean scores for each factor across the CICO intervention phase will be reported The five-item BEP Acceptability Questionnaire (Hawken & Horner, 2003) was used to assess the social validity of the intervention with the participants Questions on the BEP Acceptability Questionnaire assessed the extent to which the CICO was perceived to (a) improve problem behavior at school, (b) improve academic performance, (c) be worth the time and effort, (d) be worth recommending to others, and (e) be easy to implement Scores on the BEP Acceptability Questionnaire were recorded on a Likert-type scale (1-6), with higher scores indicating a more favorable impression of the CICO intervention The participants responded to the questionnaire once per week for a total of eight measure- August 2011 / 229 ments per respondent The mean scores for each question across the CICO intervention phase will be reported Reliability of Measures Interobserver agreement for problem behavior observation data Interobserver agreement (lOA) for problem behavior data was collected using a second independent observer for 33% of the sessions evenly distributed across all phases of the study The lOA was calculated by adding the number of intervals of agreement of problem behavior for each session, then dividing by the total number of observed intervals for each session, and then multiplying this ratio by 100 to obtain a percentage Interobserver agreement of CICO sessions As an additional measure of treatment integrity, the third author completed the CICO daily intervention student protocol during direct observation of 33% of all the CICO intervention sessions Treatment integrity was calculated by the number of items on the checklist completed divided by the total number of items on the checklist and multiplied by 100 Interscorer agreement The primary author designed a list of scoring instructions for the mathematics probes used in the investigation One scorer and the primary investigator scored a sample of 15 probes independently The rules were clarified and revised until there was at least 90% agreement on a set of 45 sample probes Scorers were then cleared for scoring of the probes Approximately 33% of the total probes were independently scored by the one scorer and the primary investigator across all phases of the study Design and Procedure The experimental design for this study was a combined series multiple baseline across students design The CICO intervention was first applied to the student with the most stable baseline in terms of problem behaviors After an intervention effect was demonstrated and the subsequent students' baselines remained stable, the intervention was applied to the student with the next most stable baseline (Carr, 2005) To demonstrate empirical control and to avoid delays to intervention, students were paired in dyads to create two multiplebaseline pairs for the current project Two phases were implemented for this study: baseline and CICO 230 / August 2011 Visual analyses for level, trend, and variability were used to determine effects as well as two statistical procedures for effect sizes Effect sizes were calculated using the percentage of nonoverlapping data points all (PND; Olive & Smith, 2005) PND is calculated by dividing the number of nonoverlapping data points with baseline by the total number of intervention data points The lowest baseline data point was used to establish the overlap of baseline data points with intervention data points for observed behavior and ODRs, whereas the highest baseline data point was used for the percentage of DPR points earned Benchmarks for PND scores have also been established by Scruggs and Mastropieri (2002) Specifically, PND scores below 50% suggest an ineffective intervention effect, scores between 50% and 70% suggest a questionable intervention effect, scores between 70% and 90% suggest an effective intervention effect, and scores greater than 90% suggest a very effective intervention effect Previous researchers (Campbell, 2004; Olive & Smith, 2005) concluded that PND is valid for documenting the effects of interventions in single-subject research Baseline During baseline, typical schoolwide behavior support procedures were in place for all students, including those participating in this study During baseline, direct observation of problem behavior and assessment of academic skills for target students was conducted The ODRs were monitored through collection of the written ODR reports In addition, the students' teachers were given three practice DPRs the week prior to implementation of CICO to ensure that only the target behaviors were rated These practice DPRs served as a baseline measure of the percentage of DPR points earned Check-in/Checkout The CICO process involved the following five elements: (a) Students were required to check in with the guidance counselor of their choice before school The counselor provided the student with a DPR form that was carried to class for feedback throughout the day When students checked in, they were asked if they had their DPR from the day before signed by their parents and if they had their materials ready for the school day They received praise Behavioral Disorders, 36 (4), 225-240 and a lottery ticket for a weekly drawing for checking in Also during check in, students were prompted to identify daily goals and given feedback to encourage success, (b) At three specified times of the day, students approached the teacher with the CICO report card, and the teacher provided the student with feedback about the student's behavior by rating either (did not meet expectations), {somewhat met expectations), or (met expectations) Teachers also provided immediate verbal praise for students who met behavioral expectations for that time period and corrective feedback if students did not meet the expectations, (c) At the end of the school day, students took the DPR to the counselor to check out The percentage of points earned for the day was calculated, and students received verbal praise and rewards if they met their daily point goal Students could choose from among specific rewards determined by a forced-choice preference assessment including stickers, pencils, time with a preferred adult, or extra recess time the following day Similar to the procedures used by Hawken et al (2007), 80% of the total points earned was the daily goal for all students in this study If students did not meet their daily goal, the counselor provided information on what to work on for the following school day (d) Students then took their DPR home to be signed by a parent/ guardian, (e) The DPR was signed by a parent/ guardian and returned the following morning Results Direct Observation of Problem Behavior Figure I summarizes the results across participants During baseline, all participants displayed variable levels of problem behaviors, with an overall mean of 32.8% (range, 21-42.7) Upon introduction of the CICO intervention phase, Lauren and Andrew displayed an immediate reduction in level and trend, whereas Pam and Stanley displayed gradual decreases in level and trend The participants were observed to engage in an overall mean of 21.4% (range, 16.9-29.3) across the CICO phase Lauren At baseline, Lauren displayed problem behaviors in 31.3% (range, 28%-34%), with an increasing trend and moderate variability With regard to the CICO intervention phase, an immediate decrease in level and trend (mean Behavioral Disorders, 36 (4), 225-240 of 17.9%; range, 12%-25%) was observed as compared with the baseline phase The PND for Lauren was 100%, suggesting that the CICO intervention was effective at decreasing her observed problem behaviors Andrew On average, Andrew displayed problem behaviors in 42.7% (range, 39%48%) with relatively no trend with moderate variability across the baseline phase With regard to the CICO intervention phase, an immediate decrease in level and trend (mean of 21.6%; range, 19%-26%) was observed as compared with the baseline phase The PND for Andrew was 100%, suggesting that the CICO intervention was effective at decreasing his observed problem behaviors Pam At baseline, Pam displayed problem behaviors in % (range, 17%-24%) with increasing trend and moderate variability With regard to the CICO intervention phase, a gradual decrease in level and trend was observed as compared with the baseline phase Pam averaged 16.9% (range, 10%25%) of problem behaviors across the phase The PND for Pam was 63%, suggesting that the CICO intervention was somewhat effective at decreasing her observed problem behaviors Stanley At baseline, Stanley displayed problem behaviors in 36% (range, 32%-40%) with a gradually increasing trend and moderate variability In the CICO intervention phase, a gradual decrease in level and trend was observed as compared with the baseline phase Stanley averaged 29.3% (range, 25%6%) of problem behaviors across the phase The PND for Stanley was 75%, suggesting that the CICO intervention was effective at decreasing his observed problem behaviors Office Discipline Referrals Figure summarizes the results across participants During baseline, all participants displayed slight variability with ODRs, with an overall mean of 3.3 (range, 1.7-5.0) ODRs per week Upon introduction of the CICO intervention phase, all participants displayed a reduction in level and trend Overall, the participants had a mean of 1.2 (range, 0.63-2.3) ODRs Lauren At baseline, Lauren had 2.3 ODRs per week (range, 2-3) with a relatively level trend and slight variability In the CICO intervention phase, a gradual decrease in level and trend was observed as compared with the baseline phase Lauren averaged ODR per August 2011 / 231 100 Baseline CICO 80 Lauren 60 40 20 12 12 15 18 21 24 27 30 33 100 Andrew? 80 60 O 40 X ta 20 I 123 12 15 IE I I—I—I 21 I I 24 I—I—I I—I I 27 30 I I i_j 33 36 39 42 Pam 60 40 20 12 12 15 18 21 24 27 30 33 100 Stanley 80 60 40 20 -•—>—I—I—I—I—I I 123 I—I—i—I 12 I 15 I iXi t—1 I I • • • • 18 21 24 27 SESSIONS 30 33 36 39 42 Figure Percentage oí intervals with problem behavior for Lauren, Andrew, Pam, and Stanley week (range, 0-3) The PND for Lauren was 75%, suggesting that the CICO intervention was effective at decreasing the student's weekly ODRs Andrew At baseline, Andrew had 4.3 ODRs per week (range, 4-5) with relatively no trend and slight variability In the CICO intervention phase, an immediate decrease in level and trend was observed as compared 232 / August 2011 with the baseline phase Andrew averaged ODR per week (range, 0-3) The PND for Andrew was 100%, suggesting that the CICO intervention was effective at decreasing the student's weekly ODRs Pam At baseline, Pam had 1.7 ODRs per week (range, 1-2) with an increasing trend and slight variability For the CICO intervention phase, a gradual decrease in level and trend Behavioral Disorders, 36 (4), 225-240 10 -1 O Baseline CICO Lauren 10 10 11 12 13 11 14 10 O Pam 10 10 11 Stanley Û 10 11 12 13 14 •WEEKS Figure Weekly office discipline referrals (ODRs) for Lauren, Andrew, Pam, and Stanley was observed Pam averaged ODR per week (range, 0-3) The PND for Pam was 63%, suggesting that the CICO intervention was somewhat effective at decreasing the student's weekly ODRs Stanley At baseline, Stanley had 5.0 ODRs per week (range, 4-6) with relatively Behavioral Disorders, 36 (4), 225-240 no trend and slight variability For the CICO intervention phase, a gradual decrease in level and trend was observed Stanley averaged 2.3 ODRs per week (range, 0-5) The PND for Stanley was 75%, suggesting that the CICO intervention was effective at decreasing the student's weekly ODRs August 2011 / 233 Percentage of DPR Points Earned Lauren During baseline, Lauren earned an average of 48% (range, 44%-51%) of her possible DPR points Under CICO, Lauren earned an average of 79% (range, 66%-96%) of her possible weekly DPR points The PND for Lauren was 100%, suggesting that CICO was effective at increasing the percentage of daily DPR points earned Andrew During baseline, Andrew earned an average of 37% (range, 32%-41%) of his possible DPR points Under CICO, Andrew earned an average of 82% (range, 69%-91%) of his possible weekly DPR points The PND for Andrew was 100%, suggesting that CICO was effective at increasing the percentage of daily DPR points earned Pam During baseline, Pam earned an average of 56% (range, 53%-59%) of her possible DPR points Under CICO, Pam earned an average of 75% (range, 58%-88%) of her possible weekly DPR points The PND for Pam was 92%, suggesting that CICO was effective at increasing the percentage of daily DPR points earned Stanley During baseline, Stanley earned an average of 44% (range, 41%-47%) of his possible DPR points Under CICO, Stanley earned an average of 78% (range, 47%-98%) of his possible weekly DPR points The PND for Stanley was 92%, suggesting that CICO was effective at increasing the percentage of daily DPR points earned Academic Progress Monitoring Figure summarizes the DCPM and EPM obtained for each student on multiple skill grade-level probes The data were visually analyzed separately with regard to changes in level, trend, and variability Students' actual rate of improvement was also visually analyzed on grade-level multiple skill probes Lauren According to the pretreatment assessment, Lauren was performing at the lower end of the instructional range for her grade level, with a median score of 14 DCPM Visual inspection revealed a level trend with a slight degreeof variability with a mean of 14.7 DCPM (range, 14-15) across the baseline phase For the CICO intervention phase, a gradual increase in level, trend, and variability was observed with a mean 16.3 DCPM (range, 14-20) Visual analysis of errors suggests a gradually decreasing trend and slight variability with a mean of 1.3 EPM (range, 1-2) across baseline Errors in 234 / August 2011 the CICO intervention phase were observed to decrease in level, trend, and variability with a mean of 0.38 EPM (range, 0-1 ) Andrew According to the pretreatment assessment using Curriculum Based Measurement (CBM), Andrew was performing at the appropriate instructional level for his grade, with a median score of 15.5 DCPM Visual inspection revealed a gradually increasing trend with some degree of variability with a mean of 15.3 DCPM (range, 14-16) across the baseline phase With regard to the CICO intervention phase, a gradual increase in level, trend, and variability was observed with a mean 17.0 DCPM (range, 1618) across the phase Visual analysis of errors suggests a relatively level trend and slight variability with a mean of 1.3 EPM (range, 0-2) across baseline Errors in the CICO intervention phase were observed to decrease in level, trend, and variability with a mean of 0.38 EPM (range, 0-2) across the phase Pam According to the pretreatment assessment using CBM, Pam was performing at the lower end of the instructional range for her grade level, with a median score of 14 DCPM Visual inspection revealed a gradually increasing trend with some degree of variability with a mean of 13.7 DCPM (range, 13-14) across the baseline phase With regard to the CICO intervention phase, a gradual increase in level, trend, and variability was observed with a mean of 14.1 DCPM (range, 11-17) across the phase It may be interesting to note that Pam achieved her lowest performance (11 DCPM) during the second week of the CICO intervention This low score coincided with her highest percentage of intervals with observed problem behavior (25%) Visual analysis of errors suggests a decreasing trend with slight variability with a mean of 1.7 EPM (range, 1-2) across baseline Errors in the CICO intervention phase were observed to gradually decrease in level and trend with moderate variability and a mean of 0.38 EPM (range, 0-1) across baseline Stanley According to the pretreatment assessment using CBM, Stanley was performing at the appropriate instructional level for his grade, with a median score of 21 DCPM Visual inspection revealed a gradually increasing trend with some degree of variability with a mean of 20.8 DCPM (range, 20-22) across the baseline phase With regard to the CICO intervention phase, a gradual increase in level, trend, and variability was observed with a mean 22.3 DCPM (range, 20-24) across the phase Visual analysis of Behavioral Disorders, 36 (4), 225-240 •I-]- Baseline 25 20 15 10 EPM 25 10 11 Andrew 20 15 10 ! O I I I R 10 11 1? 14 I i 25 ' 20 15 10 Pam 25 10 10 11 Stanley 20 15 10 "WEEKS U 12 13 14 Figure Digits correct per minute (DCPM) and errors per minute (EPM) for Lauren, Andrew, Pam, and Stanley on multiple skill worksheets errors suggests a relatively level trend with slight variability was observed with a mean of 1.8 EPM (range, 0-3) across baseline Errors in the CICO intervention phase were observed to gradually decrease in trend, level, and variability with a mean of 0.75 EPM (range, 0-2) across baseline Behavioral Disorders, 36 (4), 225-240 Social Validity Table summarizes the guidance counselors' and the participant's classroom teachers' perceptions of social validity using the BIRS Mean counselor ratings on the Acceptability August 2011 / 235 TABLE Guidance Counselor and Teacher Ratings on the BIRS BIRS Scale Guidance Counselors, M (Range) Teachers, M (Range) Acceptability 76 (74-77) Effectiveness Time of Effect TABLE Student Ratings on the BEP Acceptability Questionnaire BEP Question Student Ratings, M (Range) 69 (67-71) Improves problem behavior at school 4.5 (4-5) 33 (31-35) 36 (33-38) Worth time and effort 4.5 (4-5) (7-8) (5-7) Note The maximum scores for the factors are 90 for Acceptability, 42 for Effectiveness, and 12 for Time of Effect Higher scores indicate agreement with each factor Recommended to others (5-5) Easy to participate (4-4) Improves academic performance 3.5 (3-4) Note Higher scores indicated agreement with the question The maximum score for each question was factor were 76 (range, 74-77) out of the possible 90 Mean counselor ratings on the Effectiveness factor were 33 (range, 31-35), out of the possible 42 Mean counselor ratings on the Time of Effect factor were (range, 7-8), out of the possible 12 This finding indicates that both guidance counselors viewed the CICO intervention as acceptable and effective but were slightly concerned with the amount of time needed to implement CICO Mean teacher ratings on the Acceptability factor were 69 (range, 67-71) out of the possible 90 Mean teacher ratings on the Effectiveness factor were 36 (range, 33-38) Mean counselor ratings on the Time of Effect factor were (range, 5-7) This finding indicates that the participant's classroom teachers viewed the CICO intervention as acceptable and effective but were somewhat concerned with the amount of time needed to implement CICO Table summarizes the students' perceptions of social validity using the five-item BEP Acceptability Questionnaire (Hawken & Horner, 2003) Mean participant ratings for whether the CICO intervention was perceived to improve problem behavior at school and whether CICO was worth the time and effort were 4.5 on a 6-point scale The highest mean participant rating (5) involved whether the intervention was worth recommending to others Mean participant ratings for whether CICO was easy to participate in and for whether CICO improved academic performance were and 3.5, respectively Interobserver Agreement for Problem Bebavior Observation Data The actual overall mean lOA was 94% (range, 93%-98%); for Lauren, it was 95% (range, 94%-95%); for Andrew, 95% (93%236 / August 2011 98%); for Pam, 94% (range, 93%-95%); and for Stanley, 94% (range, 93%-95%) Treatment Integrity For Lauren, mean treatment integrity was 88%, (range, 85%-92%); for Andrew, 98% (range, 95%-100%); for Pam, 92% (range, 90%-94%); and for Stanley, 90% (89%-91%) Overall mean treatment integrity for this study was 92% (range, 85%-100%) The lOA for treatment integrity for all participants across all observed CICO sessions was 100% Interscorer Agreement For Lauren, the mean interscorer agreement was 99% (range, 98%-100%); for Andrew, 99% (range, 97%-100%); for Pam, 99% (range, 97%-100%); and for Stanley, 99% (97%-100%) Overall mean interscorer agreement was 99% (range, 97%-100%) Discussion Given that approximately 13% of elementary students demonstrate between two and six ODRs per year despite the presence of primary-level preventative measures (Horner et al., 2005), there is a great need for effective and cost-efficient secondary interventions Concurrent with the findings of previous research (Filter et al., 2007; Hawken & Horner, 2003; Hawken et al., 2007; Todd et al., 2008), the CICO intervention effectively reduced the problem behaviors of all participants In terms of direct observations of problem behaviors, the participants displayed variable levels of problem behaviors with an overall mean of 32.8% of observed intervals Behavioral Disorders, 36 (4), 225-240 with problem behaviors Upon introduction of the CICO intervention, all participants displayed a reduction in level and trend The participants engaged in an overall mean of 21.4% (range, 16.9%-29.3%) of observed intervals with problem behaviors across the CICO phase With respect to weekly ODRs, the participants displayed an overall mean of 3.3 (range, 1.7-5.0) ODRs per week across the baseline phase Upon introduction of the CICO intervention, all participants displayed a reduction in level and trend in terms of their weekly ODRs Overall, the participants had a mean of 1.2 (range, 0.63-2.3) ODRs Also concurrent with previous research, this study has shown that both school faculty and staff were able to implement the necessary procedures with a high degree of treatment integrity Actual overall mean treatment integrity for this study was 92% (range, 85%-100%) Hawken and Horner (2003) demonstrated that the CICO intervention was associated with increases in the mean level of academic engagement for all participants involved, yet few studies have examined CICO's effect on outcome variables such as academic performance and achievement Perhaps a contribution of the present study is the examination of CICO's effect on mathematics performance as measured by DCPM and EPM Mean DCPM for all participants increased over baseline when the CICO intervention was implemented Furthermore, EPM for all participants decreased from baseline when the CICO intervention was implemented These results seem to be congruent with the Hawken et al (2007) hypothesis that if implementing CICO results in students engaging in less problem behavior and spending less time in the school office, improved academic achievement may likely follow For all participants, FBA results indicated that the primary function of problem behaviors was adult attention CICO was designed to increase feedback and positive adult attention (Crone et al., 2004) By implementing an intervention designed to address the hypothesized function of the participant's problem behaviors, all participants engaged in less problem behavior and may have experienced ancillary academic benefits Another contribution of the present study is the addition the PND effect size measure Historically, single-subject researchers have not used statistics to support conclusions for intervention effectiveness (Derenne & Baron, 1999) Rather, single-subject researchers have Behavioral Disorders, 36 (4), 225-240 relied on strong internal validity of designs and use of visual analysis to document intervention effectiveness (Marascuilo & Busk, 1988; Parsonson & Baer, 1978) To date, no previously published study investigating the CICO intervention has employed a measure of effect size Recently, however, single-subject researchers have begun reporting effect sizes for singlesubject experiments (Olive & Smith, 2005) Indeed, the publication manual of the American Psychological Association (2009) suggests that all manuscripts submitted for publication include effect size calculations to facilitate interpretation of intervention outcomes Given this, PND all was calculated to help evaluate both the behavioral and academic outcomes With respect to the present investigation, the PND measure of effect size suggests that the CICO intervention was effective in reducing the participant's problems behaviors in terms of direct observations and ODRs The manner in which social validity was evaluated during the present investigation may also serve as a contribution Previous CICO studies (Hawken & Horner, 2003; Hawken et al., 2007; Todd et al, 2008) examined social validity using the BEP Acceptability Questionnaire (Hawken & Horner, 2003), which has no published psychometric properties Given the relative lack of documented psychometric properties, the present investigation employed the BIRS to assess social validity with both guidance counselors and the participant's classroom teachers The results revealed that both the guidance counselors and the teachers rated the CICO intervention as acceptable and effective Both groups also expressed some concern with the Time of Effect factor More specifically, the teachers and counselors rated disagree or slightly disagree in response to the statement, "The intervention would quickly improve the child's behavior." These results are congruent with those obtained by McCurdy, Kunsch, and Reibstein (2007), which used the Intervention Rating Profile and the Children's Intervention Rating Profile to measure perceptions of a CICO intervention Results also indicated teachers and students had strong satisfaction with the CICO intervention An additional positive outcome of the study was the direct behavioral observations that were implemented with a high degree of interobserver agreement Some previous studies (i.e., Hawken et al., 2007) have relied on ODRs as evidence of behavior change ConAugust 2011 / 237 current with the methodology of previous CICO researchers (Campbell & Anderson, 2008; Fairbanks et al., 2007; Hawken & Horner, 2003; Todd et al., 2008), the addition of independent observations lend support for the ODR findings, which provide two measures that demonstrate the intervention effects The high degree of treatment integrity as measured throughout all phases of the study may also be a strength of the current study Treatment integrity represents a critical component in evaluating the effects of an intervention Despite the importance of treatment integrity, the majority of researchers within the CICO literature have failed to examine it appropriately For example, two studies have explored the effectiveness of CICO programs that were already implemented in schools (Filter et al., 2007; Hawken, 2006; Hawken et al., 2007) Thus, treatment integrity could not be evaluated across all phases of CICO implementation Other studies (Todd et al., 2008) have failed to present any fidelity data, although they reported that the data had been collected Although this study has added to the research base, there are a few limitations that are worthy of discussion First, the study was conducted in one school district that has specific demographic, geographic, and ecological factors that limit the scope of the generalization of the results These results can be generalized only to a population of students similar to the ones that participated in the study Second, CICO's effects on mathematics may need to be interpreted with caution in that the results could be due to maturation alone Students have demonstrated improvement with exposure to the learning environment as demonstrated in several universal screening or benchmarking data sets However, during the weeks of CICO intervention, the participants exhibited fewer problem behaviors, making it highly likely that they spent more time in class and experienced fewer classroom disruptions This combination of more time in the learning environment coupled with fewer distractions may have led to increases in math engagement Third, the acceptability measures used in this study either have been used with CICO interventions but lack psychometric data or the measure has psychometric data but has no previous association with CICO Thus, the results pertaining to acceptability should be interpreted with caution However, the individual analysis has garnered interesting information about potential issues of acceptability with CICO 238 / August 2011 Fourth, pairing the participants in dyads with only two phases may raise threats to internal validity Future research may address this concern by employing a withdrawal design or a multiple baseline across three or more participants where a withdrawal is impractical Although dyads were used, the participants were in different classrooms, different classes of behaviors (i.e., problem behaviors versus academic performance) systematically changed with the implementation of CICO, and each student had a hypothesis that his or her respective problem behaviors were attention maintained Finally, because some data were collapsed into a weekly aggregate score, nuances in the individual data points cannot be evaluated However, this practice is not uncommon in behavioral research Collapsing data either into a weekly score or using medians can reduce any variability in data that can mislead or draw attention to an outlier that is not necessarily as important in the total picture In fact, by the authors using PND all as well as visual inspection, they have attempted to account for such variance and not make decisions about the data inappropriately With these limitations in mind, future research is warranted on CICO This study is promising but could be extended to examine whether CICO would have an effect on other academic measures including but not limited to reading, comprehension, other mathematics computation or application, or other areas In addition, the present study wanted to determine if there were any generalization to academic variables without directly intervening on mathematics, but programming generalization may have better effects than hoping for generalization Although there is a firm research base on CICO and its effects on behavior, more research is needed to examine the contributory value of each component in the CICO package Future studies could explore the components by conducting a component analysis or adding and removing components to determine the relational or causal effects on both behavioral and academic variables In addition, future research should also examine the effects of CICO with other target behaviors or response classes Finally, it may be interesting to examine the effects of CICO in school districts with various levels of PBS implementation (e.g., no implementation, low Behavioral Disorders, 36 (4), 225-240 levels of implementation, moderate levels of implementation, and high levels of implementation) In summary, all participants displayed fewer problem behaviors, as measured through direct observation and ODRs, when CICO was in place Furthermore, the decrease in problem behaviors may have contributed to minimal gains in basic math skills Although more research is needed, the CICO intervention appears to be an effective Tier intervention as it is continuously available, requires minimal staff effort, and allows for ongoing data collection and evaluation REEERENCES American Psychological Association (2009) Publication manual of the American Psychological Association (6th ed.) Washington, DC: Author Burns, M., VanDerHeyden, A., & Jiban, C (2006) Assessing the instructional level for mathematics: A comparison of methods School Psychology Review, 35, 401-418 Campbell, J M (2004) Statistical comparison of four effect sizes for single-subject designs Behavior Modification, 28, 234-246 Campbell, A., & Anderson, C (2008) Enhancing effects of check-in/check-out with function based support Behavioral Disorders, 33(4), 233-245 Carr, J E (2005) Recommendations for reporting multiple baseline designs across participants Behavioral Interventions, 20, 219-224 Crone, D A., Horner, R H., & Hawken, L (2004) Responding to problem behavior in the school: The behavior education program New York: Guilford Derenne, A., & Baron, A (1999) Huan sensitivity to reinforcement: A comment on Kollins, Newland, and Critchfield's (1997) quantitative literature review Behavior Analyst, 22, 35-41 Doggett, R A., Edwards, R P., Moore, J W., Tingstrom, D H., & Wylczynski, S M (2001) An approach to functional assessment in general education classroom settings School Psychology Review, 30, 313-328 Dufrene, B A., Doggett, R A., Henington, C, & Watson, T S (2007) Functional assessment and intervention for disruptive classroom behaviors in preschool and head start classrooms Journal of Behavioral Fducation, 16, 368-388 Edwards, R P (2002) A tutorial for using the functional assessment informant record for teachers Proven Practice, 4, 31-33 Elliott, S N (1988) Acceptability of behavioral treatments in educational settings In J C Witt, S N Elliott, & F M Gresham (Eds.), Handbook of behavior therapy in education (pp 121-150) New York: Plenum Behavioral Disorders, 36 (4), 225-240 Elliot, S N., & Treuting, M V (1991) The Behavior Intervention Rating Scale: Development and validation of a pretreatment acceptability and effectiveness measure Journal of School Psychology, 29, 43-51 Fairbanks, S., Sugai, C , Guardino, D., & Lathrop, M (2007) Response to intervention: Examining classroom behavior support in second grade Fxceptional Children, 73, 288-310 Filter, K J., McKenna, M K., Benedict, E A., Horner, R H., Todd, A W., & Watson, J (2007) Check in/check out: a post-hoc evaluation of an efficient, secondary-level targeted intervention for reducing problems behaviors in schools Fducation and Treatment of Children, 30, 69-84 Fleming, C B., Harachi, T W., Cortes, R C, Abbott, R D., & Catalano, R F (2004) Level and change in reading scores and attention problems during elementary school as predictors of problem behavior in middle school Journal of Fmotional and Behavioral Disorders, 12, 130-144 Hawken, L S (2006) School psychologists as leaders in the implementation of a targeted intervention School Psychology Quarterly, 27(1), 91-111 Hawken, L S., & Horner, R H (2003) Evaluation of a targeted intervention within a schoolwide system of behavior support Journal of Behavioral Fducation, 12, 225-240 Hawken, L S., MacLeod, K S., & Rawlings, L (2007) Effects of the behavior education program (BEP) of office discipline referrals of elementary school students Journal of Positive Behavior Interventions, 9, 94-101 Horner, R H., Sugai, G., Todd, A W., & LewisPalmer, T (2005) Schoolwide positive behavior support: An alternative approach to discipline in schools In L Bambara, & L Kern (Eds.), Individualized supports for students with problem behaviors: Designing positive behavior plans (pp 359-390) New York: Guilford Press Irvin, L K., Horner, R H., Ingram, K., Todd, A W., Sugai, G., Sampson, N K., & Boland, J B (2006) Using office discipline referral data for decision making about student behavior in elementary and middle schools: An empirical evaluation of validity Journal of Positive Behavior Interventions, 8, 10-23 Irvin, L K., Tobin, T J., Sprague, J R., Sugai, G., & Vincent, C G (2004) Validity of office discipline referral measures as indices of school-wide behavioral status and effects of school-wide behavioral interventions Journal of Positive Behavior Interventions, 6, 131-147 Lee, Y., Sugai, G., & Horner, R (1999) Effect of component skill instruction on math performance and on-task, problem, and off-task behavior of students with emotional and behavioral disorders Journal of Positive Behavior Interventions, 1, 195-204 Maguin, E., & Loeber, R (1995) Academic performance and delinquency In M Tonry (Ed.), Crime and justice: An annual review of research August 2011 / 239 (Vol 20, pp 145-264) Chicago, IL: University of Chicago Press Marascuilo, L A., & Busk, P L (1988) Combining statistics for multiple-baseline AB and replicated ABAB designs across subjects Behavioral Assessment, W, 1-28 March, R E., & Horner, R H (2002) Feasibility and contributions of functional behavior assessments in schools Journal of Emotional and Behavioral Disorders, 10, 158-170 McCurdy, B L., Kunsch, C, & Reibstein, S (2007) Tier II prevention in the urban school: Implementing the Behavior Education Program Preventing School Eailure, 51, 12-19 Mclntosh, K., Flannery, K., Sugai, C , Braun, D., & Cochrane, K (2008) Relationships between academics and problem behavior in the transition from middle school to high school Journal of Positive Behavior Interventions, 10, 243255 Mclntosh, K., Horner, R H., Chard, D J., Boland, J B., & Good, R H (2006) The use of reading and behavior screening measures to predict nonresponse to school-wide positive behavior support: A longitudinal analysis School Psychology Review, 35, 275-291 Morrison, G M., Anthony, S., Storino, M., & Dillon, C (2001) An examination of the disciplinary histories and the individual and educational characteristics of students who participate in an in-school suspension program Education and Treatment of Children, 24, 276-293 National Association of State Directors of Special Education (NASDE) (2005) Response to intervention: Policy considerations and implementation Alexandria, VA: NASDE Olive, M L., & Smith, B W (2005) Effect size calculations and single subject designs Educational Psychology, 25, 313-324 Parsonson, B S., & Baer, D M (1978) The analysis and presentation of graphic data In T R Kratochwill (Ed.), Single subject research: Strategies for evaluating change (pp 101-166) New York: Academic Pelham, J., Fabiano, G., & Massetti, G (2005) Evidence-based assessment of attention deficit hyperactivity disorder in children and adolescents Journal of Clinical Child & Adolescent Psychology, 34, 449-476 Roeser, R W., & Eccles, ) S (2000) Schooling and mental health In A J Sameroff, M Lewis & S M Miller (Eds.), Handbook of developmental psychopathology (2nd ed., pp 135-156) New York: Kluwer Academic/Plenum Scruggs, T E., & Mastropieri, M A (2002) On babies and bathwater: Addressing the problems 240 / August 2011 of identification of learning disabilities Learning Disability Quarterly 25, 155-168 Sugai, G., & Horner, R (2001) School climate and discipline: Coing to scale Paper presented at the National Summit on the Shared Implementation of IDEA, Washington, DC Tobin, T |., & Sugai, G M (1999) Using sixth-grade school records to predict school violence, chronic discipline problems, and high school outcomes Journal of Emotional and Behavioral Disorders, 7, 40-53 Todd, A W., Campbell, A L., Meyer, G G., & Horner, R H (2008) The effects of targeted interventions to reduce problem behaviors: Elementary school implementation of check incheck out Journal of Positive Behavior Interventions, 10, 46-55 Treuting, M., & Elliott, S N (1991) The Behavior Intervention Rating Scale: Development and validation of a pretreatment acceptability and effectiveness measure Journal of School Psychology, 29, 43-5^ Walker, H M., Ramsey, E., & Gresham, F M (2004) Antisocial behavior in school with infotrac: Evidence-based practices Belmont, CA: Wadsworth Walker, H M., & Shinn, M R (2002) Structuring school-based interventions to achieve integrated primary, secondary, and tertiary prevention goals for safe and effective schools In M R Shinn, H M Walker, & G Stoner (Eds.), Interventions for academic and behavior problems II: Preventive and remedial approaches (pp 1-25) Bethesda, MD: National Association of School Psychologists Wilkinson, L (1999) Statistical methods in psychology journals: Guidelines and explanations American Psychologist, 54, 594-604 AUTHORS' NOTE Address correspondence to Michael Mong, The University of Southern Mississippi, 730 E Beach Blvd, Long Beach, MS 39560; E-mail: michael.mong@usm.edu MANUSCRIPT Initial Acceptance: 2/27/11 Final Acceptance: 4/14/11 Behavioral Disorders, 36 (4), 225-240 Copyright of Behavioral Disorders is the property of Council for Children with Behavioral Disorders and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission However, users may print, download, or email articles for individual use  ... suspension program Education and Treatment of Children, 24, 276-293 National Association of State Directors of Special Education (NASDE) (2005) Response to intervention: Policy considerations and. .. the generalization of the results These results can be generalized only to a population of students similar to the ones that participated in the study Second, CICO's effects on mathematics may... component analysis or adding and removing components to determine the relational or causal effects on both behavioral and academic variables In addition, future research should also examine the effects