One very important element for increasing and maintaining attendance at SI sessions is the regular reporting of differences to the SI class between SI and non-SI attendees.  How these results are reported can have a significant affect on SI attendance.  The list below begins with the most influence on SI attendance to the lesser, though combining several of these can have a synergistic effect on SI session attendance:

1.    Instructor of the class with SI reports results in class.

2.    SI Leader reports results in class.

3.    SI supervisor reports results in class.

4.    Postings on classroom or hallway walls.

5.    Handouts distributed in class on differences between SI and Non-SI students in class.

          Below is an example of a test results report form you may use as is or modify to fit your data.  This kind of form makes reporting SI data much easier and more organized.

_____________________________________________________________________

                                                SUPPLEMENTAL INSTRUCTION

                                              TEST RESULTS REPORT FORM

Course/Sec/Prof _______________    SI Leader ________________ Test # 1 2 3 4 5 6

Date of test _________    Date these results announced in class __________________

I.  Comparative scores.

     Average test score for Non-SI attendees was____%

     Average test score for SI attendees was____ = ____% higher for SI attendees.

Average test grade for Non-SI attendees was ____.

Average test grade for SI attendees was ____. That is ____ letter grade(s) higher for SI attendees.

SI attendees averaged ____ points higher than non-attendees.

     Percent:                      ABC        vs.           DF grades.

                                ____ ____       ____ ____

                                       SI    Non                      SI    Non

               A         B         C         D         F

Number of SI attendees:                           ___       ___     ___     ___       ___

Number of non-SI attendees:                    ___       ___     ___     ___       ___

Percentage of SI attendees:                     ___%   ___%  ___%  ___%   ___%

Percentage of non-SI attendees:             ___%   ___%  ___%  ___%   ___%

Percentage amount higher for SI:            ___%   ___%  ___%  ___%   ___%

II.      Other Notable Data

III.          Test Questions

          ____% percent of the test questions and answers were covered in SI sessions.

IV.    Concluding Announcement

Please encourage students to attend SI sessions.

Return this completed form to the SI office after announcement is made.

_____________________________________________________________________

METHOD III - ANECDOTAL COMMENTS

From Students on How SI Helped Them

          This form of assessing the impact of SI may be in the form of live, in-class, testimony and discussion or printed in a handout and distributed in class.  It may also involve students who attended SI sessions from previous semesters.  Below are some examples of testimony on SI.  Some of this testimony was solicited in class and some gathered in SI sessions, placed on a handout, and distributed in class.

¨    The more time I spent in SI, the less time I needed to spend studying for my chemistry class.  I didn't think SI would help much at first.  Now, I see that I was wrong!

¨    SI helped me understand the material and come up with possible test questions.  This helped me prepare for my biology tests better.  I learned the answers before I took my exams because of what we covered in SI.

¨    SI covered the same material that appeared on my tests.  But what I liked best was that SI covered what I wanted and needed, when I needed it.

¨    SI helped me figure out a better system of studying.  I get better grades than I ever did before.

¨    I was able to understand glycolysis and the Krebs Cycle in SI and got them right on the test.  I didn't understand these the first time and missed these questions the first time I took this class.

From SI Faculty on How SI Helped Them

            This information may be gathered in face-to-face interviews or from a written survey.  Faculty testimony may also be in the form of live, in-class testimony or printed in a handout and passed out in class.

·           SI students ask more and better questions in class and this helps their class participation grade.  It also appeared to assist the shyer students who are typically fearful to speak up in class.

·           SI students seemed to have fewer problems understanding the more difficult concepts.

·           After the semester was over, I could clearly see that SI students earned higher test grades even among the students I knew didn't have high GPA's to begin with.

·           I have had SI in my psychology class for several semesters now and I see that students who go to SI sessions regularly get higher scores on quizzes and exams.  More importantly, they understand the material better.

·           SI students seem to understand the subject matter better, especially the traditionally more difficult concepts.

METHOD IV - INFERENTIAL STATISTICAL RESEARCH OF SI

          Good inferential statistical research is the easiest to defend among the various methods of reporting the impact of SI.  From inferential statistics, reliable conclusions may be drawn, trends, and impacts of SI may be ascertained in terms of higher grades, re-enrollment rates, graduation rates, etc.  Inferential statistics are particularly useful in cases where justification of the impact of SI is required or used as a basis to make decisions regarding the expansion of an SI program.  In these cases, SI supervisors will need data from assessment methods that are more commonly accepted as reliable for making predictions and drawing conclusions.  Toward that end, it is only inferential statistics that are rigorous enough to analyze and separate the effects of incoming variables (high school GPAs, high school class rank, SAT and ACT scores, existing college GPAs, etc., among other characteristics students bring with them to SI) and the effects of SI on outcome variables (final course grades, reenrollment rates, GPAs, etc.).  Furthermore, the voluntary attendance aspect of SI can be defended against the pesky self-selection bias, that is, a bias that suggests that only the better students attend SI.

          Below is one model for researching the impact of SI on final course grades (Congos & Schoeps, 1997).  This model may be modified to fit the needs of individual programs and available data.

          Step 1-Identify the relevant variables.  Identify the dependent (or outcome) variables, i.e., the measures of potential change influenced by SI.  Some of these variables could be:

1.    semester GPA

2.    re-enrollment rate in semesters following SI participation

3.    graduation rates

4.    course grade in the SI course

5.    individual test grades throughout the semester in the course with SI

6.    the differences between grade point average following SI participation when compared to a control group

          Identify the independent (or incoming) variables that students bring to the SI program to determine SI's impact:

1.    whether or not a person attended SI

2.    how many times a student attended SI sessions

3.    SAT scores (or ACT scores)

4.    high school class rank

5.    predicted grade point average (PGPA)

6.    current grade point average

7.    gender

8.    class

9.    age

10.     registration type (transfer, post-baccalaureate, part-time, etc.)

11.     race.

          Step 2 - For each student in the class, gather the data on the variables that you have chosen in step.  I

          Step 3 - Maintain records on the information needed for the chosen dependent variables such as the number of times each student attended SI sessions, exam scores, test grades and final grade in course, re-enrollment data, graduation date, GPA at graduation, etc.

          Step 4 - Enter the data into a computer in a format that facilitates analysis.  An example of one format to use is as follows:

              Record 1

              Column    Item

              1-9              Social Security Number

              11-31         Name (Last name first)

              33               Gender

                 35-36         Registration type

                 38-39         Birth year

                 41-43         SATV-verbal score on Scholastic Aptitude Test (SAT)

                 45-57         SATQ-quantitative score on SAT

                 49-52         High school class rank

                 54-57         High school class size

                 59-61         PGPA (predicted grade point average)

                 63-65         Number of credit hours accumulated

                 67-69         GPA (grade point average)

                 Record 2

                 Column         Item

                 1-11           Class, section, semester, year

                 33-35         Score for Test 1

                 37-38         Number of SI attendances

                 40-42         Score for Test 2

                 44-45         Number of SI attendances between tests I and 2

                 47-49         Score for Test 3

                 51-52         Number of SI attendances between tests 2 and 3

                 54-56         Score for Test 4

                 58-59         Number of SI attendances between tests 3 and 4

                 61-63         Score for final exam

                 65-66         Number of SI attendances between test 4 and final exam (FE)

                 68-70         Total score for the course

                 72               Final course grade

          Step 5 - Define what is an SI attendee.  Some programs require students to attend SI sessions anywhere from one to five times before s/he can be counted an SI attendee.

          Step 6 - Analyze the data using an appropriate data analysis software package.  One package to consider is the Statistical Analysis System (SAS).  For the remainder of the steps in this inferential research example below, the final grade in the course is used as the dependent variable.  For analyzing other dependent variables, a similar approach is taken.  However, good operational definitions of such variables as graduation rate do need to be developed.  For example, is a student considered graduated in four years, five years, six years?

          Some recoding and combining of variables needs to be done.  For example, the number of times a student attends SI sessions throughout the semester is recoded as (SITOT).  This number of attendances is used to divide the students into SI and a non-SI groupings based on how we define an SI attendee (attend I time, 3 times, etc.). Students are also grouped by grades into two categories in this example: ABC and DFW.  Withdrawals and Incompletes are treated like grades of F because often the vast majority of students who receive those grades did so to avoid an F in the course.  SATV and SATQ scores are added together to get SATC (combined SATV and SATQ).  CRANK (converted high school rank) is calculated by dividing high school class rank by high school class size.  This variable has lower values the better the student's rank in the high school class.

       The following tests are then performed: (a) chi-square tests, (b) t-tests, and

(c)     analysis of covariance.

          A.        Chi-square test of grade by SI vs. non-SI.  Using the data coded into SI VS. non-SI participants and using PROC FREQ and the options CHISQ, CELLCH12 and EXPECTED, get the contingency table, chi-square statistics, and contingency coefficient.  The chi-square and its observed significance level tell whether the two groups (SI and non-SI) differ significantly.  Any observed significance level less than the alpha set by the researcher indicates that there are significant differences between the SI and non-SI groups (a commonly used alpha is 0.05). The contingency coefficient is a correlation coefficient (for qualitative data) and gives a measure of the strength of relationship between SI participation and grade.

          B.        Independent t-tests compare SI vs. non-SI on SATV, SATQ, CRANK, PGPA (predicted grade point average), test scores, and final course grades.  The t-test compares the means for the two groups.  The t-tests for SATs, CRANK, and PGPA are used to check for self-selection bias.  If students in the SI group have higher SATs or PGPAs or lower CRANK, those students are likely to make better grades even if they did not attend SI sessions.  On the other hand, if their SATs, etc., are the same as the non-SI students, and yet the SI students make higher grades, the data then suggest that SI may be the reason.  In this example, SI students have statistically the same SATs, PGPAs, and the same CRANKs, and yet earn higher grades and test scores than the non-SI group.  In cases when SI students have lower SATs and PGPAs, even if that group shows no significant difference in final grades, the SI program is considered effective.  As with the chi-square test, the observed significance level (or p-value) should be compared with the set alpha (such as 0.05) to decide whether there are significant differences between groups.

          C.        Analysis of covariance.  To look at what ST contributes over and above the effect of SATC, an analysis of covariance (ANCOVA) is done.  This procedure tests the differences in grades between the SI and non-SI students, after the grades have been statistically adjusted for differences in incoming academic potential (e.g., SATC) and industriousness and motivation (e.g., PGPA).  In such an analysis, SATC or PGPA are the covariates.

          Using the least squares means option, the SI and non-SI group means are compared, having been adjusted for the effect of the covariate.  A significant difference in this case indicates that the SI group is different from the non-SI group above and beyond the effects attributable to academic potential (as measured by SATC) or the combination of academic potential, industriousness, and motivation (as measured by PGPA).  It is important to observe which group has the higher mean to see if it is the SI students who score higher.

          Step 7 - Present the results in a readable manner that includes relevant narratives necessary to explain and clarify the data.  The example below presents an analysis of all introductory biology classes with SI components from one semester.  While SI attendance was uncharacteristically lower than normal this semester (usually closer to 25-30%), the data are still reported to the administration.

          Table I shows the percentage of ABC vs DFW grades for the two groups: SI vs. non-SI.  The chi-square test of association shows significance beyond the 0.05 level, that is, at an OSL of 0.008. The contingency coefficient is 0.139, not a high correlation, but significant.  The direction of the difference is toward the effectiveness of SI, in that the percentage of ABC grades is greater for SI than for the non-SI group and the percentage of DFW grades is less for the SI group than for the non-SI group.

          In Table 2, which reports the results of the t-tests, the non-significant differences among SATs, CRANKs, and PGPAs are seen in the OSL values greater than 0.05. The significant difference in grades is seen in the OSL less than 0.05. Also, the significant difference in grades is in the direction of SI with higher average grades.

Table 1. Percentage of Grades by Group

Group                              ABC              DFW

Non-SI (n = 306)            62.75%         37.25%

SI (n = 46)                       82.61%         17.39%

TOTAL  65.34%             34.66%

Chi-square = 6.967, df = 1, OSL = 0.008, Contingency Coefficient = 0.1 39

Table 2.   Comparison of Means and t-Test Results^a

Group    SATV          SATQ         SATQ            CRANK    PGPA        Grade

Non-SI   422.65        467.22       889.33          0.233        2.28               1.94

               (76.07)        (75.92)       (130.30)        (0.150)      (0.32)          (1.19)

SI           419.47        451.84       871.32          0.163        2.27               2.48

               (73.67)        (65.47)       (105.94)        (0.944)      (0.31)          (1.07)

OSL       0.8099        0.2375       0.4174          0.9446      0.7725 0      .0035

^aNumber in parentheses are standard deviations.

          In order to further compare the SI vs. non-SI groups, two ANCOVAs are done with one using SATC as a covariate (in Table 3) and the other using PGPA as the covariate (in Table 4).  In Table 3 the significant F = 25.9, with OSL = 0.0001, indicates that SI attendance and SATC performance together contribute significantly to predicting GRADE in the course.  The R-square, i.e., the proportion of variability in grade that is explained by the variability in SI attendance and SATC, is not very high, but from previous research, it compares favorably; it is not easy to predict final course grade in a particular course from variables herein identified.  In fact, in large studies predicting GPA (not single course grade), where SATC does a reasonable job at predicting GPA, the R-square is not much higher than the one reported in this table.  The PGPA takes into account the SAT scores and the high school class standing.  Since PGPA can be interpreted to have elements of academic potential as well as aspects of industriousness and motivation, it is not surprising that it would be a letter predictor than SAT alone.  The results of the ANCOVA show that the mean grade of the SI group (2.57) is higher than the mean grade of the non-SI group (1.93) after adjusting for differences in SATC.  This result is a significant difference as can be seen from the OSL of 0.0006 in Table 3. The covariate PGPA gives similar results in Table 4.

Table 3. ANCOVA Table - Predicting Grade from SI and SATC

Dependent Variable: GRADE

               Source             df               55                   Ms            F         OSL

               Model                2               58.23              29.26    25.9  0.0001

               Error             280             316.39                1.13

               Total              282             374.9 1______________________                                              

               R-Square = 0. 1 56

Least Squares Means (Covariate = SATC)

                                    Grade Adjusted for

               Group                  Covariate             OSL

               Non- 1                    1.931              0.0006

               SI                            2.574

Table 4. ANCOVA Table - Predicting Grade from SI and PGPA

Dependent Variable: GRADE

               Source               df             55                Ms               F         OSL

               Model                  2          108.40            54.20       57.03 0.0001

               Error                272          258.51              0.95

               Total                274          366.91______________________                                            

               R-Square = 0.295

Least Squares Means (Covariate = PGPA)

                                    Grade Adjusted for

            Group                     Covariate              OSL

            Non-SI                      1.936                 0.0005

            SI                              2.546

            Tables 1, 3, and 4 show different total sample sizes.  The reason for this is that the SATs and PGPAs are not available for all of the students.  A researcher might decide to use only those students for whom all of the information is available; this is not done because too much good data would have been discarded.  For all of the tests, however, all of the statistics are calculated both the way they are presented here, and for the group that has all of the measurements.

          Tables 3 and 4 show that SI students had significantly higher grades in spite of the fact that the SATs, CRANKs, and PGPAs were not significantly different in the two groups.

          Step 8 - Draw a set of conclusions from your data, adapting them to the appropriate format for reports to the administration or for publication.  For example, a summary statement like that below may be made.

Conclusions

          The data strongly suggest that SI attendance has a positive impact on student academic performance.  Comparing SATs, PGPAs, and CRANKs, there appears to be no self-selection bias.  The SI attendees are, however, making higher grades than non-attendees with similar incoming characteristics.

COST EFFECTIVENESS OF SI: RE-ENROLLMENT AND GRADUATION RATES AND RETAINED INCOME

          In the absence of time and resources to do long-term research, data may be reported from the University of Missouri at Kansas City's twenty-five years of research on the impact of SI on colleges and universities.  The conclusions from this research may also be used to project the re-enrollment and graduation rates at your institution.  Using the example below, the impact of SI may be converted into retained income figures from students otherwise lost to attrition (National Center for Supplemental Instruction, 1997.

       Table 5. Re-enrollment and Graduation Rates at the University of Missouri at Kansas City from 1989 to 1995

Term that SI                                       Re-enrollment                        Graduation and

  is Offered                Group               Percentage              Re-enrollment Percentage

Fall 1989                    SI                     65.3%*                                           73.1 %*

                                    Non-SI            56.7%*                                           61.7%*

Fall 1990                    SI                     70.1 %*                                          76.00/o*

                                    Non-SI            58.3%*                                           66.4%*

Fall 1991                    SI                     70.6%*                                           75.4%*

                                    Non-SI            63.6%*                                           68.9%*

Fall 1992                    SI                     70.6%*                                           79.2%*

                                    Non-Is             53.6%*                                           62.3%*

Fall 1993                    SI                     73.4%*                                           78.6%*

                                    Non-SI            55.3%*                                           63.5%*

Fall 1994                    SI                     72.4%*                                           76.7%*

                                    Non-SI            60.8%*                                           65.9%*

Fall 1995                    SI                     74.5%*                                           80.00%*

                                    Non-SI            58.2%*                                           65.50%*

*Observed level of statistical significance of difference at or above .05 using a chi-square test.

          Table 5 shows that approximately 10 percent more of SI participants graduate than non-SI participants.  To translate this into retained income otherwise lost to attrition, we need to begin with tuition costs.  For the sake of simplifying this example, we will not include bookstore income or income from room and board in residence halls.

          If there is a tuition of $3000 per year at Old Ivy College who is thinking about starting an SI program, how may SI be presented to combat the common statement from administrators, "Where will the money come from for SI?"  One way to persuade Old Ivy administrators that SI is a wise and cost effective investment, is to look at the potential retained income.  Let’s assume that only fifty students from the freshman class attend SI each semester for the first year (in reality many upper classpersons attend SI sessions, also).  Let’s also assume that consistent with the research on SI and re-enrollment rates, that 10 percent more of these freshmen reenroll in the next semester (Arendale, 1997).  Let's also assume that SI expenses total $30,000 ($25,000 for an SI supervisor and $5,000 for SI Leaders).  SI has no retained income by the end of the first year and a cost of $30,000 (100 SI attendees times 10% = 10 retainees times $3000 yields $30,000).

          By the end of the second year, there are ten more sophomores retained because they attended SI sessions.  The total retained income for the second year of SI is $30,000.  By the end of the second year, SI has broken even with expenditures and retained income at Old Ivy.

          By the end of the third year of SI, there are ten more sophomores and ten more juniors because of SI.  The total retained income is $60,000 in the pipeline.  Now, there is a net gain of $30,000 beyond the cost of Old Ivy's SI program.

          The end of the fourth year of SI brings a total retained income of $90,000 attributable to SI's retention of ten more seniors, juniors, and sophomores, respectively.  That is $60,000 above the expenditures for SI (see Table 6).  This total figure may be low because in any year, not only freshmen attend SI sessions and because we are using a low figure of only ten retainees per year.  For example, during the first year, a number of upper class students will attend SI and 10 percent more of them will also be retained.

Table 6. Annual Retained Income from SI if Ten Freshmen Persist Until Graduation at Old Ivy

                                                1st Year          2nd Year          3rd Year         4th Year

                                                of SI                of SI                  of SI                of SI___

Annual SI costs                    $30,000          $30,000           $30,000          $30,000

Annual retained                     $0                    $30,000           $60,000          $90,000

income from ten

persisting freshmen

Annual accumulated -$30,000                    $0                     $30,000          $60,000

retained income after

subtracting annual

SI costs

          How many students does SI have to retain to make money for your institution in retained income?  What happens to the retained income figure if your bookstore and room and board expenditures are also included?  What is the retained income for your institution if more than 100 students attend SI sessions per academic year?

          It is easy to see that the financial implications for private institutions are much more significant because of higher tuition rates.  Some of the private institutions will have to retain as few as two students to exceed the costs of an SI program.

          The benefits of SI go beyond retention income.  There are benefits in the faster mainstreaming of students from differing ethnic backgrounds, more rapid mainstreaming of minority students, easing the transition of foreign students, higher grade point averages, helping students with learning disabilities succeed in college, higher final course grades (in classes with SI), higher GPAs, and facilitating retention related to the important affective connection to an institution (Ashwin, 1993; National Center for Supplemental Instruction, 1997; O'Donnell, 1996; Visor et al., 1992).

          It is not difficult to surmise that alumni of Old Ivy's SI program who are happy with the quality of the educational experience at Old Ivy would think more generously toward the annual giving, capital fund drives, and institutional development activities.  It also seems likely that satisfied students and graduates will communicate their satisfaction to prospective students and ease expensive recruiting costs.  More research needs to be done in this latter area to assess the impact of SI.