A National Study of Undergraduate Science Courses: Research-Based - - PDF document

Determining the Impact of Reformed Undergraduate Science Courses on Students

1

A National Study of Undergraduate Science Courses: Research-Based Evidence for Determining the Impact of Reformed College Science Courses on Students*

Dennis W. Sunal, The University of Alabama Cynthia Szymanski Sunal, The University of Alabama Dean Zollman, Kansas State University Cheryl L. Mason, San Diego State University Cheryl Sundberg, The University of Alabama Glenda Ogletree, The University of Alabama Abstract Evaluating undergraduate science courses and the extent to which they are standards-based reform-oriented requires research knowledge and skills in several

• domains. A review of research was described first, focusing on research-based

evidence supporting results, instrumentation, and data collection and analysis procedures used in determining the impact of college science courses on students. The presentation outlines a research design for a national study of undergraduate science courses including sampling from a population of 103 institutions, planning of a research design process, selecting and developing instrumentation, gathering and analyzing data, and interpreting results. The elements were developed to investigate formative impact by 1) using critical variables from the learning environment, course structure, department culture, and college instructor and 2) outlining innovative procedures for gathering data during the course and summative impact by gathering short and long term data from students during the course and on special student populations of graduated students. Multiple quantitative and qualitative instruments were described, to be analyzed using comparative and relational studies at multiple points in this impact design

• model. Data will also be used to develop criteria to identify differing levels of

implementation of standards-based reform characteristics in courses that are important in the development of meaningful science learning outcomes in all college students. Conclusions drawn focus on research based evidence of short- term impact on all undergraduate students and long-term effects on elementary education majors. *Research Paper Presentation at the Society of College Science Teachers (SCST) annual conference, March 29 – April 1 2007 [3:30-4:30 pm Thursday March 29] held in St. Louis, MO.

Determining the Impact of Reformed Undergraduate Science Courses on Students

2

A National Study of Undergraduate Science Courses: Research-Based Evidence for Determining the Impact of Reformed College Science Courses on Students

Numerous reports document concerns about teaching science, technology, engineering, and mathematics (STEM) in K-12 and in higher education. These concerns, expressed in documents beginning with Nation at Risk (1983), led to new standards such as National Science Education Standards (NSES) (1996), Benchmarks for Science Literacy, Project 2061(AAAS) (1993), Shaping the Future (NSF) (1996), Educating Teachers of Science, Mathematics and Technology (NRC) (2001), College Pathways to the Science Education Standards (2001) and No Child Left Behind (2001). New standards provide criteria that can be used to judge whether particular teacher and student actions serve the vision of a scientifically literate society. They bring coordination, consistency, and coherence to the improvement of science teaching (National Research Council [NRC], 1996). In 2006 a multiyear research project focusing on the impact of undergraduate science courses using criteria from the national reports and standards was funded by the National Science Foundation. The national study efforts can be found documented on the Internet web site titled National Study of Education in Undergraduate Science (NSEUS) at http://nseus.org. The goal of the national study is to investigate the short term impact on undergraduate students of undergraduate standards-based, reform entry-level science courses developed by faculty in a national professional development model and the long term impact on critical needs in the preparation and development of pre-service undergraduate K - 6 teachers of science. The overall goal of this research paper is to describe procedures and examples for developing a formative and summative research design to provide research-based evidence for determining the impact of college science courses on students. To accomplish this goal, two major areas will be discussed. 1) Summarization of the results of previous research on higher education reform in undergraduate science courses and their short- and long-term impacts. 2) Description of formative and summative research data forming the basis for a research model and design for a national study designed to determine the short- and long-term impacts of level of reform in undergraduate science teaching and learning

• n students, with special emphasis on elementary education majors.

Problem Effective K - 6 science education reform cannot be separated from standards- based reform in undergraduate science (Lederman & Gess-Newsom, 1999). To address standards-based reform recommendations, effective undergraduate STEM courses must

• ffer course experiences that involve connections between scientific ideas; provide

social, historical, and philosophical context; include meaningful laboratories; teach for

Determining the Impact of Reformed Undergraduate Science Courses on Students

3 inquiry; involve diverse learners; and include important unifying themes (Floden, Gallagher, Wong & Roseman, 1995, Lederman & Gess-Newsome; Sunal, et al., 2001). Examination of Previous Research Analysis of available literature on reform in undergraduate science courses has indicated a paucity of research on the topic. Of the literature reviewed from 1999 to the present (N = 79 articles, books, reports, or position statements), none of the studies could be described as experimental. The literature examined was categorized as: (a) program descriptions and general references, (b) case studies, (c) inquiry-based pedagogy, (d) surveys, and (e) achievement. The literature categorized as program descriptions and general references (47%, N = 37), centered on program description or was of general interest to researchers involved in the reform effort. There were 20 case studies (25%). The case studies evaluated a program and/or reform course(s). Inquiry-based pedagogy was a theme in 10% (N=8) of the studies reviewed. In some of the studies categorized as inquiry-based pedagogy, classroom observations were conducted to ascertain programmatic success reform programs and/or courses. Other studies focused on instrument development and validation, primarily designed to determine program

• success. Survey data was analyzed in eight studies (10%); professors and/or students were

surveyed about their experiences with reform programs and/or courses. Six studies considered achievement. Of the 79 studies examined, 21 studies were selected for further review based on the following criteria: (1) the research was quasi-experimental, and/or (2) the study was an analysis of case studies on one of the emergent themes found in the examination of previous research literature on the reform effort: case studies, inquiry-based pedagogy, surveys, and achievement (with the exception of the literature categorized as program descriptions and general references.) Case Studies Three of the case studies considered several themes found in several research

• studies. In a meta-ethnography, Blackwell (2002) synthesized the reports from nine

projects evaluating the impact of partnerships on teacher education and teacher induction (funded by the Office for Educational Research and Development, U. S. Department of Education), Blackwell (2002) concluded the pre-service teachers participating in the partnership programs were more successful than were pre-service teachers trained by traditional methods Additionally, the programs offered a mechanism for increased collaboration between higher education institutions and state departments, alleviating a typical barrier to reform; shared vision for teacher development and induction. Darling- Hammond (2000) also reviewed prior studies. These were case studies with results from the National Assessment of Educational Progress (NAEP) testing program, the 1993- 1994 Schools and Staffing Surveys, and data from a 50-state policy survey. From her analysis of the research base, she concluded that it is possible for the effects of well- prepared teachers (those with greater subject knowledge, pedagogical knowledge, and years of teaching experience) on student achievement to be more influential than student background factors including poverty, minority status, and language background. In a

Determining the Impact of Reformed Undergraduate Science Courses on Students

4 related study of successful teacher education programs, Darling-Hammond (2006) described successful teacher education programs at Alverno College, Bank Street College

• f Education, Trinity University, University of California at Berkeley, University of

Southern Maine, University of Virginia and Wheelock College. The remaining case studies involved in-depth review of a project or a specific course. ( Krockover, Shepardson, Adams, Eichinger, and Nakhleh (2002) reported on a partnership program, CABR, among scientists, science educators, master teachers, graduate students, and undergraduate students that resulted in undergraduate science courses which utilized more inquiry-based collaborative teaching pedagogy than previously found in such courses. Part of the process involved an evaluation of the reform courses by employing a case study strategy. Due to their participation in the CABR project, members of the team were “able to (a) implement more inquiry-based teaching that emphasized conceptual understanding; (b) provide opportunities for cooperative learning experiences, (c) use models as an ongoing theme, (d) link concepts and models to real-world situations, (e) provide a more diverse range of assessment strategies, and (f) have students present their understandings in a variety of different forms” (p.274). In addition, by using a collaborative format, graduate and undergraduate students, classroom teachers, scientists, and science educators were able to come together with differing perspectives to reform education. As a result of three case studies conducted by collaborative action-based research teams (each team was tasked with evaluation of the reform courses: Biology for Elementary Teachers, General Chemistry, and Earth Science for Elementary Teachers), the researchers concluded: lasting reform is the result of shared, team responsibility; administrative support is necessary for reform; and focus on a single aspect of the reform course in order to improve and extend the reform effort. In a case study of an inquiry-based physics course, Hubbard and Abell (2005) compared the beliefs about science teaching in an elementary science methods course between students who were completing or had completed a reform physics course and those students who did not take the course. The case study involved six of the 40 students enrolled in the methods course (two who had not taken the course, two concurrently enrolled, and two who had completed the course.) Primary data sources included a research-designed belief questionnaire administered during the second week of the course and key student written products from the methods course. Additionally, the methods course instructor‟s anecdotal records of class discussions were analyzed. As a result of the data analysis, Hubbard and Abell concluded that students completing the inquiry- based physics course expressed a more complete understanding of science teaching and learning compared to their peers and a greater ability to apply an inquiry approach to their lesson plans. Inquiry-Based Pedagogy Central to the reform effort is the effort to increase the use of student-centered pedagogy in undergraduate science courses. Judson and Sawada (2001) sought to determine the impact of the Arizona Collaborative for Excellent in the Preparation of Teachers (ACEPT). Eighty-six classroom observations of middle and high school teachers with one to three years of teaching experience were evaluated using the Reformed Teacher Observation Protocol (RTOP). Of these teachers, 53 had taken at

Determining the Impact of Reformed Undergraduate Science Courses on Students

5 least one ACEPT course and 33 were non-ACEPT teachers. Observations were both announced and unannounced. Analysis showed no significant difference between the two types of observations. The seven evaluators were not aware of the institution which the pre-service teacher had attended (i.e. which institutions participated in ACEPT and which did not). Based on average RTOP scores, teachers taking courses designed using the tenets of the ACEPT program showed a significantly higher level of reformed instruction than non-ACEPT teachers (p < 0.05). In addition, teachers who had taken two or more ACEPT courses were found to have a higher mean RTOP score than teachers with only

• ne ACEPT course. While both ACEPT-influenced and non-ACEPT teachers showed

increasing mean RTOP scores with years of experience, this increase was only found to be statistically significant for the ACEPT-influenced teachers (among the subsections of the RTOP, p = 0.005 - 0.042). In another study of the ACEPT program, Lawson et al. (2002) evaluated the effect

• f summer workshops on teachers‟ use of reformed teaching methods as well as the

effects of these teaching methods on student achievement in undergraduate science and mathematics courses. Evaluative data were collected for five courses (two from Physics, two from Biology, and one from Mathematics). Three of which were designed specifically for pre-service teachers. In addition, one course was designed specifically to teach pre-service biology teachers about reformed methods of teaching by using those reformed methods. Teaching practices were evaluated in all courses except the biology teaching methods course, using the RTOP. Student achievement was measured in each course with a different unnamed pre- and post-test, depending on the subject being

• taught. Students who participated in a course on mechanics were evaluated with the Force

Concept Inventory (FCI). For each course with RTOP data, the data from experimental sections were compared to data from at least one control section. In all four courses with RTOP data, instructors‟ mean RTOP scores strongly correlated with normalized student gains (r = 0.88, p < 0.05; r = .92, p < 0.001; r = 0.70, p < 0.05; r = 0.97, p < 0.01). In addition, Lawson et al presented preliminary data on the effectiveness of new teachers who had recently graduated. These data showed that first-year teachers who enrolled in

• ne or more ACEPT-influenced undergraduate science or math courses had a

significantly higher (p = 0.05) mean RTOP score of 48 (N = 20) than first year teachers who had not participated in an ACEPT course of 40 (N = 8). Researchers found similar data for second- and third-year teachers (mean RTOP score of 62 vs. 45, p < 0.05). In similar research, Adamson et al (2003) evaluated the effects the ACEPT Program on in-service junior and senior high school teachers‟ instruction practices, as well as the subsequent effects on these teachers‟ students as the result of taking college science courses designed with the ACEPT guidelines. Twenty-eight in-service teachers (14 ACEPT-influenced and 14 non-ACEPT-influenced) were observed using the RTOP. In addition, the achievement of students (n = 1116) from a subset of 15 high school biology classes was measured using the Biology Attitude Skills and Knowledge Survey (BASKS) during the last three weeks of the academic year. School SES (percentage of students receiving subsidized lunch), years of teaching experience, and “Regular” vs. “Honors” designations of biology classes were used as covariates. RTOP scores were found to be significantly higher for ACEPT-influenced teachers than for non-ACEPT- influenced teachers (F2,26 = 3.44, p < 0.05 for all teachers; F2,13 = 3.68, p < 0.05 for biology teachers). Achievement scores of students with ACEPT-influenced teachers were

Determining the Impact of Reformed Undergraduate Science Courses on Students

6 also significantly higher than were those of students with non-ACEPT influenced teachers (F2,13 = 6.23, p= 0.01). Results of all three studies indicated that program participants were more likely to use reform-based instructional techniques than were their counterparts who did not participate in the professional development. Additionally, results from the research of Judson and Sawada (2001) indicated student achievement was higher when the teacher had taken courses designed with the ACEPT design protocols as did Lawson, et al (2002) who reported achievement of pre-service elementary teachers in undergraduate science courses. The research base reveals the attitudes of pre-service and in-service teachers toward science and science education impacts the use of reform pedagogy in the K-12 classroom and K-12 student achievement. The change in the pedagogical techniques and beliefs of university science professors emerges from the research as well. In a case study, Wainwright, et al (2004) reported on the utilization of inquiry-based pedagogy in science content courses taught by twelve faculty fellows at five higher education institutions in Oregon. All twelve-faculty members had participated in summer institutes of the Oregon Collaborative for Excellence in the Preparation of Teachers (OCEPT) that modeled reform-based practices and fostered reflection on current issues in science, mathematics, and technological literacy for K-16 teaching. The faculty fellows were observed at least three times with the OCEPT-Teacher Observation Protocol (O-TOP). Following observations, each faculty member was individually interviewed in order to validate observational data and to add an in-depth description of the instructor‟s perspective. Six of the faculty fellows were from the science disciplines and six were from mathematics. Results indicated all faculty fellows used inquiry-based techniques; OTOP provides a profile of instruction, not an

• verall rating of inquiry pedagogy. Science faculty tended to make more interconnections

with other disciplines while mathematics faculty was more likely to promote student discourse and collaboration in small groups. In a mixed method design, Ballone-Duran, Czerniak, and Haney (2005), examined the change in beliefs and practices of teaching and learning of scientists as a result of their participation in the Toledo Area Partners in Education Support Teachers as Resources for Improving Elementary Science (TAPESTRIES) project. In the TAPESTRIES project, science faculty participated in professional development and collaborated with science educators and K-6 teachers. The study summarized results from interviews of the scientists and student surveys of the scientists‟ classrooms with the Classroom Learning Environment Survey (CLES) (Taylor, Fraser, and White, 1994). The scientists reported the courses in undergraduate science did not adequately prepare pre- service elementary teachers. Some scientists indicated elementary science students should take additional science courses and other scientists indicated undergraduate courses in science should be restructured to better meet the needs of pre-service elementary

• teachers. Four themes emerged from the interviews: (a) pre-service teachers must have

positive dispositions toward teaching and learning; (b) scientists made pedagogical and curricular changes as a result of collaborating with educators; (c) scientists began to think reflectively about their own teaching and learning; and (d) new collaborative projects emerged between scientists and educators as a result of the program. Additionally, the scientist expressed a greater understanding of the complexity of the reform process and indicated that they knew there must be more reform in undergraduate science. Results

Determining the Impact of Reformed Undergraduate Science Courses on Students

7 from administration of the CLES instrument to the students of 11 scientists involved in the study from the two universities (N = 260) indicated the scientists were beginning to incorporate reform-based changes in their courses as a result of the program. Ballone- Duran, Czerniak, and Haney delineated the following implications of the research: (a) faculty professional development facilitates change in higher education; (b) it takes time to plan, develop, and implement change; (c) personal belief systems are a barrier to reform; (d) support is necessary to implement change; and (e) collaboration is required for reform in teacher education. Surveys In related survey research, Staples (2004) compared student perceptions in a reform course (n = 55) with a traditional science course (n = 192). Quantitative and qualitative data was collected through surveys, and oral and written interviews measuring undergraduate students‟ preference of learning environment for science content, science learning, and teaching. Undergraduate education majors were also given pre- and post- test CLES surveys. Students reported teaching techniques that helped them make connections between science concepts and the real world assisted in understanding the concepts. One of the difficulties with the reform process is the development and validation

• f instrumentation designed to measure programmatic success. McGinnis, Kramer,

Shama, Graeber, Parker, and Watanabe, (2002) described the development and validity of the Attitudes and Beliefs about the Nature of and the Teaching of Mathematics and Science survey designed for the Maryland Collaborative for Teacher Preparation (MCTP), a statewide, standards-based project. This research compared teachers participating in the MCTP program with their counterparts in terms of attitudes and beliefs about science and mathematics before and after taking a reform-based course. The longitudinal study was conducted over a 2.5-year period. Factor analysis of the results indicated the first 32 items of the questionnaire should be split into 5 factors. These sections of the instrument were verified by factor analysis: “beliefs about mathematics and science (α = .76), attitudes toward mathematics and science (α = .81), beliefs about teaching mathematics and science (α = .69), attitudes toward using technology to teach mathematics and science (α = .80), and attitudes toward teaching mathematics and science (α = .60)” (p. 714). The pilot instrument was administered to 200 students in the fall of 1994 and 210 students enrolled in MCTP courses in the spring of 1995.After the pilot administration, the original 51-item questionnaire was adapted to the 37-item instrument included in the report. As a result of analyzing the data of the survey instrument, the researchers recommended using surveys that were related either to science or to mathematics. The research revealed no statistical difference between pre/post administrations of the survey of teachers taking MCTP courses. There was a statistically significant difference between the pre/post survey results of teachers who did not take the MCTP courses; after taking traditional science courses, the attitudes of teachers toward science, mathematics, and technology decreased. When the survey was administered to teachers in the MCTP program after 2.5 years, however, there was a positive significant difference.

Determining the Impact of Reformed Undergraduate Science Courses on Students

8 In a follow-up study, McGinnis, et al (2003) reported that when MCTP pre- service teachers surveys were compared to surveys of non-MCTP students (33 content and pedagogy courses distributed statewide, N = 486), in general, MCTP students started with attitudes and beliefs more in line with program goals than did non-MCTP students, and the gap between the two groups widened by the end of the course. In addition, that gap widened in some cases because MCTP students advanced beyond non-MCTP, but in

• ther cases it widened because non-MCTP students‟ scores actually declined between the

beginning and end of the course. Researchers also found that MCTP candidates‟ attitudes and beliefs changed favorably over time as they participated in the program (from Fall 1995 to Fall 1997), and that this change was significant (p < 0.001). From surveys administered over a three-year period (N = 68), researchers found that MCTP graduates differed significantly (p < 0.05) from a national sample of surveys (N = 478) on beliefs about the nature and teaching of science. “Specifically, they were less likely to believe: that science is a primarily formal way of representing the real world; that science is primarily a practical and structured guide for addressing real situations; that a liking for and understanding of students are essential for teaching science; that it is important for teachers to give students prescriptive and sequential directions for science experiments; and, that students see a science task as the same task when it is represented in two different ways. However, they were more likely to believe that if students get into debates in class about ideas or procedures covering the sciences, it can harm their learning” (p.21-22). In a similar study on pre-service teacher attitudes toward science, Le and Krapfl (2002) investigated the impact of an inquiry-based elementary science minor program of 25 semester hours on pedagogical content knowledge. Semi-structured personal interviews with nine program graduates in their 2nd - 6th year of teaching were analyzed

• qualitatively. Surveys were sent to program graduates (N = 34). Respondents reported:

(a) positive feelings about the experience and belief that the experience made a difference in how they currently teach science, (b) expressed concern about their ability to meet the requirements of special needs students, constraints of time, money, and materials, classroom management, and the reality of required curriculum in using inquiry-based teaching methodology, (c) articulated a positive self-efficacy to teach science, (d) expressed feelings of teaching “differently” from many of their peers, and (e) described typical barriers (lack of materials, lack of planning time, etc) that they had to overcome to use inquiry-based teaching methodology. Lee and Krapfl concluded that the program achieved its goals, but noted a classroom management component should be added and that the current component dealing with special needs students should be extended. Weld and Funk (2005) described similar results in a study of an inquiry-based biology course specifically designed for elementary education majors and modeled on the National Science Education Standards (NRC, 1996.) The study involved research into students (N = 61) self-perceived effectiveness as a teacher of biology. The four specific attitude factors tested in this study were: subjects‟ self-perceived change in command of biology subject matter; biology curriculum development competence; biology education pedagogical skill, and change in self-perceived effectiveness as a biology teacher. Data were collected pre- and post-course for all subjects using a survey of attitudes and self- perceived effectives as a teacher of biology. In addition, 10% of all enrollees participated in semi-structured telephone interviews immediately following pre- and post-testing, and

Determining the Impact of Reformed Undergraduate Science Courses on Students

9 data were supported by informal observations. Gains were found in pre-service teacher efficacy in teaching biology due to self-reported gains in pedagogical content knowledge. In addition, interviews revealed an increased interest in taking more science courses as a result of taking the reform biology course. Achievement One central goal in the position statements of leaders in science, mathematics, engineering, and technology is increased student achievement. There is a paucity of substantive research on the achievement of students involved in reform courses. Therefore, additional research should focus on achievement. In particular, future studies should be at least quasi-experimental in design, preferably experimental. In terms of research in student achievement in the studies reviewed, two studies of participants in reform courses in physics found that students taking reform courses had higher gains in conceptions of mechanics. Hake (1998) compared the effects of reformed versus traditional teaching practices on high school, college, and university students‟ conceptions of mechanics. The Force Concept Inventory (FCI) was administered pre- and post-course for 6542 students in 62 introductory physics courses. Students who participated in reformed courses (N = 4458) were found to have significantly greater normalized average gains (0.48 ± 0.14) than those who participated in traditional courses (N = 2084) (0.23 ± 0.04). In a similar longitudinal study, Francis et al. (1998) set out to determine whether gains in students‟ Force Concept Inventory (FCI) scores seen immediately after taking a reformed physics class would remain up to four years after completing the course. That is, “Do students retain their „Newtonian ideas‟ of the world years after finishing a reformed physics course?” The researchers concluded from this data that to a large extent, the students retained their “Newtonian ideas” of physics over the several years following instruction, supporting a fundamental shift in students‟ conceptual framework as a result of the course. Other research supports the hypothesis that pre-service teachers taking reform courses have higher achievement in undergraduate science courses. Luera and Otto (2005) described the impact of a series of undergraduate science courses utilizing inquiry-based pedagogy in life, physical, and earth-planetary sciences and a capstone course requiring pre-service elementary teachers to engage in deep exploration of one of the big ideas in science. They compared content knowledge between students who completed the reform inquiry-based science courses and those who completed the previous traditional series of courses. A least squared difference (LSD) t test with an alpha value of .05 showed that students who completed two inquiry content courses (M = 43.67) scored significantly higher in overall content knowledge than students who completed zero (M = 40.34) or one inquiry course (M = 40.44). Results of the Science Teaching Efficacy Behavior Instrument Version B, STEBI B, (Enochs & Riggs, 1990) indicated that at least one inquiry-based content course was required to see a change in efficacy: three inquiry courses (M = 56.65, n = 21), two inquiry courses (M = 54.27), one inquiry course (M = 53.26) [n = 145 for students completing one and two inquiry courses] and students who completed zero inquiry courses (M = 51.80, n = 119). While Lawson, et al (2002) and Adamson, et al (2003) concluded reform courses positively impact achievement among K-12 students and undergraduate students in

Determining the Impact of Reformed Undergraduate Science Courses on Students

10 science courses, the research conducted by Le, et al (2006) indicated that how achievement is assessed is problematic in evaluation of reform. Typically, multiple choice assessments are used rather than alternative assessment. Le, et al (2006) reported

• n Mosaic II, a longitudinal study called of the impact of reform in mathematics and

science education on student achievement. Students in five cohorts (3 mathematics and 2 science) from three districts which had recently concluded participation in the Local Systemic Change Program, were followed for three years to measure student achievement

• ver long exposure to reform education. Student age varied from grades 3-8 and

achievement was measured using the mathematics or science component of the Stanford Achievement Test Series, Ninth Edition (SAT-9) for all cohorts, as well as the open- ended version of the SAT-9 for a sub-sample of three cohorts. In addition, each year all participating teachers completed a survey, filled out classroom logs, and responded to a set of vignette-based questions about instructional practices. In selected years, classroom

• bservations and interviews were conducted with a subset of teachers. The research

indicated little impact on student achievement as measured by multiple choice tests, but positive achievement when other measures were used such as open response tests. Positive relationships tended to become stronger with sustained exposure to reform

• teaching. The researchers concluded that the way in which achievement is measured

might have a large effect on the observed relationship between reform-oriented instruction and achievement. Additionally, the research revealed that teacher competency and efficacy might have impacted success with reform-based pedagogy. Teachers reported that, despite the training they received, they believed that reform-oriented practices were likely to be less effective than traditional practices for promoting high scores on state accountability tests. In summary, the research base supports the use of inquiry-based instruction to increase student achievement in science in K-12 classroom and undergraduate students. Analysis of the studies cited indicates pre-service teachers were more likely to use inquiry-based instruction following reform based science courses and the process of reform positively impacted the use of reform pedagogy in undergraduate science courses. Reports of Reform Course Impacts on How University Faculty View Science and Science Teaching The literature also indicates the reform effort may impact how university science instructors view science and science teaching. In a qualitative study, Fedock, Zambo, and Cobern (1996) described the impact of developing and delivering a summer professional development workshop on the changes in pedagogical understanding of university science professors. Data collection utilized structured interviews of four science professors as science educators as a part of a summer life science academy for K-12

• teachers. The structured interviews centered on the professors‟ view of science and

science education, along with their perceptions of the K-12 teachers. The interviews were taped, transcribed, and coded for themes. The emerging themes were: “(a) the views of the professors prior to the academy; (b) the science professors‟ preparation for the academy; (c) occurrences during the academy; and (d) changes of thought experienced by the professors.” (p. 9)

Determining the Impact of Reformed Undergraduate Science Courses on Students

11 As reported by the professors, it appeared the professors viewed the academy as way to gain greater understanding of the nature of K-12 science education (Fedock, Zambo, & Cobern, 1996.) The professors expressed a belief that “content-driven science can be uninteresting and ineffective” (p. 12). During the interviews, the professors expressed openness toward trying new pedagogical techniques. As a result of interaction with mentor teachers in planning instruction for the academy, the professors selected more student-centered instructional techniques like the learning cycle and cooperative learning as opposed to more content-driven techniques. The professors reported they were thinking about how to incorporate the learner-centered instructional techniques like the learning cycle in their regular college courses. Finally, at the end of the academy, comments by the professors indicated the shift in view of science and science teaching had broadened. In particular, the use of the learning cycle was cited as one technique the professors reported they would use in their traditional college courses. The authors noted the scientists examined their own teaching techniques and made changes in their instructional techniques to better meet the needs of their students. However, in a study by Bachkus and Thompson (2006), the reform effort was reported as not having made as much progress in terms of pre-service teachers‟ understanding of the nature of science. In a national survey of college science educators (N = 311, chi square p = .01), Bachkus and Thompson (p. 77) concluded, “In spite of exhortations otherwise in the current reform and post national standards period, the results of this survey suggested that very little is done formally toward ensuring a presence of the nature of science with pre-service science teacher preparation programs.” Results of the survey revealed science methods courses, science research projects, and science content courses were necessary components of pre-service teacher education preparation in order to facilitate pre-service teachers‟ understanding of the nature of science. Recent Studies Conducted to Determine Instrumentation and Data Gathering Protocols for Investigating the Impact of Reform in Undergraduate Science Several studies were conducted to determine the effectiveness of instruments and data gathering protocols to determine the impact of reform in undergraduate science with a small samples of courses. Study One In one, short term only, pilot study a sample of six institutions, each including a reform, NOVA, course and a comparison course, were investigated. Instruments used were; Views of the Nature of Science (VNOS) (Bell & Lederman, 2000) Survey on the Nature of Science (SONS) (Lawson, Drake, Johnson, Kwon, & Scarpone, 2000) Thinking About Science Survey Instrument (TSSI) (Cobern, 2000) Classroom Learning Environment Survey (CLES) (Taylor & Fraser, 1991, 1997)

Determining the Impact of Reformed Undergraduate Science Courses on Students

12 Science Teaching Efficacy and Beliefs Instrument (STEBI) (Riggs & Enochs, 1990), The study involved pre and post testing undergraduate science courses in the same format in 12 different undergraduate science classes, totaling 252 students (Sunal, D. Sunal, C., Whitaker, K., Odell., MacKinnon, C., 2003). A second administration involved repeating testing in the same classes and same institutions over a two-semester period. The study found that undergraduate students in reform-oriented undergraduate classes demonstrated a significantly higher end-of-course growth and level of achievement in understanding of the nature of science as compared to students in non- reform undergraduate science courses. The learning climate was significantly more positive in reformed courses as compared to traditional non-reform courses. The results

• f this part of the study were:

Undergraduate student understanding of the nature of science varies between classes and universities, Little or no growth in understanding of the nature of science was measured over a

• ne semester period in the average university science course using traditional

pedagogy, Students in reform-based undergraduate science courses showed significant growth in understanding and perception of the nature of science and a significantly higher post-test (end of course) level of understanding and perception of the nature of science as compared to similar students in traditional non-reform science courses measured over a semester period. Results from site visits and interviews indicated that reform course students perceived a significantly different learning environment that was more compatible with the reform goals of the national science standards, when compared to students in non-reform courses. Reformed course students reported a more positive classroom learning environment and these results were significantly related to learning outcomes. The results demonstrated that issues affecting change vary according to the institutional size, institutional mission institution, past experiences of faculty, location along the change process timeline, and other factors. A collaborative team approach, effective faculty development, faculty action research, and classroom and departmental climate were related to the outcomes measured and form the key elements of reform in undergraduate science courses. Study Two In a second pilot study, completed with a sample of 30 institutions, research was conducted to better understand the change processes necessary for university science teaching reform to be successful (Sunal D., Sunal, C., Sundberg, C., Odell, M., Whitaker, K., Seale, R., 2002).. The professional development processes involved faculty cognitive perceptions of learning, teaching skills, and pedagogical knowledge, as well as faculty culture in teaching science courses as a result of series of NASA/NOVA faculty development workshops with extended development programs were conducted at U.S.

Determining the Impact of Reformed Undergraduate Science Courses on Students

13 locations to explore, develop strategies, and implement changes in science classrooms. A review of research and these professional development experiences provided a base to carry out research activities related to understanding change in science faculty. Ninety four faculty participants in the programs were selected to be involved in the study. Ethnographic and case study approaches were used to collect and analyze data. Many faculty members encountered in this study had conceptions of the change process that inhibited successful action. These included perceptions of need for change, barriers to be dealt with, and beliefs about and efficacy toward effective classroom teaching and student learning. Knowledge of pedagogy, innovative course design types, the process of creating course change, and appropriate barriers to address were related to successful course change. Sustained change in undergraduate science courses did not occur unless there was faculty dissatisfaction with existing conceptions of science teaching. Creating cognitive conflict with existing faculty conceptions of teaching is an important role of successful professional development. Innovative pedagogical ideas for course change must be made clear and plausible through a variety of collaborative experiences if faculty are to attempt their use. Additional results from the study were summarized in themes that provided specific conditions as important for implementation and institutionalization of science course reforms to take place. Interaction, collaboration, of faculty between colleges, for example Arts and Sciences and Education, relates to successful sustained change in undergraduate science courses. Collegial and administrative support is important in successful change for most science faculty. The greater the change attempted, the greater the need for support. Administrator presence in some part of the change process facilitates greater change. Change begins with the goal to be accomplished, not with the personal or contextual barriers to be overcome. Connections with others having a similar goal build a core of active faculty and administrators. Planning for incremental change is a successful development process for course change. Conducting action research is an important component for most science faculty in creating successful course change. Joining a network of faculty within and/or outside an institution who continuously collaborate and disseminate results of change in teaching is a factor in sustaining success in course change. While most higher education institutions implement staff development activities, the professional development process is limited and elements are unconnected. Their effectiveness can be improved. These research results provide a predictive model for assisting faculty change and help determine which faculty professional development efforts may be successful.

Determining the Impact of Reformed Undergraduate Science Courses on Students

14 Study Three A third pilot study was conducted to determine the feasibility of investigating long term impact of one reform and one traditional undergraduate science course at a single institution. The purpose of this study was to obtain an understanding of the interactions that take place during a reformed standards-based college science course and how those interactions may have a long term effect on science teaching and learning. This study focused on identifying factors that may influence elementary preservice teacher professional development in science education (Staples, 2002). Instruments used were Science Teaching Efficacy and Beliefs Instrument (STEBI) (Riggs & Enochs, 1990), Constructivist Learning Environment Survey (CLES) (Taylor, Fraser & White, 1997), Thinking About Science Survey Instrument (TSSI) (Cobern, 2000), and Reform Teacher Observation Protocol (RTOP) (Sawada, Turley, Falconer, Benford & Bloom, 2002). The quantitative and qualitative data results indicated that reformed (NOVA) college science courses support the mandate for an inquiry (constructivist) learning environment for science teaching and learning science (National Research Council, 1996). The methods implemented during the course served as conditions by which students negotiated meaning for science in their lives. The students used the course methods as a tool for personal science success and for developing coherent pedagogical content knowledge (PCK) for teaching science. The students with low personal beliefs (STEBBI) about science negotiated the conditions in the course as a support base for learning science. The students with high personal beliefs (STEBBI) about science negotiated the conditions as a science resource for classroom instruction. The transactional system approach used in this study analyzed the dynamics that transpired during the course, highlighting the effects of the course on education majors' different levels of beliefs about science. The quantitative results indicated that students in the reform science course had a significantly higher score on the TSSI. CLES results showed that education majors had a significantly higher preference for a constructivist-learning environment. Overall, the undergraduate reform course education majors had higher Science Teaching Outcome Expectancy (STEBBI) posttest scores later in the teacher education program. Elementary classroom science lesson observations revealed that reform elementary classroom teachers taught science instruction that is "standards-based." Reformed Teaching Observation Protocol (RTOP) observation results related to reform elementary classroom teachers described an environment that facilitated success in student learning of science. These teachers engaged the students in the learning of

• science. The students in the reformed science course did not convey a sense of

disenfranchisement from science. Those teachers with experience in a reformed (NOVA) science course had higher STEBBI scores, science teaching outcome expectancy, for their students. The atmosphere in the reformed teachers' classrooms promoted science literacy and science for all. This

• utcome is consistent with previous studies of elementary teachers experienced in

constructivist learning approaches improving to science teaching and learning (Aldridge, Fraser, Taylor, & Chen, 2000; Finson, 2001). The students in the elementary classrooms

Determining the Impact of Reformed Undergraduate Science Courses on Students

15

• f teachers with experience in the reformed science course had higher mean scores on the

science section of the SAT9, standardized science test, as compared to the population sample of students of the traditional science course teachers. The elementary classroom teachers who were not exposed to the reformed course had lower RTOP scores. During classroom observations, the environment was not

• inclusive. In-depth questioning, reflection, and detailed predictions were not promoted.

During the interviews the traditional elementary teachers who had experienced only traditional higher education science courses felt that a textbook was key to teaching science successfully. Study Four In a fourth pilot study the science departmental cultures at three higher education institutions undergoing undergraduate reform activities was investigated. The purpose of the study was to determine factors associated with faculty development processes and the creation of reformed science courses by higher education faculty who had participated in a model staff development project. A purposeful and convenient sample of three university faculty teams were subjects for this qualitative study. Each team had attended a NASA Opportunities for Visionary Academics (NOVA) workshop, received funding for course development, and offered innovative courses (Bland-Day, 1999). Five questions were addressed in this study: a) What methods were used by faculty teams in planning the courses? b) What changes occurred in existing science courses? c) What factors affected the team collaboration process? d) What personal characteristics of faculty members were important in successful course development? and e) What barriers existed for faculty in the course development process? Data was collected at each site through individual faculty interviews (N = 11), student focus group interviews (N = 15), and classroom observations. Secondary data included original funding proposals. Analysis of data resulted in four factors of staff development processes that were positive indicators of change. First, the team collaborative processes were crucial in successful course development. Second, the use of instructional grants to fund course development, NASA/NOVA, gave credibility to the faculty involved in course development. Third, the faculty members taking the lead in creating teams actively sought out faculty members in the sciences who had previous experience teaching at the K-12 level or in informal education. Finally, college departmental environments were found to have an impact on the success of the reformed course development projects. Description of the National Research Model A primary concern in making the connection between faculty development and the goal of improving the pre-service teacher learning outcomes is the type of subject matter structures developed by those pre-service teachers. The subject matter structures reported by in-service teachers are content-oriented and initially formed as a result of undergraduate science courses, then reinforced by the act of teaching. Two implications can be made. First, teachers seem to be heavily influenced by the types of undergraduate science courses they take. Thus, a renewed emphasis is needed on the organization,

Determining the Impact of Reformed Undergraduate Science Courses on Students

16 breadth, and depth of content found in undergraduate science programs. Second, the teaching in undergraduate courses reinforces, or at least does not challenge, the fragmented nature of students‟ knowledge and their relative inability to apply that knowledge within the context of teaching (Gess-Newsome, 1999, p. 211). In studies of typical higher education pre-service and in-service programs it has been consistently noted that development of the desired integration of subject matter and pedagogy is not accomplished (Gess-Newsome, 1999; Smith, 1999). Gess-Newsome (p. 209) noted “...subject matter structures were [not] integrated and consistent with the vision of national reforms in science education. This investigation made us aware ...that the nature and validity of instruction in college science courses was absolutely critical.” Teacher education programs can only accomplish so much without the support of a reform climate in the undergraduate science courses pre-service teachers take. “Subject matter structures are largely determined in college science courses and it appears that these courses do not yield coherent and integrated subject matter structures” (Gess- Newsome, p. 209). Coherency includes not only the specific content selected but also standards-based instruction. Thus, the learning environment and course structure are critical factors in the development of meaningful science learning outcomes and also in the development of the pedagogical content knowledge needed by teachers. Traditional approaches in teaching undergraduate introductory science courses do not work effectively with many of today‟s students, particularly for non-science majors. In addition, most science faculty have little professional training in teaching (NRC, 2003). Reforms can be successful (Fullan, 1993; Sundberg & Monaca, 1994). Several recent reforms have impacted hundreds of universities and their students. The National Science Foundation‟s (NSF) goal leading to reforms for higher education includes the idea that “all students have access to supportive, excellent undergraduate education in STEM and all students learn these subjects by direct experience with the methods and processes of inquiry” (NSF, 1996, p. 1). The NRC‟s call for reforms focused on “the ultimate goal of undergraduate education should be for individual faculty and departments to improve the academic growth of students” (2003, p. 14). Pedagogical content knowledge (PCK) is the teacher‟s comprehension of how to help students understand specific subject matter. It includes knowledge of how particular subject matter topics, problems, and issues can be organized, represented, and adapted to the diverse interests and abilities of learners and then presented for instruction (Fisher, 2004, Magnusson, Krajcik & Borko, 1999). Faculty throughout the country support the need to reform science courses, but most indicate they do not have the understandings or pedagogical content knowledge to accomplish such change (Herr, 1988; Ory, 2000). To help students understand scientific ideas is a complex task (Hewitt & Seymour, 1991) that takes a significant amount of pedagogical content knowledge and skills in addition to science understandings. Yet, most science faculty have little, if any, professional training in teaching (NRC, 2003, pp. 14-15). Effective science teaching involves (a) purposeful, research-informed, development of standards-based lessons with effective teaching strategies that actively involve students in learning, (b) use of effective strategies of student assessment and feedback, and (c) self-evaluation of the effectiveness of teaching using action research (NRC, 2001). Efforts to improve faculty members‟ science teaching have involved professional development strategies; workshops, written materials on effective practices, expert and

Determining the Impact of Reformed Undergraduate Science Courses on Students

17 peer consultation and mentoring, and funded course development (Weimer & Lenze, 1994). Research on specific efforts shows limited success. Studies combining several strategies have been more successful (Sunal et al, 2001). Professional development based

• n a coherent paradigm for integrating various strategies has been most successful.

Previously reported research supports a professional development program centered on the belief that collaboration, as well as intervention, play important roles in the process of change (Harwood, 2004). Ongoing collaboration during the change process creates a personal change in beliefs (Peterman, 1993). Change in beliefs does not follow the implementation of change, but is produced within the individual or group of involved individuals during the change process itself. It is the change process that produces the most meaningful and long-lasting professional development. A professional development program using cognitive apprenticeship focuses on intervention and collaboration and has been shown to be an effective means of creating change (Collins, Brown & Newman, 1989). Cognitive apprenticeship involves a group of learners exchanging roles from teacher to learner and back again. The apprenticeship is designed to change implicit everyday knowledge to explicit informed practical knowledge through shared reflection and action research. The cognitive apprenticeship involves three phases. The first phase is a sharing of beliefs, making them public (elicitation phase). The second phase is an attempt to create a cognitive dissonance through discussion, reflection, and observation of alternative approaches for teaching (reflection phase). The third phase is a reconstruction of ideas related to defining effective learning and teaching in undergraduate science classes (reconstruction phase). This change process is continuous and iterative involving a faculty member in doing, reflecting, learning, and changing. The basic processes are a part of an action research strategy (Sunal et al., 2001). The implications of previous research on faculty development in higher education and the supporting framework of cognitive apprenticeship provide guidelines for a more effective model of professional development in higher education. These guidelines were used to support the establishment of the NOVA professional development program. The NOVA program incorporates specific conditions cited in the research literature as necessary for successful course reform, implementation, and institutionalization; (1) interaction of faculty between colleges (e.g. Arts and Sciences and Education), (2) participation in a collaborative team representing differing expertise, (3) a positive college and department climate relating to the reform effort‟s goals, (4) administrator presence and support in the change process, (5) beginning with the reform goals to be accomplished rather than with personnel or contextual barriers, (6) collaborative interactions building on effective interpersonal skills and trust, (7) planning for incremental rather than initial massive change, (8) ongoing and consistent monitoring of the reform activities using action research, (9) sustaining through collaboration in a network of faculty within and outside of the institution (Sunal et al., 2001). Figure 1 illustrates the process and components of professional development in undergraduate science course reform and its outcomes. Faculty change is enabled by an effective professional development model that supports change in learning environment, course structure, pedagogical content knowledge, and collaboration. The resulting

Determining the Impact of Reformed Undergraduate Science Courses on Students

18 reformed undergraduate science course demonstrates these four change characteristics as students interact in the classroom. Students‟ learning outcomes include a deeper understanding of appropriate content and pedagogical content knowledge (PCK) which are incorporated into longer term pre-service teacher outcomes, characterizing graduated in-service teachers in their own classroom science teaching. The focus is on the change process, which responds to national mandates to develop science literacy among diverse students (Corno & Snow, 1986). The anticipated outcome for K-6 classrooms is teachers with increased ability to facilitate greater science literacy among K-6 students.

Pedagogical content knowledge Professional development model Faculty change Learning environment Course structure Collaboration Reformed undergraduate science course Undergraduate student learning

• utcomes, short

term Course graduates, as inservice teachers.

• utcomes,

long term

Figure 1: Professional development impact model The NOVA Professional Development Model In response to the needs identified in undergraduate science teaching, guidelines were developed for a national professional development model for higher education faculty that incorporated the processes shown in Figure 1 (Peterman, 1993; Weimer & Lenze, 1994; Loucks-Horsley et al., 1998; Sunal et al, 2001; Sunal, 2004a, Zollman, 1997, 2004). The model‟s development was sponsored by the National Aeronautics and Space Administration‟s pre-college preparation program, NOVA (NASA Opportunities for Visionary Academics). Since 1996, NOVA has invited the participation of undergraduate faculty concerned with how universities prepare pre-service teachers. Through NOVA, entry-level reform science courses are developed by collaborative teams

• f faculty in the sciences and education. Currently, 102 institutions have implemented

167 reformed undergraduate science courses in a national network of institutions. Participation in NOVA included opportunities for, and commitment to, enhanced knowledge and skills through workshops, exemplary models, grants, mentoring, evaluation site visits, and collaboration within and between higher education institutions. The NOVA professional development model was delivered in 3 phases: (1) planning and preparation, involving training, collaboration, and action planning for addressing baseline needs in faculty skills and knowledge enhancement; (2) development and implementation, involving initial course change, action research, mentoring, and sharing

• f expertise; and (3) continuing development and long-term sustaining activity, involving

action research, networking, monitoring including site visits, and dissemination (Sunal et al., 2004). The NOVA professional development model includes:

Determining the Impact of Reformed Undergraduate Science Courses on Students

19

• 1. a team approach with faculty and administrators in a systemic initiative

(collaboration) Phases 1, 2, 3

• 2. Intensive professional development (28 hours) addressing
• a. higher education concerns reflected in the national science standards (learning

environment, course structure, pedagogical content knowledge [PCK]) (Siebert & McIntosh, 2001; NRC, 1996; AAAS, 1993)

• b. best practices and exemplary demonstration models from research in science

curriculum, pedagogy, assessment, collaborative learning, and working with student diversity in higher education, (learning environment, PCK) (Backer, 2002; Christopher & Atwood, 2004; Francis, Adams & Noonan, 1998; Krinsky, Anderson & Kidane, 1998; Project Kaleidoscope, 2005; Scharmann, Stalheim-Smith & James, 2004; Slater & Sireci et al, 2003; Sunal, 2004b; Swanson & Bilderback, 1998; Wycoff, 2000)

• c. action research planning and methodology (PCK, collaboration)

(Raubenheimer, 2004)

• d. best practices in research and methods in the use of technology to facilitate

science learning (learning environment, PCK) (Odell et. al. 2004)

• e. course change through grant writing skills (collaboration) Phase 1
• 3. development of a proposal for course change that is reviewed with feedback

provided (collaboration) Phase 1

• 4. development of standards-based reform undergraduate science courses in a range
• f institutions from Bachelor‟s degree granting through research universities

(learning environment, course structure, PCK, collaboration) (Goldston, Clement, & Spears, 2004; Gardner, 2004) Phases 1, 2

• 5. financial support to implement reform science courses on a long term basis

(collaboration) Phases 1, 2, 3

• 6. continuous mentoring and monitoring of progress including evaluation site visits

during development and implementation (collaboration) Phases 1, 2, 3

• 7. action research conducted by faculty teams that examines student and faculty

development (PCK, collaboration) Phases 2, 3

• 8. continuous long-term professional development activities based on best practices

research over multiple years (PCK, collaboration) Phase 1, 2, 3

• 9. collaboration and sharing of expertise and practices between faculty within an

institution and among different institutions (PCK, collaboration). (Sunal, 2002). Phase 3 Initial comparisons of small samples of courses in pilot studies found positive results for the use of this model that indicated it met the specific conditions identified above for successful course reform (Bland-Day, 1999; Staples, 2002; Sunal et al., 2003b, 2003c). In addition, these studies found increased undergraduate student achievement, long-term change in efficacy in science teaching, positive attitudes toward science, more effective use of research based science teaching practice among the pre-service elementary teachers after they graduate. These pilot studies found long-term impacts increased as students gained additional experiences in coursework and in classroom teaching rather than declining over time (Gabel, 2004; McCormick, & MacKinnon, 2004; Jordan, Elmore & Sundberg, 2004; Sunal et al., 2001; Sunal et al., 2003; Waggoner et al., 2004)

Determining the Impact of Reformed Undergraduate Science Courses on Students

20 Other studies reporting results of faculty development and course reform projects support the NOVA model reforms. Evaluation and research results reported by these projects support the key elements of the NOVA model described above. The research procedures and instruments from reviewed studies were used to develop the data collection instruments and procedures in the research project being reported in this article (Backer, 2002; Francis, Adams & Noonan, 1998; Krinsky, Anderson, & Kidane, 1998; Project Kaleidoscope, 2005; Slater & Sireci et al, 2003; Swanson & Bilderback, 1998; Wycoff, 2000). NOVA defines standards-based reform science courses as classrooms consisting

• f multiple levels of communication and multiple contexts for communication (Sunal et

al., 2004). Teachers and students bring individual frames of reference to the classroom environment and these perspectives shape the ways that individuals construct meaning from classroom interactions (Morine-Dershimer & Kent, 1999). To study reform courses, characteristics must be established that define such a course. The following NOVA reform course characteristics result from this definition: emphasis on facilitating all students’ learning of science use of pedagogy engaging students’ prior knowledge use of structured inquiry pedagogy with active and extended student participation as a regular part of the instruction refocusing of the role of the instructor who works to become a reflective practitioner using action research use of integrated multiple learning formats not only separated lecture and lab refocusing of science content on a few key ideas covered in depth use of interdisciplinary approaches in course content use of student group reflection and learning activities focused on interactive and collaborative learning through shared responsibility emphasis on evidence-based learning, using relevant and real data reflecting the way science is done use of diverse technology in most course activities to facilitate learning focusing on performance assessment forming the greater part of course assessment (DeBoer, 2004; Herppert & French, 2004; Mason & Gilbert, 2004) Research Questions Because undergraduate science influences pre-service teachers‟ ability to implement standards-based reform in K-6 schools, this proposed project addresses the

• verall problem of, How do we enable higher education faculty to implement effective

standards-based reform in science courses to all students? Since significant professional development efforts are underway to enable higher education faculty to reform undergraduate courses, it is important that we investigate, in these existing courses, the central research question, What are the impacts on short- and long-term student

• utcomes of faculty professional development that focuses on the implementation of

standards-based reform in undergraduate science courses? This study involved a national population of 103 higher education institutions with reform courses created by faculty collaborative teams, who participated in the

Determining the Impact of Reformed Undergraduate Science Courses on Students

21 NASA/NOVA professional development program over the last 10 years, and other courses, not a part of the reform process, taught at the same institutions. The research procedures and instruments from reviewed studies were used to develop the data collection instruments and procedures. The research questions are linked to the theoretical professional development impact model in Figure 1. The NSUES study will address the following questions:

• 1. How does the NOVA professional development model change undergraduate science

faculty members’ teaching practice based on evidence of their descriptions of their curricular priorities, lesson planning, instruction, and explanations of their pedagogical decision-making?

• 2. How do reform science course characteristics - learning environments, course

structure, pedagogical content knowledge, and collaboration - differ between reform (treatment) and comparison (traditional) courses and how does this relate to the learning outcomes of undergraduate students?

• 3. How do the levels of reform science course characteristics - learning environments,

course structure, pedagogical content knowledge, and collaboration - differ between reform courses (treatment only) and how do these differences relate to the learning

• utcomes of undergraduate students?
• 4. How do the science course characteristics of reform and traditional courses compare

in the long-term based on graduated in-service K-6 teachers in their own science classrooms? To address these questions the research design involves comparative and relational studies that investigate comparisons between reform and non-reform classes at the same institution and comparisons of courses demonstrating differing levels of reform among institutions that have participated in the NOVA professional development model. Study Design Connections to the Professional Development Model The research questions are linked to the theoretical professional development impact model in Figure 2. This research design uses comparative and relational analyses in different stages of the model; where professional development impacts faculty change (Stage 1, research question 1), faculty change is demonstrated through reform course development and implementation that impacts short-term undergraduate science course student learning outcomes (including pre-service teachers as a special subgroup) (Stage 2, research questions 2, 3), and reform course development and implementation impact long-term on in- service teacher outcomes (Stage 3, research question 4).

Determining the Impact of Reformed Undergraduate Science Courses on Students

22

Pedagogical content knowledge Professional development model Faculty change Learning environment Course structure Collaboration Reformed undergraduate science course Undergraduate student learning

• utcomes, short

term Coure graduates,as pre-service teachers,

• utcomes,

long term NSEUS Stage 1 NSEUS Stage 2 NSEUS Stage 3

Figure 2: Study analysis of professional development impact model Sample Faculty collaborative teams who participated in the NOVA professional development model created 185 reform courses at 103 higher education institutions. From this population of institutions a stratified random sample (n = 30, 30% of the population) was selected. The population was stratified by institutional type based on the most recent Carnegie Classification (1) comprehensive/research, (2) Baccalaureate/M.A. granting, and (3) HBCU/HHIS/tribal college (The Carnegie Foundation for the Advancement of Teaching, 2007). We will also look specifically at comparisons involving HBCU, HHIS and tribal colleges. From the stratified random sample of institutions, 30 reform courses will be selected (where there is more than 1 reform course at an institution, 1 course will be randomly selected) and matched with 30 comparison courses at the same institutions. Based on typical enrollments in such courses, the analysis will involve approximately 4000 students. At each sample institution, a reformed entry-level science course and a comparison entry level science course whose instructor has not participated in a national professional development program are selected. Data will be gathered from the 60 reform and comparison instructors, the undergraduate science classrooms, undergraduate students in the courses (including pre-service teachers as a special subgroup), and 180 selected graduates of the science courses who are now in-service teachers in elementary schools and from their elementary classrooms. We realize that a faculty member in the comparison course could have been influenced by the reform efforts demonstrated in the NOVA course at the same institution (e.g. discussions at faculty meeting, observing reformed classes). However, if this bias

• ccurs, it would reduce the statistical significance and effect size rather than the reverse.

Thus, the effect of the reform course would be underestimated rather than overestimated. Data was collected that utilized quantitative and qualitative instrumentation and techniques at multiple points. Use of comparison and relational analysis techniques was designed to increase validity. Comparison analyses examined differences between reform and comparison courses. Relational analyses examine relationships between the

Determining the Impact of Reformed Undergraduate Science Courses on Students

23 characteristics of reforms implemented in the NOVA courses and level of student

• utcomes. Criteria to identify level of implementation of reform for an undergraduate

science course and its predicated impact on students will be one of the final results of the study. During and following the project an interactive NSEUS national web site for the dissemination of the model, its implementation, use, and practical implications for colleges and universities, science faculty professional development, and pre- and in- service teacher training is available for faculty, teachers and all stakeholders in teacher

• education. The URL for the web site is http://undergradsciencereform.org. The timeline

for the NSEUS data collection and analysis is shown in Figure 3. In the last year of the project a National Conference on Undergraduate Science Education will be held. The conference will be conducted both in the face-to-face and Online (Cyber-conference)

• formats. The conference will focus on implications for science faculty professional

development and design of reform standards-based undergraduate science courses to improve undergraduate student science learning and pre-service teacher outcomes

National Study of Education in Undergraduate Science Timeline

Pilot studies, survey and characterize population, plan data collection protocols and instruments, and select study sample Complete pilot studies, conduct training in research protocols for all data collectors, & begin to collect data from sample institutions Collect & begin analysis of data from national sample in institutions and elementary classrooms, conduct data analyses further develop and implement dissemination plan Collect data from national sample for institutions and elementary classrooms conduct analyses of data further develop & implement dissemination plan Complete data collection and analysis, dissemination of results on undergraduate science reforms, conduct national conference on undergraduate science reform and its impact on undergraduate students and teachers Year 1 Year 2 Year 3 Year 4 Year 5

Figure 3: NSEUS Timeline Instrumentation and Data Collection Data will be collected from each sampled institution over a four year period and specifically for two years before, during, and after each course offering. Additional data will be longitudinal, sampling students who have completing a course and have graduated and taken a teaching position at an elementary school. Data collection is completed via site visits by project data collectors to each institution on multiple occasions. Data collected are both quantitative and qualitative as indicated in the professional development impact model (Figure 1 and Table 1). The data collection instruments have

Determining the Impact of Reformed Undergraduate Science Courses on Students

24 been previously developed and published in the literature (Table 1). The project instruments were pilot tested in small sample studies and doctoral dissertations. The data collection instruments used are the Reformed Teaching Observation Protocol (RTOP) used in undergraduate science courses and elementary K-6 classrooms

• f in-service teacher graduates of the reform science courses and other teachers in

comparison classes in the same elementary school, Classroom Learning Environments Survey (CLES) used in undergraduate science courses and elementary K-6 classrooms of in-service teacher graduates of the reform science courses and other teachers in comparison classes in the same elementary school), Views on the Nature of Science (VNOS) and Thinking About Science Survey Instrument (TSSI) (Cobern, 2000) used in undergraduate science courses and with elementary K-6 classroom in-service teacher graduates of the reform science courses and other teachers in comparison classes in the same elementary school, Science Teaching Efficacy and Beliefs (STEBI-B) used with elementary K-6 in-service teacher graduates of the reform science courses and other teachers in comparison classes in the same elementary school, and Draw-A-Scientist Test (DASTT-C) in both undergraduate science courses and elementary K-6 classrooms of in- service teacher graduates of the reform science courses and other teachers in comparison classes in the same elementary school. The means of capturing and portraying Science Pedagogical Content Knowledge (PCK) linked to particular science content and teaching practices developed for both science faculty and elementary teachers‟ PCK uses Content Representation (CoRe) and Pedagogical and Professional experience Repertoires (PaP- ers) (Loughran, Mulhall & Berry, 2004). In addition, the following data collection is utilized: content analysis of course materials artifacts used in both undergraduate science courses and with elementary K-6 classrooms of pre-service graduates of the science courses, review of NOVA science faculty individual action research documents, protocols for interviews of undergraduate students and in-service graduates of science courses, and course faculty. The data collection instruments are summarized in Table 1. Table 1: Instruments Used for Data Collection Instrument Reliability Validity Reference Reformed Teaching Observation Protocol (RTOP) 0.97 for whole instrument established with national standards Sawada, Turley, Falconer, Benford & Bloom, 2002 Classroom Learning Environments Survey (CLES) 0.91 validity established with theory of constructivism and conceptual change Taylor, Fraser & White, 1994 Views on the Nature

• f Science (VNOS)

and Thinking About Science Survey Instrument (TSSI) high face and content validity of the instrument were established with experts in research settings Bell & Lederman, 2000 and Cobern, 2000

Determining the Impact of Reformed Undergraduate Science Courses on Students

25 Science Teaching Efficacy and Beliefs (STEBI-A & B) Personal Science Teaching Efficacy = 0.90, Outcome expectancy = 0.79 content and empirical validity established in numerous studies Riggs & Enochs, 1990 Draw-A-Scientist Test (DASTT-C) Authors determined that validity and reliability indicated internal consistency on all sub-scores and total score Thomas, Pedersen, & Finson, 2001 Analysis and Interpretation of Results The instruments, protocols and content analyses result in a large collection of quantitative and qualitative data. To increase validity we will use comparison and relational analysis techniques (see Table 2). Comparison analyses examine differences between reform and comparison courses. Relational analyses examine relationships between the characteristics of reforms implemented in the NOVA courses and level of student outcomes. Quantitative data analysis consists of (a) initial data “cleaning” and variable recoding, followed by the generation of (b) descriptive statistics, (c) multivariate analysis

• f variance using a profile analysis and appropriate follow-up measures, and (d)

regression analysis to measure the relationships and contributions of the various components of the professional development impact model to outcomes. Qualitative analysis consists of (a) grounded theory coding of interviews with teachers for the conceptual vocabulary teachers use to describe their science teaching (Strauss & Corbin, 1998), (b) constant comparative analysis of this conceptual vocabulary with the concepts and models recommended by the NOVA program (Merriam, 2002), (c) comparison of these qualitative outcomes between participating teachers and the control group, (d) a summative comparative analysis of themes that emerge in the qualitative data with patterns that emerge in the quantitative data, (e) testing emergent hypotheses about these relationships between themes and patterns (Miles & Huberman, 1994, 2001), (f) searching for multiple explanations of these relationships supported by the data, and (g) writing a summary report of the qualitative data analysis. Effect sizes will be determined for all quantitative analyses. Criteria for success of the project will be determined by conclusions drawn from the research questions utilizing quantitative and qualitative data analyses; including evidence and effect sizes of short-term impacts on students and long-term effects on graduated in-service teachers in their own classroom science teaching; identification of characteristics of reform courses that produce significant impacts on students; identification of characteristics of effective faculty, and dissemination of effective processes and information. The data collection and analyses are summarized in Table 2.

Determining the Impact of Reformed Undergraduate Science Courses on Students

26 Table 2: Research Question Variables, Data Collection, and Analysis Research Question Variables Measured Instruments Analysis 1 faculty curricular priorities, lesson planning, instruction, & pedagogical decision making faculty interviews, STEBI-B, RTOP, questionnaire, content analysis of course materials, review of NOVA faculty action research products Qualitative and quantitative comparisons will be made between reform variables predicted by the NOVA professional development model and post outcomes 2 faculty curricular & instructional priorities, classroom learning environment, course structure, PCK, collaboration, student learning outcomes faculty & student interviews, STEBI-B, CLES (instructor version), RTOP, CoRe, PaP-ers, content analysis of course materials, DASTT-C, CLES (student version), VNOS Qualitative methods, multivariate analysis of variance, and effect sizes will be used to compare reform and traditional courses based

• n the implementation
• f reforms and student

learning outcomes 3 faculty curricular & instructional priorities, classroom learning environment, course structure, PCK, collaboration, student learning outcomes faculty & student interviews, STEBI-B, CLES (instructor), RTOP, CoRe, PaP-ers, content analysis of course materials, DASTT-C, CLES (student), VNOS, Qualitative comparisons will be used as well as regressing student achievement onto the levels of reform science course characteristics 4 graduated K-6 teacher classroom learning environments, attitudes, instruction, curricular priorities, student achievement DASTT-C, CLES (instructor), CLES (student),STEBI-B, VNOS, RTOP, interviews, standardized achievement test (K-6) Qualitative methods, multivariate analysis of variance & effect sizes compare undergraduate course characteristics & teaching characteristics

• f graduates of reform

courses and traditional courses Findings to Date of the Multiyear Study Study results and conclusions, to date, from data analyses involved 1) a national survey of NOVA reform higher education institutions completed to characterize the study population and the sustained impact of reforms conducted in undergraduate science courses and 2) pilot studies conducted to determine validity and reliability of instruments,

Determining the Impact of Reformed Undergraduate Science Courses on Students

27 feasibility of data collection, other site visitation research protocols, and identification of characteristics of effective faculty collaboration and higher education institution departmental contexts. Summary Currently, there is a major emphasis in higher education on rethinking undergraduate science instruction, particularly introductory courses that meet the academic needs of education majors. This effort is being driven by the NCLB, new accreditation policies, national and state science standards, and national and state governmental agencies. On the surface, changing undergraduate science courses is perceived as deceptively easy, but in reality, there are many issues that higher education faculty must confront before innovations can be implemented and sustained. This national study establishes criteria for identifying varying levels of standards based reform in undergraduate science courses and examines the impact they have that relates to improved science literacy of all affected students. This research advances the understanding of characteristics of entry-level undergraduate science courses impacting student subject matter knowledge and PCK of pre-service teachers that translate into more effective science teaching in the K-6 school classroom. Undergraduate science courses influence science literacy of all affected students and especially teachers‟ ability to implement standards-based reform in their own

• classrooms. To develop expertise in teaching undergraduate science, faculty require

additional knowledge and professional skills of basic issues involving models of innovation in undergraduate science, effective components of successful educational course reform in science teaching, understanding various perspectives on reform in undergraduate science, developing skills for conducting educational action research, and awareness of research supported best practice. ________________________________________________________________________ Work on the research project was supported by NSF grant ESI-0554594 titled Undergraduate Science Course Reform Serving Pre-service Teachers: Evaluation of a Faculty Professional Development Model. The opinions expressed in this paper are those

• f the authors and do not necessarily reflect those of the Foundation.

Correspondence should be sent to: Dennis Sunal, dwsunal@bama.ua.edu ________________________________________________________________________ References Abell, S., Boone, W., Arbaugh, F., Lannin, J., Beilfuss, M., Volkmann, M., & White, S. (2006). Recruiting future science and mathematics teachers in alternative certification programs; Strategies tried and lessons learned. Journal of Science Teacher Education, 17. 165-183. Adamson, S. L., Banks, D., Burtch, M., Cox, III, F., Judson, E., Turley, J. B., Benford, R., & Lawson, A. E. (2003). Reformed undergraduate instruction and its subsequent impact on secondary school teaching practice and student

• achievement. Journal of Research in Science Teaching, 40, (10), 939-957.

Determining the Impact of Reformed Undergraduate Science Courses on Students

28 Backer, P. (2002). Mathematics and Science Teacher Education Program (MASTEP) final report. Retrieved August 4, 2005 from Web site http://www.mastep.sjsu.edu/final_report.htm Bell, R. L., & Lederman, N. G. (2000, April). Decision making on science and technology issues: Do views of the nature of science really matter? Paper presented at the annual meeting of the National Association for Research in Science Teaching, New Orleans, LA. Blackwell, P. J. (2002). Putting the pieces of teacher education together: Lessons learned by the Eisenhower initial teacher professional development programs. (Eric Document Reproduction Service, ED 465 745). Retrieved January 8, 2006, from the World Wide Web. Bland-Day, J. (1999). Barriers associated with faculty creating change in teaching at higher education institutions. (Doctoral dissertation, The University of Alabama, 1999). Dissertation Abstracts International. Christopher J. E. & Atwood, R. K. (2004). Interdisciplinary Curriculum Planning in a College Course (In Sunal, D., Wright, E., & Bland-Day, J. (Eds.) Research in Science Education: Reform in Undergraduate Science Teaching for the 21st Century. Greenwich, CT: Information Age Publishing, 123-136. Cobern, W. W. (2000). The thinking about science survey instrument (TSSI) –SLCSP

• 151. Kalamazoo, MI: Scientific Literacy and Cultural Studies Project.

Darling-Hammond, L. (2006). Powerful teacher education: Lessons from exemplary

• programs. San Francisco: Jossey Bass Pub. Retrieved from the Internet December

17, 2006 at http://www.tcrecord.org/Content.asp?ContentID=12735 Darling-Hammond, L. 2000. Teacher quality and student achievement: A review of state policy evidence. Education Policy Analysis Archives, 8(1) [On-line]. Available: http://epaa.asu.edu/epaa/v8n1/. DeBoer, G. E. (2004). A Science Research Organization Perspective on Reform in Teaching Undergraduate Science (In Sunal, D., Wright, E., and Bland-Day, J. (Eds.) Research in Science Education: Reform in Undergraduate Science Teaching for the 21st

• Century. Greenwich, CT: Information Age Publishing, 317-328.

Ellis, R. A., Taylor, C. E., & Drury, H. (2006). University student conceptions of learning science through writing. Australian Journal of Education, 50(1), 6-28. Fedock, P., Zambo, R., & Cobern, W. (1996). The professional development of college science professors as science teacher educators. Science Education, 80(1), 5-19. Fisher, K. M. (2004). The Importance of Prior Knowledge in College Science Instruction (In Sunal, D., Wright, E., & Bland-Day, J. (Eds.) Research in Science Education: Reform in Undergraduate Science Teaching for the 21st Century. Greenwich, CT: Information Age Publishing, 69-83. Floden, R., Gallagher, J., Wong, D. & Roseman, J. E. (1995). Project 2061 blueprint for teacher education: A symposium presentation. East Lansing, MI: Michigan State University. Francis, G., Adams, J., & Noonan, E., Do they stay fixed? Results of a longitudinal study

• f students 3 1/2 years after taking reformed physics courses at MSU-Bozeman.

The Physics Teacher, 36, (1998), p. 488. Gabel, D. L. (2004). A Model for Reform in Teaching Chemistry: With a Focus on Prospective Elementary Teachers. (In Sunal, D., Wright, E., & Bland-Day, J.

Determining the Impact of Reformed Undergraduate Science Courses on Students

29 (Eds.) Research in Science Education: Reform in Undergraduate Science Teaching for the 21st Century. Greenwich, CT: Information Age Publishing, 425- 444. Gardner, F. (2004). A Small Four-Year College Perspective on Reform in Teaching Undergraduate Science. (In Sunal, D., Wright, E., & Bland-Day, J. (Eds.) Research in Science Education: Reform in Undergraduate Science Teaching for the 21st Century. Greenwich, CT: Information Age Publishing, 267-284. Goldston, J. M. (2004). A University Science Education Researcher‟s Perspective on Reform in Teaching Undergraduate Science. (In Sunal, D., Wright, E., & Bland-Day, J. (Eds.) Research in Science Education: Reform in Undergraduate Science Teaching for the 21st Century. Greenwich, CT: Information Age Publishing, 371-388 Goldston, J. M., Clement, M., & Spears, J. (2004). A Large University Perspective on Reform in Teaching Undergraduate Science: A Geologist‟s Personal Practice Theories and Pedagogical Change. (In Sunal, D., Wright, E., & Bland-Day, J. (Eds.) Research in Science Education: Reform in Undergraduate Science Teaching for the 21st Century. Greenwich, CT: Information Age Publishing, 245- 266. Guskey, T. R. (2000). Evaluating Professional Development. Thousand Oaks, CA: Corwin Press, Inc. Hake, R. R. 1998. Interactive engagement versus traditional methods: a six-thousand- student survey of mechanics test data for introductory physics courses. American Journal of Physics, 66, 64. Hammrich, Penny L. (2001). Preparing graduate teaching assistants to assist biology

• faculty. Journal of Science Teacher Education, 12, 67-82.

Harwood, W. S. (2004). Science Education Reform: Factors Affecting Science and Science Education Faculty Collaborations. (In Sunal, D., Wright, E., & Bland-Day, J. (Eds.) Research in Science Education: Reform in Undergraduate Science Teaching for the 21st Century. Greenwich, CT: Information Age Publishing, 53-68. Herppert, J. A. & French, A. (2004). Scientists‟ Perspective on Reform in Teaching Undergraduate Science: Serving Preservice Science Teachers in the Context of a Heterogeneous Student Population. (In Sunal, D., Wright, E., & Bland-Day, J. (Eds.) Research in Science Education: Reform in Undergraduate Science Teaching for the 21st Century. Greenwich, CT: Information Age Publishing, 329-350. Hewitt, N., & Seymour, E. (1991, April), Factors contributing to high attrition rates among science, mathematics, and engineering undergraduate majors. Washington, DC: Alfred P. Sloan Foundation. Hubbard, P., & Abell, S. (2005). Setting sail or missing the boat: Comparing the beliefs

• f preservice teachers with and without an inquiry-based physics course. Journal
• f Science Teacher Education, 16, 5-25

Jordan, W., Elmore, B. & Sundberg, C. W. (2004). A Model for Reform in Teaching in Engineering and Technology: With a Focus on Prospective Elementary Teachers (In Sunal, D., Wright, E., & Bland-Day, J. (Eds.) Research in Science Education: Reform in Undergraduate Science Teaching for the 21st Century. Greenwich, CT: Information Age Publishing, 511-524. Judson, E., & Sawada, D. (2001). Tracking transfer of reform methodology from science and math college courses to the teaching style of beginning teachers of grades 5-

Determining the Impact of Reformed Undergraduate Science Courses on Students

30

• 12. Arizona Collaborative for Excellence in the Preparation of Teachers,

Technical Report No. PRGO1-2. (ERIC Document Reproduction Service ED 455 208). Krockover, G. H., Shepardson, D. P., Adams, P. E., Eichinger, D., & Nakhleh, M. (2002). Reforming and assessing undergraduate science instruction using collaborative action-based research teams. School Science and Mathematics, Lawson A., Drake, N., Johnson, J., Kwon, Y., Scarpone, C. (2000). How good are students at testing alternative explanations of unseen entities, The American Biology Teacher, 62(4), 249-255. Lawson, A.E., Benford, R., Bloom, I., Carlson, M.P., Falconer, K.F., Hestenes, D. O., Judson, E., Piburn, M.D., Sawada, D., Turley, J., & Wyckoff, S. (2002, Mar-Apr). Evaluating college science and mathematics instruction: A Reform Effort That Improves teaching Skills. Journal of College Science Teaching, Le, V., Stecher, B., Lockwood, J., Hamilton, L., Robyn, A., Williams, V. Ryan, G., Kerr, K., Martinez, J., & Klein, S. (Rand Corporation) (2006). Improving Mathematics and Science Education: A Longitudinal Investigation of the Relationship Between Reform-Oriented Instruction and Student Achievement. Arlington VA: Rand

• Education. Retrieved from the Internet Oct 27, 2006 at

http://www.rand.org/pubs/monographs/MG480 Lederman N. G. & Gess-Newsome, J. (1999). Examining pedagogical content

• knowledge. Gess-Newsome, J. & Lederman, N. G., (Eds.). Boston: Kluwer.

Luera, G. R., & Otto, C. A. (2005). Development and evaluation of an inquiry-based elementary science teacher education program reflecting current reform

• movements. Journal of Science Teacher Education, 16, 241-258.

Magner, D. K. (1992, October 21). A booming reform movement for introductory science

• courses. Chronicle of Higher Education, pp. 17-18.

McCormick, B. & MacKinnon, C. (2004). A Model for Reform in Teaching in the Biological Sciences: Changing the Culture of an Introductory Biology Course. (In Sunal, D., Wright, E., & Bland-Day, J. (Eds.) Research in Science Education: Reform in Undergraduate Science Teaching for the 21st Century. Greenwich, CT: Information Age Publishing, 393-408. National Research Council. (1996). National Science Education Standards. Washington, DC: National Academy Press. Raubenheimer, C. D. (2004). Professional development of university science faculty through action research (In Sunal, D., Wright, E., & Bland-Day, J. (Eds.) Research in Science Education: Reform in Undergraduate Science Teaching for the 21st Century. Greenwich, CT: Information Age Publishing, 199-224 Scharmann, L. C., James, M. C., & Stalheim-Smith, A. (2004). Assessment in college science courses (In Sunal, D., Wright, E., & Bland-Day, J. (Eds.) Research in Science Education: Reform in Undergraduate Science Teaching for the 21st Century. Greenwich, CT: Information Age Publishing, 137-152. Seymour, E. (1995). Revisiting the “problem iceberg”: Science, mathematics, and engineering students still chilled out: Examining the causes of student attrition in science-based fields on a variety of campuses. Journal of College Science Teaching, 24(6), 392-400.

Determining the Impact of Reformed Undergraduate Science Courses on Students

31 Slater, C. S., Sireci, S. G., Berger, J., Brown, M., O‟Meara, K., Hintze, J. DeCecco, L. (2003, August) STEMTEC II Evaluation report for year 1 – fall 2002/spring

• 2003. Retrieved August 15, 2005 from STEMTEC Web site:

http://k12s.phast.umass.edu/~stemtec/ Riggs, I., & Enochs, L. (1990). Towards the development of an elementary teacher's science teaching efficacy belief instrument. Science Education, 74, 625-637. Staples, K. (2002), The Effect of a Nontraditional Undergraduate Science Course on Teacher and Student Performance in Elementary Science Teaching. (Doctoral dissertation, University

• f Alabama, 2002). Dissertation Abstracts International.

Sunal, D. (2004). Innovative pedagogy for meaningful learning in undergraduate science. (In Sunal, D., Wright, E., Bland-Day, J. (Eds.) Research in Science Education: Reform in Undergraduate Science Teaching for the 21st Century. Greenwich, CT: Information Age Publishing, 85-122. Sunal, D., MacKinnon, C., Raubenheimer, C. D. & Gardner, F. (2004). A case study of a national undergraduate science reform effort (In Sunal, D., Wright, E., & Bland- Day, J. (Eds.) Research in Science Education: Reform in Undergraduate Science Teaching for the 21st Century. Greenwich, CT: Information Age Publishing, 225- 240. Sunal, D., Bland, J., Sunal, C., Whitaker, K., Freeman, M., Edwards, L., Johnson, R. & Odell, M. (2001). Teaching science in higher education: Faculty professional development and barriers to change. School Science and Mathematics. 101(5), 246-257. Sunal, C., Karr, C., Smith, C., & Sunal, D. (2003a). Modeling scientific systems using three artificial intelligence concepts, Journal of College Science Teaching 32(6), 400-405. Sunal, D. Sunal, C., Whitaker, K., Odell., MacKinnon, C. (2003). The effect of standards- based reform in university courses on undergraduates’ science knowledge, research paper presented at the American Educational Research Association (AERA) international conference in Chicago, IL. Sunal D., Sunal, C., Sundberg, C., Odell, M., Whitaker, K., Seale, R. (April, 2002). Effects of changing college level science courses to reflect the national science standards on undergraduates science knowledge - the nature of science. Research paper presented at the National Association for Research in Science Teaching international conference, New Orleans. Sundberg, M. D., & Monaca, G. J. (1994). Creating effective investigative laboratories for undergraduates. Bioscience, 44, 698-704. Taylor, P. B., Fraser, B., & Fisher, D. (1997). Monitoring constructivist learning classroom environments. International Journal of Educational Research, 27, 293-302. Taylor, P., & Fraser, B. (1991, April). CLES: An instrument for assessing constructivist learning environments. Paper presented at the annual meeting of the National Association for Research in Science Teaching, Fontane, WI. Tobias, S. (1990). They’re not dumb, they’re different: Stalking the second tier. Tucson, AZ: Research Corporation. Tobias, S. (1992). Revitalizing undergraduate science. (pp. 11-23). Tucson, AZ: Research Corporation.

Determining the Impact of Reformed Undergraduate Science Courses on Students

32 U.S. Department of Education. (2001, August 21). No child left behind.. Retrieved August 8, 2003, from http://www.ed.gov/offices/OESE/esea/nclb/titlepage.html Waggoner, C., Schaffner, M., Keller, K. L., and McArthur, J. (2004). A model for reform in teaching in the biological sciences: Infrastructure for inquiry in an introductory biology laboratory (In Sunal, D., Wright, E., & Bland-Day, J. (Eds.) Research in Science Education: Reform in Undergraduate Science Teaching for the 21st

• Century. Greenwich, CT: Information Age Publishing, 409-424.

Wainwright, C., Flick, L., Morrell, P. D., & Schepige, A. (2004). Observation of reform teaching in undergraduate level mathematics and science courses. School Science and Mathematics, 104(7), 322-335.