LEADing EOT-PACI Programs to National Success

he Learning through Evaluation, Adaptation, and Dissemination (LEAD) Center at the University of Wisconsin, Madison, is the first evaluation research center of its kind in the nation and has provided an invaluable service to EOT-PACI since 1997. Each year, LEAD evaluates a different project or set of projects that are seen as being integral to the overall mission of EOT-PACI. LEAD assists these efforts in gradually achieving national impact by providing evaluation feedback on what could make a project more successful and uncovering the essential elements of an established project's success so that these same elements may be used to scale up the project and replicate it at other sites.

Established in 1994, the LEAD Center provides third-party evaluation research to support reform efforts in higher education at the undergraduate, graduate, and faculty levels and in support of university outreach efforts. LEAD is structured as a project-based organization with each project funded by the client served. Examples range from Wisconsin projects funded by the Provost's office to initiatives such as EOT-PACI funded by the National Science Foundation. And projects are also led by members of the EOT-PACI Leadership Team: Greg Moses at Wisconsin, Richard Tapia at Rice University, and Roscoe Giles at Boston University.

SAS EVALUATION

A RECIPE FOR SUCCESS

SAS EVALUATION

One EOT-PACI program that has benefited from its work with LEAD Center is the Spend a Summer with a Scientist (SaS) Program at Rice University. The SaS program provides opportunities for undergraduate and graduate students to participate in university activities and work for the summer under the guidance of center researchers. Tapia, professor of computational and applied mathematics at Rice University, created the SaS program in 1989 to improve the recruitment and retention of women and minorities into the computational sciences. After having been in existence for eight years, the program appeared to be having a remarkable impact on both its graduate and undergraduate participants, and Tapia was interested in measuring this impact and determining whether the SaS program could be replicated at other institutions.

To do this, SaS needed a formal evaluation and documentation of the program. Starting in 1997, the LEAD evaluation team conducted a year-long evaluation that established the effectiveness of SaS, and identified key elements of the program that helped to bring about its successful outcomes. With the impact measures and strategic guidelines provided by the LEAD evaluation team, SaS began sharing its successful strategies with others, providing a model for similar outreach programs considering expansion.

The LEAD evaluation found that the SaS program did indeed have a powerful impact on the career choices and preparation levels of the students who participated in it. Of the undergraduate participants who have since graduated, 63% went on to enroll in graduate school, while 33% gained employment in mathematics, computational science, or engineering. For graduate students, the rate of retention at the time of the evaluation was 97%, with 57% of graduate student participants still in graduate school. Of those who had completed their graduate training, 66% have received doctoral degrees. Many students stated that they would not have completed their degrees--or thought to enroll in graduate school at all--had it not been for their participation in SaS. They spoke about their experiences within the program and their interactions with Tapia and other minority students as being key in their decisions to persist in computational science.

LEAD evaluators found that SaS had many elements that were key to its success. In SaS, Rice faculty members and Program Director Richard Tapia had close mentoring relationships with student researchers. The research projects and mentors were carefully matched to meet each student's needs, whether they were graduate students completing dissertation work or undergraduates who had never done research before. Unlike participants in many summer research programs, students in SaS were often involved for a number of years, and the undergraduates and graduates were brought together in meetings and working groups throughout the summer. This allowed an entire matrix of mentoring relationships to form between the students at different levels and the faculty and staff involved in the program.

Because of the Rice graduate students involved in the program, it was much more than just a summer program: It created an ongoing community of underrepresented students that provided advice, support, peer instruction, and professional development opportunities throughout the year. Another important component of the program was its frequent open discussions regarding race and ethnicity. Finally and critically, SaS was fully integrated into the life of the department and received legitimacy through its connections to a respected research center.

Motivating Students through Experience

Regina Hill, a graduate student in the Rice University Computational and Applied Mathematics (CAAM) Department and a participant in the Spend a Summer with a Scientist program, speaks on her successful efforts to solve the wave equation using the finite element method with mass lumping. Professor William Symes of the CAAM Department was her mentor.

A RECIPE FOR SUCCESS

SaS is only one example of LEAD's evaluation efforts for EOT-PACI. In 1998, the LEAD Center completed its evaluation of the Education Center on Computational Science and Engineering at San Diego State University. This evaluation helped the Center formulate strategies for bringing high-performance computing applications into the undergraduate curriculum. Through EOT-PACI, LEAD also evaluated the Computer Research Association's Distributed Mentor Program (DMP). The results showed that the DMP program has a profound impact on the career aspirations of the female undergraduates, graduate students, and faculty who participate.

The following year, LEAD assisted EOT-PACI in forming direct collaborations with tool developers in NPACI's thrust areas. NPACI's EOT management received feedback on how these partnerships are forged and how to make them more effective. The GERS program at Wisconsin was evaluated in 1999-2000 to assist this program in formulating strategies for increasing the number of underrepresented minorities receiving degrees from Wisconsin's graduate engineering programs.

LEAD will soon begin its evaluation of the Girls Are GREAT program co-sponsored by the San Diego Girl Scouts Imperial Council. This science enrichment program focuses on minority and low-income 1st-8th grade girls. The evaluation will focus on the potential for scaling the program within the Girl Scouts of America. In addition, LEAD will focus on the EOT-PACI K-12 Team's Workshop on Bringing Modeling and Visualization Applications into K-12 Teacher Preparation. This workshop is part of a larger effort to incorporate training in modeling and visualization applications into K-12 teacher education.

LEAD will continue to provide evaluation and scale-up support for selected EOT-PACI projects for the remainder of the PACI Initiative. "As is often the case, trying to write a definition of 'success' for the LEAD Evaluation Team is a little odd," says Alexander. "Success for us is to help other PIs and projects document their successes and to uncover strategies that increase the ease and the frequency with which they reach their goals." --EN