This document has been archived.

Preface

Using a question-and-answer format, the report defines informal science education, presents an overview of the ISE evaluation conducted by COSMOS Corporation, and summarizes its major findings and recommendations. The narrative analysis of the data presented in the report draws on results of site visits, focus groups, interviews, and surveys of people with science-related careers and those receiving ISE funds.

The evaluation data were compiled by Dr. Katherine Zantal-Wiener, project director; Dr. Cheryl Sattler; and Dr. Robert K. Yin. Others who assisted were Darnella Davis, Dana Edwards, Suzanne Merchlinsky, and Jennifer Elcano. Two firms, Westat, Inc., and the Educational Testing Service, served as subcontractors and assisted in the design and data collection phases of the task. In addition, Drs. Zoe Barley and Mark Jenness of Western Michigan University gave preliminary advice on the selection of the sample of site visits. The American Association for the Advancement of Science (AAAS) and the National Science Teachers Association (NSTA) provided access to their membership lists to conduct surveys of people with science-related careers.

The project team is grateful for the guidance provided by a specially convened group of advisors who were: Christine Dwyer, Senior Vice President, RMC Research Corporation; Dr. Mark Jenness, Co-Director, Science and Math Program Improvement, Western Michigan University; Nancy Kolb, President and Executive Director, Please Touch Museum; and Dr. Kenneth Phillips, Aerospace Curator, California Museum of Science and Industry. These advisors contributed a unique perspective based on their years of experience in the field of informal science education. Data collection would not have been possible without the assistance of the ISE program staff and the institutions and grantees in the informal science education community. Their cooperation in tracking archival documents, arranging site visits, and recruiting focus group members is greatly appreciated.

This report was prepared under contract RED 94-52970 awarded to COSMOS Corporation. The contents of this document are the responsibility of the authors and do not necessarily reflect the views of the National Science Foundation or its staff.
 

Table of Contents

1. What Is Informal Science Education?
2. What Is NSF's Role in Furthering Informal Science Education?
3. What Is the "ISE" Evaluation?
4. What Is the NSF's Investment in the ISE Program?
5. How Is the ISE Investment Leveraged?
6. Is the ISE Program Accomplishing Its Goals?
7. What Do People Say About ISE-Funded Activities?
8. What Are the Policy, Practice, and Program Implications?
9. Has the Investment Been Worthwhile?

1. Distribution of ISE Funds Across Activity Areas, FY1984-FY1994
2. Six Goals That Guided the Evaluation
3. Contextual Conditions and the Evaluation Goals of the NSF's Informal Science Education Program
4. NSF's Informal Science Education Funding, FY1984-FY1994
5. Distribution of ISE Funds Across Institution Type, FY1984-FY1994
6. Twenty-five Institutions Receiving the Greatest Amount of Support from NSF's Informal Science Education Program, FY1984-FY1994
7. Influence of ISE Funds on the Informal Science Education System Compared to Non-NSF Funds, as Reported by Organization Directors
8. Most Memorable Informal Science Education Activities From Their Childhood, as Reported by People with Science Careers
9. Sources of Ideas Learned in Youth that are Still Used, as Reported by People with Science Careers
10. Early Informal Science Education Activities that Initiated Connections with Schools, as Reported by People with Science Careers
11. Influence of ISE Funds as Compared to Non-NSF Funds on Increasing Collaboration, As Reported by Organizational Directors
12. Topical Distribution of ISE Projects, FY1990-FY1996

1. What Is Informal Science Education?

A major goal of the National Science Foundation (NSF) is to "promote the discovery, integration, dissemination, and employment of new knowledge in service to society" (National Science Foundation, 1995, p. 13, NSF #95-24). One strategy for achieving this goal is to "infuse education with the joy of discovery and an awareness of its connection to exploration."

"Informal science education" is voluntary, self-directed, and lifelong. It is learning that provides an experiential base and motivation for further activity and learning. NSF's Informal Science Education (ISE) program supports projects in which "learning is... motivated mainly by intrinsic interests, curiosity, exploration, manipulation, fantasy, task completion, and social interaction. This informal learning can be linear or nonlinear and often is self-paced and visual- or object-oriented" (National Science Foundation, 1997, p. 8, NSF #97-20).

Informal science learners -- people of all ages, interests, and backgrounds -- are discovering science in places outside of schools, with materials and activities initially not developed for school use or as part of a curriculum. Participants in informal science education activities engage with science on their own initiative and not as part of a mandated school experience. Participation reaps several benefits: a better understanding of concepts, topics, processes, and thinking in scientific and technical discipline; increased knowledge about career opportunities in these fields; and increased appreciation and understanding of science and mathematics, and their applications.

"Direct observation at nature centers led to my first feeling of intense interest and curiosity; [I] developed a need and interest in 'knowing' and 'exploring' for answers."

-- Individual with a science-related career

Many people with science-related careers credit their initial interest in SMET to informal rather than formal exposure, identifying museums and science centers as the most important stimulants to their childhood interest.
 

2. What Is NSF's Role in Furthering Informal Science Education?

All aspects of the ISE program are meant to stimulate and maintain a lifelong interest in math and science. The program strives to:

• Increase the number of youth, particularly the underrepresented (e.g., minorities, girls, the physically disabled) and underserved (e.g., rural communities), who are excited about SMET and who pursue such activities both in and out of school;
• Promote linkages between informal and formal education, resulting in improved and creative SMET education in all learning environments;
• Stimulate parents and other adults to become effective proponents for better quality and more universally available SMET education in both formal and informal settings and to encourage them to support their children's science and mathematics endeavors in the home and elsewhere;
• Bring informal science education programs and activities to relatively large areas currently without, or minimally reached by, such opportunities (e.g., rural areas and inner-city environments); and
• Enrich the quality of life by improving the science and technological literacy of children and adults so that they are informed about the implications of SMET in their everyday lives; are motivated to pursue further experiences in these areas; and are aided in making informed, responsible decisions about related policy issues having societal implications.

• Apply recent research in SMET education;
• Contribute to strengthening the infrastructure of informal science education through such activities as electronic networking, technical assistance, and professional development;
• Conduct research on the informal education process to determine the effectiveness of innovative techniques for motivating and informing the public about topics in science, mathematics, and technology; and
• Incorporate new material into existing programs so as to increase the science, mathematics, and technology interests and literacy of their audiences.

Any organizationpublic or private, profit or nonprofitmay submit proposals to provide informal science education experiences. Projects that explore a variety of new and emerging alternatives for reaching large audiences, as well as for improving traditional methods, are encouraged. Such alternatives include projects that use commercial broadcasting, new strategies for interactive exhibits in museums, popular press aimed at specific audiences, newly developed video and interactive learning media, home-learning resources, and home- or community-based adult education projects.

As shown in Exhibit 1, from fiscal year (FY) 1984 to FY1994, ISE funds went mostly to develop television programs and exhibits, with a slightly higher percentage of funds used to support exhibits. The rest of the funds were awarded for activities such as after-school and community programs, professional and staff development, radio programs, school-linked programs, and films.
 

3. What Is the "ISE" Evaluation?

The evaluation was designed to examine the impact of the ISE program from 1984 to 1994, with special emphasis on the period between 1990 and 1994. Past NSF-funded studies have focused on evaluating individual project goals rather than investigating the long-term impact of the ISE program. Furthermore, a review of existing evaluations of informal science education projects revealed an absence of data on the evaluation goals most relevant to the ISE program. For example, one study has examined the support provided by science-technology centers to schools and teachers (ASTC, 1996). In another study, the ISE program was the subject of a pilot study of short-term assessments that were designed to help NSF deal with difficult assessment questions (Knapp, et al., 1988). And neither NSF-funded nor non-NSF-funded informal science education project evaluations have focused on collaborations with community groups (except Ghostwriter, which received an ISE planning grant) or on affecting the informal science education system.

Thus, the design of the present evaluation, conducted by COSMOS Corporation, was based on the presumption that informal science education is not necessarily a set of interventions with short-term impacts but, rather, a means of influencing the context within which science is learned and experienced over the long term. The longer-term impacts that were the focus of the present evaluation can be derived from the evaluation goalsfour ISE program goals (AD) and two additional goals (EF)that were used to assess NSF's entire ISE program.

Experts, knowledgeable about informal science education and the ISE program, provided COSMOS with guidance regarding the design and conduct of the evaluation. The experts were: Christine Dwyer, Senior Vice President, RMC Research Corporation; Dr. Mark Jenness, Co-Director, Science and Math Program Improvement, Western Michigan University; Nancy Kolb, President and Executive Director, Please Touch Museum; and Dr. Kenneth Phillips, Aerospace Curator, California Museum of Science and Industry.

The methodology used for the evaluation was designed from a broad perspective of informal science education, since outcomes such as increased scientific literacy cannot be attributed to a single project. Therefore, instead of using the typical linear input-output approach, the study methodology acknowledged that the ISE program is influenced by family, community, and formal educationthat is, by the context in which informal science education activities take place.

The evaluation framework in Exhibit 3 incorporated the six evaluation goals (AF), while respecting the contextual nature of informal science educationconsidering it in the aggregate and not on a project-by-project basis. The design thereby provides a comparative logic for interpreting and comparing evaluation resultsthe comparison of ISE program activities and outcomes with those activities not funded by NSF. Similarly, responses by people directly associated with ISE projects were compared with those of people in science careers. The design also considered NSF's cost-sharing requirements and the fact that ISE funding is supplemented by other funds from federal, state, and local agencies, corporations, or foundations.
 

Because it is difficult to disentangle co-funded activities, informal science education projects and activities not funded by the NSF were studied as a comparison group. Three caveats about the comparison group should be noted. First, the projects in the comparison group have characteristics similar to, but not exactly like, those funded by NSF. Second, the comparison projects could have been influenced by the ISE program guidelines if: 1) the organization had submitted an earlier proposal that was reviewed by the ISE program staff and incorporated the reviewers' comments into a later proposal that may have been funded by a non-NSF entity or even NSF; or 2) the organization may have received a prior ISE grant and used the reviewers' comments to report on the development of a non-NSF project. Third, the project may have received ISE funds after the data collection commenced. Moreover, because of the difficulty in identifying informal science education projects not funded by NSF, the final selection was reviewed with the ISE program staff and advisors.

To the extent possible, parallel data were collected from both ISE projects and non-NSF informal science education projects. Data were collected from the following sources:

ISE program activities:

• Archival data from the ISE program files;

 
• Surveys from 210 (out of 277) funded project directors during the years 19841994;

 
• Surveys from 64 (out of 179) funded organizational directors;

 
• Site visits to 15 institutions representing 52 of 347 funded awards (many institutions received multiple awards); and

 
• Focus groups at 22 institutions, covering participants in informal science activities.

• Surveys of random samples of 262 members from the American Association for the Advancement of Science (AAAS) membership list and the National Science Teachers Association (NSTA) database; the responses were pooled and are reported as those belonging to "individuals with science-related careers";

 
• Site visits to six institutions representing non-NSF ISE-funded projects; and

 
• Focus groups from six institutions, covering participants in informal science activities.

4. What Is the NSF's Investment in the ISE Program?

Exhibit 5 depicts the percentages of ISE funds that were awarded to a diverse group of organizations and institutions.
 

Children's Television Workshop (CTW), producer of several children's science series, has received $31,506,743 in ISE funds since 1984, more than any other recipient. Exhibit 6 depicts additional recipients that share the largest portion of ISE support. Of the "top 25" institutions listed in Exhibit 6, 36 percent fall into the media category, like CTW, while 40 percent are museums and science and technology centers.
 

5. How Is the ISE Investment Leveraged?

NSF's performance goals recognize the importance of collaboration among business, industry, researchers, schools, and informal science institutions to achieve excellence in SMET. An emphasis on collaboration is seen in the number of museums and science centers that are either project venues or collaborators. Effective partnerships bring together the best minds in this society and also provide fiscal, human, and in-kind resources.

The NSF-funded institutions evaluated used initial ISE funds to leverage additional resources from state and local governments, foundations, local businesses and industries, and other federal agencies. Often, leveraged funds were used to develop additional educational materials or to extend outreach activities to a broad-based audience. In addition, the planning phases of most projects involved obtaining other funds needed to finance project components not funded by the ISE program. Some sites reported that after obtaining ISE funding, other funders cut short their own review processes, relying instead on NSF's peer review processa sign of the confidence that private funders have in the NSF management model.

Directors of the ISE projects state that despite the difficulty experienced in obtaining additional support, few leveraging or cost-sharing activities would occur without the catalyst of partial funding from the ISE program.

For the 10-year period studied, data show that leveraging resulted in national recognition and additional funds. For instance, the Africa Exhibit in Chicago raised 73 percent of the total project costs from the Rockefeller Foundation, the Field Museum (Chicago),the National Endowment for Humanities, and the Joyce Foundation (Chicago). Similarly, Scienceminders, a community-based project, received 63 percent of its project budget from a private investor. Further, leveraging provided additional resources and in-kind support to extend scientific learning through outreach and dissemination activitiessuch as distributing teacher and parent guides, or forming associations with technology-related venues.

Public demand for programs at the Lowell Observatory increased beyond the institution's capabilities. With ISE funding, the observatory expanded public programs for children and families and leveraged additional monies for a visitor center.

-- Site visit report

6. Is the ISE Program Accomplishing Its Goals?

Goal A: Increase the number of youth, especially those from underrepresented or underserved groups (e.g., minorities, persons with disabilities, women), who are excited by science, mathematics, and technology, and who pursue such activities both in and out of school.

ISE media projects include some of the most popular children's series on public television. For instance, the Magic School Bus -- an animated television series created by Scholastic Productions and featuring a female teacher-heroine who involves her class in hands-on science adventures -- is carried by over 300 PBS stations providing coverage to 96 percent of total U.S. households at least once a week. Nielsen ratings in 1995 showed the program to be top ranked among two- to five-year-olds. Reading Rainbow, a series for five- to eight-year-olds that centers each episode on a theme from a featured book, routinely claims similarly large audiences because of its daily distribution.

But even with these impressive numbers, it was still difficult to assess youth excitement about science -- other than inferring it from sheer audience/attendance data. Therefore, most of the data and corresponding conclusions about youth excitement were a result of interviews, observations, document reviews, and focus groups conducted as part of the evaluation team's site visits. For example, young people who participated in the prototype of Testing the Theory at Boston's Museum of Science -- where visitors form hypotheses, perform experiments, collect data, and test their own theories of scientific processes at various workstations -- were still asking questions about their experiences two months later as part of a focus group. When asked if they like school science, they responded with an emphatic "No." When asked what they had done at the exhibit, they immediately answered, "We explored." These young people considered the exhibit to be "fun science" and different from classroom science. Further, participants could all recall the different experiments they had done two months earlier.

Formal evaluation results reviewed during a site visit to the Children's Television Workshop (CTW) demonstrated some gain in excitement by viewers of 3-2-1 Contact, a science and technology series produced by CTW for 8- to 12-year-olds. This excitement was seen through such effects as increases in science interest from pre- to post-testing, a general shift away from the perception of science as boring, improved knowledge of the factual information presented by the program, and interest in pursuing follow-up activities.

As far as reaching more youth in minority groups, females, and persons with disabilities, the most direct evidence comes from those ISE activities designed specifically for that purpose. These activities put more effort into showing how they were trying to reach their targeted audiences. For instance, Explora!, an emerging science center in Albuquerque, New Mexico, had over 70,000 visitors in 1994 (up from 22,000 during the 6 months prior), 47 percent of whom were Hispanic, compared to 35 percent in the local population. Native Americans made up 14 percent of the visitors, compared to 11 percent of the local population, and African Americans comprised 3 percent, compared to 1 percent of the greater Albuquerque population. School visitors to the Tools of the Astronomer exhibit in Flagstaff, Arizona were demographically profiled as 50 percent Native American, 20 percent Hispanic, 20 percent Caucasian, and 10 percent African American. Similarly, staff involved in SERIES -- hands-on curricula for use in schools or community settings -- reported that of 200 participating youth at one site, 50 percent were Hispanic, 30 percent Caucasian, and 20 percent African American.

Goal B: Promote linkages between formal and informal science education.

Goal E: Stimulate collaborations that establish linkages among a variety of organizations and individuals in formal and informal education communities.

Notable, too, are the collaborations with formal education established by ISE projects. Outreach materials and activities developed by the projects have helped to integrate informal science education into the formal science curriculum. Teachers were able to draw on these materials and activities as a creative resource, which they developed and accessed through their participation with ISE projects such as: serving on advisory panels, participating in development and evaluation teams, and writing curricula or outreach materials. Teachers also used materials developed through museum-sponsored workshops to extend concepts learned on field trips. From one year to the next, they also use taped episodes of Magic School Bus and Reading Rainbow, along with accompanying outreach materials, in the classroom.

Scienceminders, a YWCA project, encourages teenage girls' involvement in science. YWCA-sponsored baby-sitting classes include training on how to use informal science activities with young children.

Goal C: Stimulate parents and other adults to be informed advocates for better quality and more accessible science, mathematics, and technology education in both formal and informal settings; encourage them to support their children's science and mathematics endeavors in the home and elsewhere.

Goal D: Enrich the quality of life by improving the science literacy of children and adults so they are better informed about the implications that science, mathematics, and technology have for their everyday lives, thereby enabling them to further pursue science and mathematics experiences and to make informed, responsible decisions about science policy issues with societal implications.

Testing the Theory deliberately tries to spark parent-and-child interaction. The exhibit's staff have overheard excited discussions between parents and children prompted by the questions posted in the exhibit. Similarly, parents participating in focus groups reported that after watching Reading Rainbow, they were not intimidated by science or by helping their children with school science projects. One young participant commented that the program ties science to real lives and everyday things. Five other children at another focus group concurred with that opinion.

Goal F: Have a broad and long-term impact on the informal science education system.

Instead, the most notable differences between NSF-funded and non-NSF-funded activities had to do with overall project planning and strategy. Unlike the non-NSF-funded projects, the ISE program provided a more comprehensive and structured process for its projects. Processes recommended by NSF -- review and revision of proposals, use of advisory panels, formative evaluation, summative evaluation, and dissemination -- were not all found in any of the six non-NSF-funded sites that were visited. Dissemination was largely limited to advertising through fliers or existing information sources. Further, non-NSF-funded projects lacked two key ingredients of NSF projectsleveraging and interdisciplinary topics. Only one project, YouthALIVE!, which serves as the umbrella for all youth programming in a Florida museum, had both of these characteristics. It also was the only project visited that maintained a direct focus on reaching previously underserved populations in museums across the country.

Both adults and children tested hypotheses through exhibits in "Testing the Theory" Observations of visitors in an area dubbed the "Test Tube" revealed that each exhibit appealed to multiple ages. In some cases, the adults led the youth, while in others, the process was reversed. These interactions, unaided by museum staff, involved hypotheses named by both young people and adults, and included the collecting and recording of data.

ISE funds seem to be targeting project activities in the right direction -- that is, toward achieving the six ISE program goals. In emphasizing the roles of museums and science centers as project venues or collaborators, NSF has recognized the importance of these institutions in stimulating children's interest in science. The comments of those in science-related careers attest to this positive influence.
 

7. What Do People Say About ISE-Funded Activities?

ISE project directors were asked questions to ascertain the degree to which their projects involved youth and adults in informal science education and otherwise met program goals. Teachers, scientists, engineers, chemists, and researchers reported on their current activities, as well as on experience from their own youth (e.g., the memorable events and experiences that sparked their interest in science and mathematics). By comparing their responses with ISE project strategies -- many designed to similarly stimulate young audiences -- the evaluation found that the ISE program is headed in the right direction. Questions to organization directors solicited the institutional perspective to determine the influence and impact of ISE funding compared to influences from other funding sources.

Collectively, these responses presented a positive picture of program strategies in promoting linkages with formal science education, involving youth and adults in informal science activities, emphasizing project evaluation, and increasing collaboration among institutions. Therefore, according to the "experts," the ISE program is achieving its goals. The following summarizes their responses to each of these goal areas.

• More linkages with formal science education

On average, 73 percent of surveyed organization directors strongly felt that ISE funds had furthered various aspects of informal science education; whereas only 43 percent said this was true of non-NSF funds.

• Greater youth and adult involvement

Exhibit 8 shows that people with science-related careers reported visits to museums as their most memorable informal science experiences as children. These experiences stimulated their early interest in science, reinforced connections with their formal science classes in school, and implanted science ideas that they still use today (see Exhibits 9 and 10).
 

"After visiting a museum in California, I became very interested in sharks, dinosaurs, and minerals. I went to the library after visiting the museum to learn more."

"Direct observation at nature centers led to [my] first feeling of intense interest and curiosity; I developed a need and an interest in 'knowing' and 'exploring' for answers."

"Visiting museums, zoos, planetaria, and aquaria made me realize how everything is interconnected."

Even though these individuals credited museums and science centers with stimulating their early interest in science, they ranked visits to such venues as only their third most frequent activity during the past six months. This divergence actually lends more credibility to their recollections of childhood experiences as representing real memories and not just contemporary voicing of support for museums.

"Greater emphasis on the evaluation process has led to the training of staff to fill the role of evaluators and has improved the quality of our exhibits... The exacting nature of the NSF ISE proposal process has resulted in a more disciplined approach to project development and grant writing. NSF ISE enabled us to create a high-quality, award-winning traveling exhibit that brings national attention to this institution and an excellent permanent gallery that attracts the general public."

-- Organization director

• More emphasis on evaluation

"The Chicago museums, especially the Museum of Science and Industry, kept alive an interest in science that was never nurtured in elementary, secondary, or higher education. Science in school was a chore. Science in museums was discovery and delight. It still is."

-- Individual with a science-related career

Organization directors also claimed that NSF influencein contrast to other funding sources -- enhanced efforts to publish and disseminate evaluation results from their projects.

• More collaboration

ISE projects' greater collaboration with museums fits well with the program goals of reaching large and diversified audiences, linking informal and formal science education, and stimulating youth excitement. Open-ended responses to survey questions reflected this trend among ISE projects:

"The ISE program has enhanced our involvement with other museums in a way that would probably not have otherwise happened."

"In the museum world, zoos and aquariums are often considered step-children -- not really museums! Or for that matter not really science education venues. NSF funds have allowed us to prove this opinion wrong."
"We now have a very close relationship with the science centers in the state, which has led to other projects."

8. What Are the Policy, Practice, and Program Implications?

Policy Impact

While the evaluation found the ISE program to be meeting its goals, an important discovery was made in the data collection and assessment process:

The ISE program is meeting its goals, but the content emphasis in the ISE portfolio is widely scattered and not reflective of broader NSF priorities in science, mathematics, engineering, and technology (SMET) education.

Interestingly, the shortfall here is not a failure of the program to achieve its goals, but rather a fundamental problem with the goals themselves. The existing ISE program goals -- four of the six evaluation goals -- do not directly address the curriculum content of supported projects. Because the program goals are process goals, they cannot assure that ISE projects will align with any desired NSF content priorities.

The Science Carnival Consortium developed exhibits, demonstrations, and manuals for science museums opening in areas with no viable access to an informal science institution. Resources were used to stimulate and encourage teachers and youth from schools and community groups to visit the museum.

Informal Science Education Practices

The ISE program should revisit the established process and content goals to make them correspond more directly with NSF's current content emphasis in SMET education. It appears the ISE program has successfully reinforced several informal education practices that deserve continued attention and emphasis. Many innovative ISE projects offered activities that expanded or provided new perspectives in science and related fields. Participants were able to carry out scientific procedures or to "experience" science in a hands-on manner, as well as to appreciate the implications of science and mathematics in everyday life. Observed ISE practices also produced high-quality outreach materials and activities that helped to transfer knowledge and awareness of experienced phenomena to the home, community, or school. Many ISE projects produced teacher guides, involved teachers as advisors, and generated materials for classroom use, including books, tapes, videos, software, posters, and teaching kits. Such outreach strategies encourage further pursuit of particular topics, stimulate participation in related activities, and encourage families to reinforce and extend the concepts they learned as participants in an informal science education activity.

Outreach strategies and activities also were used to encourage involvement of underrepresented populations. The success of this endeavor is unknown, however, as few formal data exist to support it. Projects not specifically dedicated to encouraging involvement of underrepresented populations -- while they may have served large numbers of youth and adults -- did not appear to serve distinctively high proportions of minorities or underserved people.

The program should emphasize specific practices which will help to ensure that future projects embody those practices that NSF views as most relevant and integral to science, mathematics, engineering, and technological priorities.

It was easier to assess the projects' informal science education practices in terms of reaching across generations, which they did well, upholding the ISE program's concept of science for everyone of all ages. Many projects, especially those carried out at science centers, used intergenerational activities. If youth were the main target audience, activities were also developed to encourage adult participation. In developing activities that simultaneously engaged all ages, ISE projects stimulated greater family participation and public interest in informal science education venues and, in doing so, strengthened the capacity of the entire ISE program.

Along with project-specific strategies, the current ISE program management strategies also help strengthen the entire program. For example, NSF program officers review preliminary ISE proposals prior to submission. They employ stringent proposal review criteria and look for the inclusion of material such as formative evaluations, advisory committees, and collaborations. Such factors, in turn, increase resources, provide a multidisciplinary focus, and target and increase access for underrepresented groups, and are determinants in deciding awards. Projects passing NSF's "strict scrutiny" may have a higher likelihood of success because they are carefully designed, thoroughly researched, and judged to be high quality. Another capacity-building feature of the program is the support it provides to institutions through initial stages of development, encouraging formative evaluation as part of that process.

ISE Program Administration

The ISE program should more explicitly define for future applicants the informal science practices that it considers essential or high priority. NSF could even provide examples or suggestions of practices best suited for accomplishing each of the program's individual goals instead of focusing on the goals as a whole. NSF also may wish to identify practices for reaching underrepresented populations and identify strategies for better determining their effectiveness. Quality control of projects -- achieved through extensive use of advisory and other technical review committees -- was another element strongly encouraged by the program. At almost every project visited, expert advisory committees were used in a number of ways. Often, several advisory groups -- both internal and external -- were formed. In many cases, separate groups dealt with various project components, such as educational materials. Other advisory committee activities produced collaborations that shaped the project's implementation. Sometimes, the advisory function was to help increase access for underserved populations through devices such as "equity teams."

Another important finding relative to ISE program administration concerns evaluation. While formative evaluations are part of nearly every project -- and might not have been conducted without NSF's encouragement -- summative evaluations are only now being designed and implemented. These evaluations should obtain strong support in the future. Specifically, evaluations need to increase information on the number of adults and children participating in or observing an activity, and on the short- and long-term outcomes of that participation. Again, data on minorities and other underrepresented populations need to be more explicit so as to better assess project impacts in this area.

Also found wanting was the final report information submitted by ISE projects, which tended to cover only administrative or other project processes -- but not SMET education issues and outcomes. Missing were data on participants, lessons learned from formative evaluation, consequent improvements in the quality of an activity or product, and strategies needed to ensure continuance beyond the funding period. Final report guidelines should emphasize the inclusion of outcome, not process data.

In summary, the ISE program should continue requiring many of the good things it requires now: using advisory and technical committees for quality control, encouraging the use of evaluation -- especially summative evaluation -- and focusing on the role of minorities and other underserved populations. The ISE program also should explore ways of making follow-up inquiries to determine later outcomes when funding only the development or production phase of a larger effort. Finally, NSF should better define final report requirements for its ISE program and require data that show whether the project attained its goals. Program staff also should develop a process for maintaining final reports in their files and for using report data in defining future competitions and in responding to policy inquiries.
 

9. Has the Investment Been Worthwhile?

ISE funds have been well used in directing projects' attention toward greater collaboration with a diverse set of institutions and toward more formal use of evaluation to provide formative and summative feedback. In all these aspects -- the role of museums and science centers, the importance of collaboration, and the importance of evaluation -- NSF-funded ISE activities stand apart from those not funded by NSF.
Site visits to NSF-funded ISE projects revealed that most used resources to diversify topics or activities, stretching the NSF investment and audience reach even further. Exhibit projects incorporated teacher kits, take-home materials, or film elements; while radio, television, and film media projects also included books, Internet components, CD-ROMs, and training materials and guidebooks. This variety of activities has two main effects: activities cross the borders of different institutions such as home, school, and informal science settings; and different learning types and different ages are accommodated, i.e., "there's something for everyone."

A number of projects were interdisciplinary as well, an approach that strengthened young peoples' understanding of the relevance of the sciences in our daily lives -- as when one child remarked after viewing an episode of Life by the Numbers, "Math is important, it's everywhere." It also helps participants view other disciplines as both relevant to science and as opportunities to become engaged in hands-on learning. This approach is complementary to the system reform efforts undertaken in other NSF-EHR programs, as well as Goals 2000, which speaks to integrated learning across traditional disciplines, such as mathematics, science, history, and English.

"America's future demands investment in our people, institutions, and ideas. Science is an essential part of this investment, an endless and sustainable re-source with extra-ordinary dividends."

President Clinton and Vice President Gore, Science in the National Interest.

In promoting collaborations, ISE projects extended resources and incorporated a variety of other materials and perspectives that brought science across more borders and to more audiences. The NSF portfolio includes many exhibit consortia which, by their very nature, promote collaboration and continue the themes of expansion and diversity reflected in varied project activities and disciplines.

Finally, NSF strategies have paid off by ensuring that, through summative evaluation and dissemination activities, project materials and lessons live on to guide the community and future projects. For example, NSF-sponsored consortia disseminated books of "lessons learned" to orient new organizations to developing and maintaining science centers. New or recently completed projects have planned for or used summative evaluations. In many ISE projects, however, these evaluations were incomplete or were afforded varying levels of importance. This program area may therefore require more oversight and development. Doing so will ensure that summative evaluations are completed and that results are complete and more useful to the ISE program.
 

References

National Science Foundation, User Friendly Handbook for Project Evaluation: Science, Mathematics, Engineering and Technology Education, Arlington, Virginia, National Science Foundation, 1993, NSF #94-183.

National Science Foundation, National Science Foundation In A Changing World: The National Science Foundation Strategic Plan, Arlington, Virginia, National Science Foundation, 1995, NSF #95-24.

National Science Foundation, Elementary, Secondary, and Informal Education: Program Announcement and Guidelines, Arlington, Virginia, National Science Foundation, 1997, NSF #97-20.
 

The Foundation provides awards for research and education in the sciences and engineering. The awardee is wholly responsible for the conduct of such research and preparation of the results for publication. The Foundation, therefore, does not assume responsibility for the research findings or their interpretation.

The Foundation welcomes proposals from all qualified scientists and engineers and strongly encourages women, minorities, and persons with disabilities to compete fully in any of the research- and education-related programs described here. In accordance with federal statutes, regulations, and NSF policies, no person on grounds of race, color, age, sex, national origin, or disability shall be excluded from participation in, be denied the benefits of, or be subject to discrimination under any program or activity receiving financial assistance from the National Science Foundation.

Facilitation Awards for Scientists and Engineers with Disabilities (FASED) provide funding for special assistance or equipment to enable persons with disabilities (investigators and other staff, including student research assistants) to work on NSF projects. See the program announcement or contact the program coordinator at (703) 306-1636.

The National Science Foundation has TDD (Telephonic Device for the Deaf) capability, which enables individuals with hearing impairment to communicate with the Foundation about NSF programs, employment, or general information. To access NSF TDD, dial (703) 306-0090; for FIRS, 1-800-877-8339.
 
 
 

NSF 98-65