This document has been archived.

Table of Contents
 

A Report on the Evaluation of the

National Science Foundation's

Informal Science Education Program

 

Prepared under Contract RED 94-52970 
by 
COSMOS Corporation

March 1998

Mary Sladek 
Contracting Officer's Technical Representative

Any opinions, findings, conclusions, or recommendations
expressed in this report are those of the participants, and do
not necessarily represent the official views, opinions, or policy
of the National Science Foundation.
 

The National Science Foundation

Directorate for Education and Human Resources 
Division of Research, Evaluation and Communication

 

An REC-sponsored 
Report on Evaluation

 


Preface

This report summarizes the findings of an evaluation of the National Science Foundation's (NSF) Informal Science Education (ISE) program. The program provides support for a variety of informal science education activities, including museum exhibits; television series and programs for youth or the general public; films on science and mathematics topics; exhibits or educational programs at science and natural history museums, science-technology centers, aquaria, nature centers, biological gardens, arboreta, zoological parks, and libraries; and educational programs and activities at community and youth centers. NSF awarded a contract to COSMOS Corporation and its subcontractors, Educational Testing Service, and Westat, Inc., to evaluate the impact of the ISE program. The evaluation was conducted and prepared for the NSF Directorate for Education and Human Resources (EHR), Division of Research, Evaluation and Communication. Mary Sladek, the Contracting Officer's Technical Representative (COTR) for the study, provided general direction and oversight.

Using a question-and-answer format, the report defines informal science education, presents an overview of the ISE evaluation conducted by COSMOS Corporation, and summarizes its major findings and recommendations. The narrative analysis of the data presented in the report draws on results of site visits, focus groups, interviews, and surveys of people with science-related careers and those receiving ISE funds.

The evaluation data were compiled by Dr. Katherine Zantal-Wiener, project director; Dr. Cheryl Sattler; and Dr. Robert K. Yin. Others who assisted were Darnella Davis, Dana Edwards, Suzanne Merchlinsky, and Jennifer Elcano. Two firms, Westat, Inc., and the Educational Testing Service, served as subcontractors and assisted in the design and data collection phases of the task. In addition, Drs. Zoe Barley and Mark Jenness of Western Michigan University gave preliminary advice on the selection of the sample of site visits. The American Association for the Advancement of Science (AAAS) and the National Science Teachers Association (NSTA) provided access to their membership lists to conduct surveys of people with science-related careers.

The project team is grateful for the guidance provided by a specially convened group of advisors who were: Christine Dwyer, Senior Vice President, RMC Research Corporation; Dr. Mark Jenness, Co-Director, Science and Math Program Improvement, Western Michigan University; Nancy Kolb, President and Executive Director, Please Touch Museum; and Dr. Kenneth Phillips, Aerospace Curator, California Museum of Science and Industry. These advisors contributed a unique perspective based on their years of experience in the field of informal science education. Data collection would not have been possible without the assistance of the ISE program staff and the institutions and grantees in the informal science education community. Their cooperation in tracking archival documents, arranging site visits, and recruiting focus group members is greatly appreciated.

This report was prepared under contract RED 94-52970 awarded to COSMOS Corporation. The contents of this document are the responsibility of the authors and do not necessarily reflect the views of the National Science Foundation or its staff.
 


Table of Contents

  Preface Exhibits References


1. What Is Informal Science Education?

Where do people get their understanding of science and mathematics? Exposure to concepts of science, math, engineering, and technology (SMET) education can come from conversations, classrooms, television programs, special events, and other sources. Our understanding is not attributable to a single sourceand is certainly not solely the domain of classrooms and laboratories. Many people are motivated by intrinsic interests outside of formal learning settings. In fact, a common memory of an early science-related experience is likely to be that of an informal science eventvisiting a zoo or science exhibit, seeing a science program on television, talking to a scientist about his or her work, exploring nature in the backyard, or even doing kitchen "experiments."

A major goal of the National Science Foundation (NSF) is to "promote the discovery, integration, dissemination, and employment of new knowledge in service to society" (National Science Foundation, 1995, p. 13, NSF #95-24). One strategy for achieving this goal is to "infuse education with the joy of discovery and an awareness of its connection to exploration."

"Informal science education" is voluntary, self-directed, and lifelong. It is learning that provides an experiential base and motivation for further activity and learning. NSF's Informal Science Education (ISE) program supports projects in which "learning is... motivated mainly by intrinsic interests, curiosity, exploration, manipulation, fantasy, task completion, and social interaction. This informal learning can be linear or nonlinear and often is self-paced and visual- or object-oriented" (National Science Foundation, 1997, p. 8, NSF #97-20).

Informal science learners -- people of all ages, interests, and backgrounds -- are discovering science in places outside of schools, with materials and activities initially not developed for school use or as part of a curriculum. Participants in informal science education activities engage with science on their own initiative and not as part of a mandated school experience. Participation reaps several benefits: a better understanding of concepts, topics, processes, and thinking in scientific and technical discipline; increased knowledge about career opportunities in these fields; and increased appreciation and understanding of science and mathematics, and their applications.

Television series and programs for youth or the general public; films on science and mathematics topics; exhibits or educational programs at science and natural history museums, science-technology centers, aquaria, nature centers, biological gardens, arboreta, zoological parks, and libraries; and educational programs and activities at community and youth centers are all part of informal science education. Cumulative experiences in and through these environments provide interaction and reinforcement, influenced by family, school, peers, and community.

Many people with science-related careers credit their initial interest in SMET to informal rather than formal exposure, identifying museums and science centers as the most important stimulants to their childhood interest.
 

2. What Is NSF's Role in Furthering Informal Science Education?

NSF's Informal Science Education (ISE) program is one of several programs within its Elementary, Secondary, and Informal Education (ESIE) division that work to provide quality learning opportunities in science, mathematics, engineering, and technology (SMET) education. Other programs include the Teacher Enhancement Program, Instructional Materials Development, Presidential Awards for Excellence in Mathematics and Science Teaching, and Advanced Technological Education. These other programs address the needs of students in preK through grade 12, using formal (classroom) education settings to promote quality SMET education. The ISE program is distinctive by fostering collaboration and partnerships among the institutions and organizations previously mentioned that support informal science education, the higher education community, industry, elementary and secondary schools, federal agencies, state and local governments, and other institutions involved in science and engineering. The ISE program develops the capacity and resources to support reform throughout the entire educational system.

All aspects of the ISE program are meant to stimulate and maintain a lifelong interest in math and science. The program strives to:

Related objectives also encourage projects to: The ISE program funds activities that align with these goals. One project among many that incorporates myriad elements is the Magic School Bus, an innovative educational television series for young children that involves the collaboration of individuals or organizations from more than one area. To broaden its impact beyond television viewers, this project established linkages between the series and museums, youth organizations, the Public Broadcasting System (PBS), and a commercial entity (Scholastic Productions). Resulting collaborations have produced teacher and parent guides, traveling exhibits to children's museums and science-technology centers, instructional displays at book fairs, museum activity trunks distributed to more than 100 museums, and youth outreach materials disseminated through national organizations.

Any organizationpublic or private, profit or nonprofitmay submit proposals to provide informal science education experiences. Projects that explore a variety of new and emerging alternatives for reaching large audiences, as well as for improving traditional methods, are encouraged. Such alternatives include projects that use commercial broadcasting, new strategies for interactive exhibits in museums, popular press aimed at specific audiences, newly developed video and interactive learning media, home-learning resources, and home- or community-based adult education projects.

As shown in Exhibit 1, from fiscal year (FY) 1984 to FY1994, ISE funds went mostly to develop television programs and exhibits, with a slightly higher percentage of funds used to support exhibits. The rest of the funds were awarded for activities such as after-school and community programs, professional and staff development, radio programs, school-linked programs, and films.
 

Exhibit 1

DISTRIBUTION1 OF ISE FUNDS ACROSS ACTIVITY AREAS,
FY1984-FY1994

Exhibit 1 - Distribution of ISE Funds Across Activity Areas, FY1984-FY1994
 

3. What Is the "ISE" Evaluation?

This summative evaluation of the ISE program describes the impact of ISE projects funded between FY1984 and FY1994. While "formative" evaluation is designed to improve an activity in the development phase, "summative" evaluation examines the cumulative effects of a program to assess long-term impact. The User-Friendly Handbook for Project Evaluation: Science, Mathematics, Engineering, and Technology Education defines summative evaluation as "designed to present conclusions about the merit or worth of an intervention and recommendations about whether it should be retained, altered, or eliminated" (National Science Foundation, 1993, p. 96, NSF #94-183).

The evaluation was designed to examine the impact of the ISE program from 1984 to 1994, with special emphasis on the period between 1990 and 1994. Past NSF-funded studies have focused on evaluating individual project goals rather than investigating the long-term impact of the ISE program. Furthermore, a review of existing evaluations of informal science education projects revealed an absence of data on the evaluation goals most relevant to the ISE program. For example, one study has examined the support provided by science-technology centers to schools and teachers (ASTC, 1996). In another study, the ISE program was the subject of a pilot study of short-term assessments that were designed to help NSF deal with difficult assessment questions (Knapp, et al., 1988). And neither NSF-funded nor non-NSF-funded informal science education project evaluations have focused on collaborations with community groups (except Ghostwriter, which received an ISE planning grant) or on affecting the informal science education system.

Thus, the design of the present evaluation, conducted by COSMOS Corporation, was based on the presumption that informal science education is not necessarily a set of interventions with short-term impacts but, rather, a means of influencing the context within which science is learned and experienced over the long term. The longer-term impacts that were the focus of the present evaluation can be derived from the evaluation goalsfour ISE program goals (AD) and two additional goals (EF)that were used to assess NSF's entire ISE program.

These goals are listed in Exhibit 2. The evaluation generated aggregate-based outcomes, using a diverse array of data collection methodssite visits, surveys, and secondary analysesto provide an integrated and singular evaluation of program impact.
 

Exhibit 2

SIX GOALS THAT GUIDED THE EVALUATION

 
Four NSF ISE program goals (AD) and two additional goals (EF) guided this evaluation (the two additional goals were identified by NSF as goals to be investigated as part of the evaluation, even though they are not among the program's original goals): 

A. Increase the number of youth, especially those from underrepresented or underserved groups (e.g., minorities, persons with disabilities, women), who are excited by science, mathematics, and technology, and who pursue such activities both in and out of school. 

B. Promote linkages between informal and formal science education. 

C. Stimulate parents and other adults to be informed advocates for better quality and more accessible science, mathematics, and technology education in both formal and informal settings; encourage them to support their children's science and mathematics endeavors in the home and elsewhere. 

D. Enrich the quality of life by improving the science literacy of children and adults so they are better informed about the implications that science, mathematics, and technology have for their everyday lives, thereby enabling them to further pursue science and mathematics experiences and to make informed, responsible decisions about science policy issues with societal implications. 

E. Stimulate collaborations that establish linkages among a variety of organizations and individuals in formal and informal education communities. 

F. Have a broad and long-term impact on the informal science education system. 

 
 

The methodology used for the evaluation was designed from a broad perspective of informal science education, since outcomes such as increased scientific literacy cannot be attributed to a single project. Therefore, instead of using the typical linear input-output approach, the study methodology acknowledged that the ISE program is influenced by family, community, and formal educationthat is, by the context in which informal science education activities take place.

The evaluation framework in Exhibit 3 incorporated the six evaluation goals (AF), while respecting the contextual nature of informal science educationconsidering it in the aggregate and not on a project-by-project basis. The design thereby provides a comparative logic for interpreting and comparing evaluation resultsthe comparison of ISE program activities and outcomes with those activities not funded by NSF. Similarly, responses by people directly associated with ISE projects were compared with those of people in science careers. The design also considered NSF's cost-sharing requirements and the fact that ISE funding is supplemented by other funds from federal, state, and local agencies, corporations, or foundations.
 

Exhibit 3

CONTEXTUAL CONDITIONS AND THE EVALUATION GOALS OF 
THE NSF'S INFORMAL SCIENCE EDUCATION PROGRAM

Exhibit 3 - Contextual Conditions and the Evaluation Goals of the NSF's Informal Science Education Program
 

Because it is difficult to disentangle co-funded activities, informal science education projects and activities not funded by the NSF were studied as a comparison group. Three caveats about the comparison group should be noted. First, the projects in the comparison group have characteristics similar to, but not exactly like, those funded by NSF. Second, the comparison projects could have been influenced by the ISE program guidelines if: 1) the organization had submitted an earlier proposal that was reviewed by the ISE program staff and incorporated the reviewers' comments into a later proposal that may have been funded by a non-NSF entity or even NSF; or 2) the organization may have received a prior ISE grant and used the reviewers' comments to report on the development of a non-NSF project. Third, the project may have received ISE funds after the data collection commenced. Moreover, because of the difficulty in identifying informal science education projects not funded by NSF, the final selection was reviewed with the ISE program staff and advisors.

To the extent possible, parallel data were collected from both ISE projects and non-NSF informal science education projects. Data were collected from the following sources:

ISE program activities:

Non-NSF-funded sources: Funded projects were sorted into four project types: media, science centers and museums, professional development, and community based.
 

4. What Is the NSF's Investment in the ISE Program?

ISE program funding decisions are guided by the four program goals identified earlier. As seen in Exhibit 4, funding has increased steadily over the program's history, with two major increases totaling $18.5 million occurring between FY1990 and FY1993.
 

Exhibit 4

NSF'S INFORMAL SCIENCE EDUCATION FUNDING, FY1984-FY1994

Exhibit 4 - NSF's Informal Science Education Funding, FY1984-FY1994
 

Exhibit 5 depicts the percentages of ISE funds that were awarded to a diverse group of organizations and institutions.
 

Exhibit 5

DISTRIBUTION1 OF ISE FUNDS ACROSS INSTITUTION TYPE, FY1984-FY1994

Exhibit 5 - Distribution of ISE Funds Across Institution Type, FY1984-FY1994
 

Children's Television Workshop (CTW), producer of several children's science series, has received $31,506,743 in ISE funds since 1984, more than any other recipient. Exhibit 6 depicts additional recipients that share the largest portion of ISE support. Of the "top 25" institutions listed in Exhibit 6, 36 percent fall into the media category, like CTW, while 40 percent are museums and science and technology centers.
 

Exhibit 6

TWENTY-FIVE INSTITUTIONS RECEIVING THE GREATEST AMOUNT OF SUPPORT FROM NSF'S INFORMAL SCIENCE EDUCATION PROGRAM, FY1984-FY1994

Institution
$ Amount
of Support
Number of
Awards
Children's Television Workshop (New York City) 
31,504,743
9
Exploratorium (San Francisco)
6,896,034
11
Franklin Institute (Philadelphia)
4,992,302
11
National Public Radio (Washington, D.C.)
4,679,663
4
Scholastic Productions (New York City)
4,660,647
1
Field Museum of Natural History (Chicago)
4,405,938
4
New York Hall of Science (Corona, New York)
4,121,628
8
Museum of Science (Boston)
3,886,634
8
WGBH (Boston)
3,767,035
13
Oregon Museum of Science and Industry (Portland)
3,430,481
4
Museum of Science and Industry (Chicago)
3,429,628
3
QED Communications (Pittsburgh)
3,266,044
5
Science Museum of Minnesota (St. Paul)
3,061,183
5
KCTS Associates (Seattle)
2,977,824
3
Pacific Science Center (Seattle)
2,665,545
3
Nebraskans for Public TV (Lincoln)
2,653,000
4
American Association for the Advancement of Science (Washington, D.C.)
2,542,541
5
Cornell University (Ithaca, New York)
2,499,017
5
Association of Science and Technology Centers (Washington, D.C.)
2,477,382
11
Girls, Inc. (New York City)
2,399,081
3
Prism Productions (Acampo, California)
2,289,267
2
WNET (New York City)
2,167,536
4
Children's Museum (Boston)
2,167,536
5
Center of Science & Industry (Columbus, Ohio)
1,875,938
3
Science Museum of Charlotte (Charlotte, North Carolina)
1,846,756
3
Source: NSF ISE Database 
 

5. How Is the ISE Investment Leveraged?

It is important to note that in all the projects evaluated, ISE funding awards represented only a fraction of a project's overall costa condition that speaks to NSF's cost-sharing, or leveraging, requirements. In fact, NSF views itself as a "catalyst" to project development. Its investment role is designed to strengthen and promote the nation's capabilities in SMET areas by providing the tools, the programs, and the funding to help others pursue the frontiers of science education and learning.

NSF's performance goals recognize the importance of collaboration among business, industry, researchers, schools, and informal science institutions to achieve excellence in SMET. An emphasis on collaboration is seen in the number of museums and science centers that are either project venues or collaborators. Effective partnerships bring together the best minds in this society and also provide fiscal, human, and in-kind resources.

The NSF-funded institutions evaluated used initial ISE funds to leverage additional resources from state and local governments, foundations, local businesses and industries, and other federal agencies. Often, leveraged funds were used to develop additional educational materials or to extend outreach activities to a broad-based audience. In addition, the planning phases of most projects involved obtaining other funds needed to finance project components not funded by the ISE program. Some sites reported that after obtaining ISE funding, other funders cut short their own review processes, relying instead on NSF's peer review processa sign of the confidence that private funders have in the NSF management model.

Directors of the ISE projects state that despite the difficulty experienced in obtaining additional support, few leveraging or cost-sharing activities would occur without the catalyst of partial funding from the ISE program.

For the 10-year period studied, data show that leveraging resulted in national recognition and additional funds. For instance, the Africa Exhibit in Chicago raised 73 percent of the total project costs from the Rockefeller Foundation, the Field Museum (Chicago),the National Endowment for Humanities, and the Joyce Foundation (Chicago). Similarly, Scienceminders, a community-based project, received 63 percent of its project budget from a private investor. Further, leveraging provided additional resources and in-kind support to extend scientific learning through outreach and dissemination activitiessuch as distributing teacher and parent guides, or forming associations with technology-related venues.

6. Is the ISE Program Accomplishing Its Goals?

The evaluation revealed that, to varying degrees, the ISE program is meeting all of its expressed goals. These achievements, stated as bulleted headings at the end of this section, are followed by supporting evidence obtained from site visits, focus groups, existing evaluations, and surveys. Many ISE projects become popular with the public. Available attendance records and viewer ratings indicate that large numbers of young people are attracted to these exhibits and public television programs. For instance, What Makes Music?, a traveling exhibit for children and adults about the physics of sound and music, attracted an estimated 2.2 million people to the 18 museums that showed it from 1988 to 1995. The exhibit, which attracted over 160,000 visitors during a 78-day run at The Franklin Institute Science Museum in Philadelphia, was especially popular with school groups. The Tools of the Astronomer exhibit at the Lowell Observatory in Flagstaff, Arizona, is having a similar effect. The exhibit draws nearly 80,000 visitors yearly, compared to the 6,000 visitors that came prior to its opening. Lowell Observatory is a fully operational, private planetary research organization. Tools of the Astronomer is a set of interactive exhibits designed and constructed by Lowell astronomers and program staff to teach both the history and modern concepts of astronomical research in an informal science education setting.

ISE media projects include some of the most popular children's series on public television. For instance, the Magic School Bus -- an animated television series created by Scholastic Productions and featuring a female teacher-heroine who involves her class in hands-on science adventures -- is carried by over 300 PBS stations providing coverage to 96 percent of total U.S. households at least once a week. Nielsen ratings in 1995 showed the program to be top ranked among two- to five-year-olds. Reading Rainbow, a series for five- to eight-year-olds that centers each episode on a theme from a featured book, routinely claims similarly large audiences because of its daily distribution.

But even with these impressive numbers, it was still difficult to assess youth excitement about science -- other than inferring it from sheer audience/attendance data. Therefore, most of the data and corresponding conclusions about youth excitement were a result of interviews, observations, document reviews, and focus groups conducted as part of the evaluation team's site visits. For example, young people who participated in the prototype of Testing the Theory at Boston's Museum of Science -- where visitors form hypotheses, perform experiments, collect data, and test their own theories of scientific processes at various workstations -- were still asking questions about their experiences two months later as part of a focus group. When asked if they like school science, they responded with an emphatic "No." When asked what they had done at the exhibit, they immediately answered, "We explored." These young people considered the exhibit to be "fun science" and different from classroom science. Further, participants could all recall the different experiments they had done two months earlier.

Formal evaluation results reviewed during a site visit to the Children's Television Workshop (CTW) demonstrated some gain in excitement by viewers of 3-2-1 Contact, a science and technology series produced by CTW for 8- to 12-year-olds. This excitement was seen through such effects as increases in science interest from pre- to post-testing, a general shift away from the perception of science as boring, improved knowledge of the factual information presented by the program, and interest in pursuing follow-up activities.

As far as reaching more youth in minority groups, females, and persons with disabilities, the most direct evidence comes from those ISE activities designed specifically for that purpose. These activities put more effort into showing how they were trying to reach their targeted audiences. For instance, Explora!, an emerging science center in Albuquerque, New Mexico, had over 70,000 visitors in 1994 (up from 22,000 during the 6 months prior), 47 percent of whom were Hispanic, compared to 35 percent in the local population. Native Americans made up 14 percent of the visitors, compared to 11 percent of the local population, and African Americans comprised 3 percent, compared to 1 percent of the greater Albuquerque population. School visitors to the Tools of the Astronomer exhibit in Flagstaff, Arizona were demographically profiled as 50 percent Native American, 20 percent Hispanic, 20 percent Caucasian, and 10 percent African American. Similarly, staff involved in SERIES -- hands-on curricula for use in schools or community settings -- reported that of 200 participating youth at one site, 50 percent were Hispanic, 30 percent Caucasian, and 20 percent African American.

The diversity and array of linkages and their effect on exciting youth and helping adults and families become better informed are impressive. These linkages include collaborations involving media, university, educational, professional, and community organizations. Linkages add a multidisciplinary aspect to projects and provide them with resources that are often too costly to purchase otherwise.

Notable, too, are the collaborations with formal education established by ISE projects. Outreach materials and activities developed by the projects have helped to integrate informal science education into the formal science curriculum. Teachers were able to draw on these materials and activities as a creative resource, which they developed and accessed through their participation with ISE projects such as: serving on advisory panels, participating in development and evaluation teams, and writing curricula or outreach materials. Teachers also used materials developed through museum-sponsored workshops to extend concepts learned on field trips. From one year to the next, they also use taped episodes of Magic School Bus and Reading Rainbow, along with accompanying outreach materials, in the classroom.

Products and materials produced by ISE projects, especially hands-on activities, also provide linkages to professional development activities. Examples include SERIES, which provided curriculum training for use by the Salvation Army, Public Housing Authority, Boy Scouts, Campfire Girls, Fish and Game Commission, and the National Association of Environmental Education; Magic School Bus and Reading Rainbow, whose materials were used in the National Teacher Training Institute; and the Science Carnival Consortium, which provided emerging science centers with exhibits and demonstrations. The evaluation found that many of the ISE activities were designed to encourage adult participation and family involvement. In Tools of the Astronomer, workshops were set up specifically for family activities, helping to increase the number of Friends of Lowell Observatory from 200 to more than 1,000 during a five-year period, as well as increase contributions to the observatory's discretionary budget by about $100,000. The adoption of the SERIES curriculum by local civic organizations such as 4-H groups has helped reverse the negative attitudes of migrant farm workers toward their children's participation in such groups, as the parents now understand the importance of science to their children's education and future. The SERIES program has also had success in "turning around" troubled children. In one case reported by the series project director, the school attendance and behavior of four urban youth improved after they were selected to work with fourth graders.

Testing the Theory deliberately tries to spark parent-and-child interaction. The exhibit's staff have overheard excited discussions between parents and children prompted by the questions posted in the exhibit. Similarly, parents participating in focus groups reported that after watching Reading Rainbow, they were not intimidated by science or by helping their children with school science projects. One young participant commented that the program ties science to real lives and everyday things. Five other children at another focus group concurred with that opinion.

Evidence for meeting the sixth goal is found in the collective achievement of the other five goals. Such an inference, though, must be qualified by noting that a comparison with non-NSF-funded sources through site visits and existing evaluations did not reveal strong differences in the quality and type of activities, objectives, and outcomes.

Instead, the most notable differences between NSF-funded and non-NSF-funded activities had to do with overall project planning and strategy. Unlike the non-NSF-funded projects, the ISE program provided a more comprehensive and structured process for its projects. Processes recommended by NSF -- review and revision of proposals, use of advisory panels, formative evaluation, summative evaluation, and dissemination -- were not all found in any of the six non-NSF-funded sites that were visited. Dissemination was largely limited to advertising through fliers or existing information sources. Further, non-NSF-funded projects lacked two key ingredients of NSF projectsleveraging and interdisciplinary topics. Only one project, YouthALIVE!, which serves as the umbrella for all youth programming in a Florida museum, had both of these characteristics. It also was the only project visited that maintained a direct focus on reaching previously underserved populations in museums across the country.

In addition, a smaller variety of activities was included within each project than was found in NSF-funded projects -- the project investments appeared to be more limited. A significant number of non-NSF-funded projects, however, did include collaborations with other institutions, possibly prompted by resource limitations.

ISE funds seem to be targeting project activities in the right direction -- that is, toward achieving the six ISE program goals. In emphasizing the roles of museums and science centers as project venues or collaborators, NSF has recognized the importance of these institutions in stimulating children's interest in science. The comments of those in science-related careers attest to this positive influence.
 

7. What Do People Say About ISE-Funded Activities?

To find out how various aspects of the ISE program were viewed by people with careers in science, ISE grantee organizations, and project directors of ISE projects and activities, questions were posed in three different surveys targeting each group. The surveys provided data on how the ISE program has influenced the personal and professional science experiences and involvement of these three groups.

ISE project directors were asked questions to ascertain the degree to which their projects involved youth and adults in informal science education and otherwise met program goals. Teachers, scientists, engineers, chemists, and researchers reported on their current activities, as well as on experience from their own youth (e.g., the memorable events and experiences that sparked their interest in science and mathematics). By comparing their responses with ISE project strategies -- many designed to similarly stimulate young audiences -- the evaluation found that the ISE program is headed in the right direction. Questions to organization directors solicited the institutional perspective to determine the influence and impact of ISE funding compared to influences from other funding sources.

Collectively, these responses presented a positive picture of program strategies in promoting linkages with formal science education, involving youth and adults in informal science activities, emphasizing project evaluation, and increasing collaboration among institutions. Therefore, according to the "experts," the ISE program is achieving its goals. The following summarizes their responses to each of these goal areas.

Asked to compare NSF's influence with that of other funding sources, organization directors said NSF programs, compared to non-NSF-funded programs, netted more public participation, evaluation, and linkages with other organizations -- strongly suggesting the influence of the ISE program on the informal science education system (see Exhibit 7).

Exhibit 7

INFLUENCE OF ISE FUNDS ON THE INFORMAL SCIENCE EDUCATION SYSTEM COMPARED TO NON-NSF FUNDS, AS REPORTED BY ORGANIZATION DIRECTORS

Aspect of Informal Science Education
Directors Indicating "Strongly Agree"
ISE Influence
Non-NSF Influence
No.1
%
No.1
%
Increased public participation
50
83
22
56
Used evaluation for program or project improvement
47
83
16
40
Collaborated or linked with other organizations
47
78
19
48
Reached previously underserved population
45
76
21
51
Included new fields of science in public awareness
42
73
15
38
Utilized new technologies
42
72
17
43
Collaborated or linked with informal science education
41
70
17
41
Collaborated or linked with formal education
37
66
16
37
Increased public support
37
63
13
33
Used technical advisors for planning the product, exhibit, or activity
36
63
15
38
1Multiple responses permissible. 
Source: Survey of Directors of Organizations Receiving ISE Funds, COSMOS Corporation (1996). 
  Both ISE project directors and individuals with science-related careers rated the same two activities as most successful in involving youth and adults in informal science: school visits or outreach programs and program coordination with organizations (e.g., Girl Scouts). Individuals in science-related careers found presentations at summer camps -- not an NSF ISE activity -- to be successful, even though science camps were infrequently used to involve youth and adults.

Exhibit 8 shows that people with science-related careers reported visits to museums as their most memorable informal science experiences as children. These experiences stimulated their early interest in science, reinforced connections with their formal science classes in school, and implanted science ideas that they still use today (see Exhibits 9 and 10).
 

Exhibit 8

MOST MEMORABLE INFORMAL EDUCATION ACTIVITIES FROM THEIR CHILDHOOD, AS REPORTED BY PEOPLE WITH SCIENCE CAREERS

Type of Informal Science Education Activity 
People with 
Science-related 
Careers 
(n = 254) 
%
Visiting a planetarium, aquarium, or zoo 92.9
Visiting a science or natural history museum 85
Having a science-related hobby or science toys 74
Reading science books or magazines for fun 72.8
Watching science shows on TV or listening to science programs on the radio 65
Visiting a botanical garden 58.3
Participating in science fairs 41.7
Participating in a community-sponsored youth program involving science 21.3
Source: Survey of People With Science Careers, COSMOS Corporation (1996).
 

Exhibit 9

SOURCES OF IDEAS LEARNED IN YOUTH THAT ARE STILL USED, AS REPORTED BY PEOPLE WITH SCIENCE CAREERS

Type of Informal Science Education Activity 
People with 
Science-related 
Careers 
(n = 244) 
%
Visiting science centers, museums, or exhibits 51.6
Watching or listening to media programs about science (on TV, films, radio) 44.7
Participating in community youth programs about science 11.5
Source: Survey of People With Science Careers, COSMOS Corporation (1996).
 

Exhibit 10

EARLY INFORMAL SCIENCE EDUCATION ACTIVITIES THAT INITIATED CONNECTIONS WITH SCHOOLS, AS REPORTED BY PEOPLE WITH SCIENCE CAREERS

Type of Informal Science Activity 
People with 
Science-related 
Careers 
(n = 250) 
%
Things I saw at science centers, museums, or exhibits 50.0
Science programs I watched on TV or listened to on the radio 44.8
Activities I did while participating in community youth programs about science 13.6
Source: Survey of People With Science Careers, COSMOS Corporation (1996).
 
Therefore, the program's targeting of science centers and museums -- a distinguishing feature -- focuses on the institutional arena that those in science careers say most stimulated their excitement for the sciences. In their own words, they state:

"After visiting a museum in California, I became very interested in sharks, dinosaurs, and minerals. I went to the library after visiting the museum to learn more."

"Direct observation at nature centers led to [my] first feeling of intense interest and curiosity; I developed a need and an interest in 'knowing' and 'exploring' for answers."

"Visiting museums, zoos, planetaria, and aquaria made me realize how everything is interconnected."

Even though these individuals credited museums and science centers with stimulating their early interest in science, they ranked visits to such venues as only their third most frequent activity during the past six months. This divergence actually lends more credibility to their recollections of childhood experiences as representing real memories and not just contemporary voicing of support for museums.

The NSF influence appears to be significant. For example, people with science-related careers reported a correlation between their current informal science education activities and those prompted by NSF. The activities identified for each category fell in about the same sequential order. Topping both lists was "encouraging friends to attend science-related programs." As shown in Exhibit 7, NSF-funded organization directors noted the emphasis on evaluation that came with NSF funds, compared to non-NSF funds. The following is representative of how a number of respondents described this emphasis: "Staff have adopted an evaluation system literally for every existing and new program activity, label, publication, exhibit, etc. [NSF funds] have expanded virtually every area of education and communication and forced/directed us to complete other critical plans, e.g., signage master plan, etc. Other sources have expanded our educational opportunities offered, but not nearly to the 'depth or width' or extent of the NSF project."

Organization directors also claimed that NSF influencein contrast to other funding sources -- enhanced efforts to publish and disseminate evaluation results from their projects.

Directors from diverse types of organizations receiving ISE support claimed that ISE funds promoted collaborations with museums or science centers, previously noted as having been most important in stimulating early interest in science. In contrast, museums were only the fourth category of collaborations influenced by non-NSF funding (see Exhibit 11).

ISE projects' greater collaboration with museums fits well with the program goals of reaching large and diversified audiences, linking informal and formal science education, and stimulating youth excitement. Open-ended responses to survey questions reflected this trend among ISE projects:

"The ISE program has enhanced our involvement with other museums in a way that would probably not have otherwise happened."

"In the museum world, zoos and aquariums are often considered step-children -- not really museums! Or for that matter not really science education venues. NSF funds have allowed us to prove this opinion wrong."
"We now have a very close relationship with the science centers in the state, which has led to other projects."

Exhibit 11

INFLUENCE OF ISE FUNDS AS COMPARED TO NON-NSF FUNDS ON INCREASING COLLABORATION, AS REPORTED BY ORGANIZATIONAL DIRECTORS (n=179)

Collaborator Type
Respondents Indicating Increased Collaborations
NSF Influence
Non-NSF Influence
No.1
%
No.1
%
Museum
40
69
28
58
Elementary school
35
64
33
71
Community-based science or math education program
35
63
22
51
Individual researcher
35
63
16
35
Private business or industry
33
59
30
66
College or university
29
54
22
49
Youth organization
29
50
30
62
Secondary school
25
46
26
57
Social service organization
22
39
19
41
TV or radio station
21
38
17
38
Day care center
8
14
12
25
Faith institution
7
12
6
13
1Multiple responses permissible. 
Source: Survey of Directors of Organizations Receiving ISE Funds, COSMOS Corporation (1996).

8. What Are the Policy, Practice, and Program Implications?

Based on the evidence presented in this report, the major conclusion from this summative evaluation is that the ISE program is meeting the intended goals and is having an impact on its target population. The ISE program can make even greater contributions by reassessing and possibly modifying the goals. To organize the results, the findings and recommendations were grouped into three main areas: policy impact, informal science education practices, and ISE program administration.

Policy Impact

While the evaluation found the ISE program to be meeting its goals, an important discovery was made in the data collection and assessment process:

As shown in Exhibit 12, an analysis of ISE projects funded from FY1990 to FY1996 reveals few projects focusing on: 1) mathematics -- only 5.4 percent of the total -- need to be given more prominence because of mathematics' critical role for all of science; and 2) computer sciences, engineering, and technology -- together only 5.1 percent of the total -- again, are not reflective of the attention devoted to these areas nationally in science, mathematics, engineering, and technology education. Moreover, in today's society both mathematics and computer science serve as gateways to all other sciences and disciplines.

Interestingly, the shortfall here is not a failure of the program to achieve its goals, but rather a fundamental problem with the goals themselves. The existing ISE program goals -- four of the six evaluation goals -- do not directly address the curriculum content of supported projects. Because the program goals are process goals, they cannot assure that ISE projects will align with any desired NSF content priorities.

Exhibit 12

TOPICAL DISTRIBUTION OF ISE PROJECTS1
FY1990-FY1996
(n=310)

General Topic
Number 
Science
277
Mathematics
17
Computer Science, Engineering, and Technology2
16
1The topical fields were generated by the EHR Database. To verify the topical classifications, all project abstracts were read and classifications were changed if necessary. After all classifications were verified, the ISE project officers reviewed the classifications for accuracy. 
2The EHR database does not separate these three fields. 
  Not surprisingly, for the period under evaluation, the ISE program announcement and formal guidelines did not explicitly signal the desired content priorities for proposed projects. To close the resulting gap in the future, NSF should specify the desired project direction and content in its communication to applicants. NSF would thereby foster applications and awards that better and more surely complement its broader goals in improving SMET education. The NSF program should also emphasize more traditional fields (such as mathematics or computer science), as well as newer fields (such as materials science). Such enhancement would give the program the potential to play an even more effective role in the future.

Informal Science Education Practices

The ISE program should revisit the established process and content goals to make them correspond more directly with NSF's current content emphasis in SMET education. It appears the ISE program has successfully reinforced several informal education practices that deserve continued attention and emphasis. Many innovative ISE projects offered activities that expanded or provided new perspectives in science and related fields. Participants were able to carry out scientific procedures or to "experience" science in a hands-on manner, as well as to appreciate the implications of science and mathematics in everyday life. Observed ISE practices also produced high-quality outreach materials and activities that helped to transfer knowledge and awareness of experienced phenomena to the home, community, or school. Many ISE projects produced teacher guides, involved teachers as advisors, and generated materials for classroom use, including books, tapes, videos, software, posters, and teaching kits. Such outreach strategies encourage further pursuit of particular topics, stimulate participation in related activities, and encourage families to reinforce and extend the concepts they learned as participants in an informal science education activity.

Outreach strategies and activities also were used to encourage involvement of underrepresented populations. The success of this endeavor is unknown, however, as few formal data exist to support it. Projects not specifically dedicated to encouraging involvement of underrepresented populations -- while they may have served large numbers of youth and adults -- did not appear to serve distinctively high proportions of minorities or underserved people.

The program should emphasize specific practices which will help to ensure that future projects embody those practices that NSF views as most relevant and integral to science, mathematics, engineering, and technological priorities.

It was easier to assess the projects' informal science education practices in terms of reaching across generations, which they did well, upholding the ISE program's concept of science for everyone of all ages. Many projects, especially those carried out at science centers, used intergenerational activities. If youth were the main target audience, activities were also developed to encourage adult participation. In developing activities that simultaneously engaged all ages, ISE projects stimulated greater family participation and public interest in informal science education venues and, in doing so, strengthened the capacity of the entire ISE program.

Along with project-specific strategies, the current ISE program management strategies also help strengthen the entire program. For example, NSF program officers review preliminary ISE proposals prior to submission. They employ stringent proposal review criteria and look for the inclusion of material such as formative evaluations, advisory committees, and collaborations. Such factors, in turn, increase resources, provide a multidisciplinary focus, and target and increase access for underrepresented groups, and are determinants in deciding awards. Projects passing NSF's "strict scrutiny" may have a higher likelihood of success because they are carefully designed, thoroughly researched, and judged to be high quality. Another capacity-building feature of the program is the support it provides to institutions through initial stages of development, encouraging formative evaluation as part of that process.

ISE Program Administration

The ISE program should more explicitly define for future applicants the informal science practices that it considers essential or high priority. NSF could even provide examples or suggestions of practices best suited for accomplishing each of the program's individual goals instead of focusing on the goals as a whole. NSF also may wish to identify practices for reaching underrepresented populations and identify strategies for better determining their effectiveness. Quality control of projects -- achieved through extensive use of advisory and other technical review committees -- was another element strongly encouraged by the program. At almost every project visited, expert advisory committees were used in a number of ways. Often, several advisory groups -- both internal and external -- were formed. In many cases, separate groups dealt with various project components, such as educational materials. Other advisory committee activities produced collaborations that shaped the project's implementation. Sometimes, the advisory function was to help increase access for underserved populations through devices such as "equity teams."

Another important finding relative to ISE program administration concerns evaluation. While formative evaluations are part of nearly every project -- and might not have been conducted without NSF's encouragement -- summative evaluations are only now being designed and implemented. These evaluations should obtain strong support in the future. Specifically, evaluations need to increase information on the number of adults and children participating in or observing an activity, and on the short- and long-term outcomes of that participation. Again, data on minorities and other underrepresented populations need to be more explicit so as to better assess project impacts in this area.

Also found wanting was the final report information submitted by ISE projects, which tended to cover only administrative or other project processes -- but not SMET education issues and outcomes. Missing were data on participants, lessons learned from formative evaluation, consequent improvements in the quality of an activity or product, and strategies needed to ensure continuance beyond the funding period. Final report guidelines should emphasize the inclusion of outcome, not process data.

In summary, the ISE program should continue requiring many of the good things it requires now: using advisory and technical committees for quality control, encouraging the use of evaluation -- especially summative evaluation -- and focusing on the role of minorities and other underserved populations. The ISE program also should explore ways of making follow-up inquiries to determine later outcomes when funding only the development or production phase of a larger effort. Finally, NSF should better define final report requirements for its ISE program and require data that show whether the project attained its goals. Program staff also should develop a process for maintaining final reports in their files and for using report data in defining future competitions and in responding to policy inquiries.
 

9. Has the Investment Been Worthwhile?

Perhaps the best indicator of whether NSF's investment in its ISE program has paid off is the apparent effect of greater exposure to informal science education opportunities -- that is, more people are becoming more "comfortable" with science. By offering varied activities across multiple disciplines, reaching diverse populations, and effecting advantageous collaborations with other organizations, ISE projects have been able to produce a positive change in people's attitudes toward science and related topics.

ISE funds have been well used in directing projects' attention toward greater collaboration with a diverse set of institutions and toward more formal use of evaluation to provide formative and summative feedback. In all these aspects -- the role of museums and science centers, the importance of collaboration, and the importance of evaluation -- NSF-funded ISE activities stand apart from those not funded by NSF.
Site visits to NSF-funded ISE projects revealed that most used resources to diversify topics or activities, stretching the NSF investment and audience reach even further. Exhibit projects incorporated teacher kits, take-home materials, or film elements; while radio, television, and film media projects also included books, Internet components, CD-ROMs, and training materials and guidebooks. This variety of activities has two main effects: activities cross the borders of different institutions such as home, school, and informal science settings; and different learning types and different ages are accommodated, i.e., "there's something for everyone."

A number of projects were interdisciplinary as well, an approach that strengthened young peoples' understanding of the relevance of the sciences in our daily lives -- as when one child remarked after viewing an episode of Life by the Numbers, "Math is important, it's everywhere." It also helps participants view other disciplines as both relevant to science and as opportunities to become engaged in hands-on learning. This approach is complementary to the system reform efforts undertaken in other NSF-EHR programs, as well as Goals 2000, which speaks to integrated learning across traditional disciplines, such as mathematics, science, history, and English.

Another theme driving ISE program success and found throughout the projects was a concern for the inclusion of diverse groups in as many ways as possible -- as focus group participants, movie and television characters, subjects of exhibit pictures, and project participants. For example, Reading Rainbow uses an African American male, and Magic School Bus uses a creative female teacher and multi-ethnic youth to break down the stereotypical images of who is interested in science and to demonstrate that science pervades our daily lives. By diversifying the idea of who scientists are and involving students in interactive and self-directed exploration, these projects allow youth and adults to picture themselves doing science. The SERIES project specifically noted this outcome: youth in focus groups reported that their younger female students, when asked to draw a scientist after participating in SERIES, drew themselves.

In promoting collaborations, ISE projects extended resources and incorporated a variety of other materials and perspectives that brought science across more borders and to more audiences. The NSF portfolio includes many exhibit consortia which, by their very nature, promote collaboration and continue the themes of expansion and diversity reflected in varied project activities and disciplines.

Finally, NSF strategies have paid off by ensuring that, through summative evaluation and dissemination activities, project materials and lessons live on to guide the community and future projects. For example, NSF-sponsored consortia disseminated books of "lessons learned" to orient new organizations to developing and maintaining science centers. New or recently completed projects have planned for or used summative evaluations. In many ISE projects, however, these evaluations were incomplete or were afforded varying levels of importance. This program area may therefore require more oversight and development. Doing so will ensure that summative evaluations are completed and that results are complete and more useful to the ISE program.
 

References

Knapp, Michael S., M. St. John, A. Zucker, P.M. Shields, M. Stearns, T. Middleton, and D.M. Shaver, An Approach to Assessing Initiatives in Science Education, SRI International, Menlo Park, California, 1988.

National Science Foundation, User Friendly Handbook for Project Evaluation: Science, Mathematics, Engineering and Technology Education, Arlington, Virginia, National Science Foundation, 1993, NSF #94-183.

National Science Foundation, National Science Foundation In A Changing World: The National Science Foundation Strategic Plan, Arlington, Virginia, National Science Foundation, 1995, NSF #95-24.

National Science Foundation, Elementary, Secondary, and Informal Education: Program Announcement and Guidelines, Arlington, Virginia, National Science Foundation, 1997, NSF #97-20.
 


The Foundation provides awards for research and education in the sciences and engineering. The awardee is wholly responsible for the conduct of such research and preparation of the results for publication. The Foundation, therefore, does not assume responsibility for the research findings or their interpretation.

The Foundation welcomes proposals from all qualified scientists and engineers and strongly encourages women, minorities, and persons with disabilities to compete fully in any of the research- and education-related programs described here. In accordance with federal statutes, regulations, and NSF policies, no person on grounds of race, color, age, sex, national origin, or disability shall be excluded from participation in, be denied the benefits of, or be subject to discrimination under any program or activity receiving financial assistance from the National Science Foundation.

Facilitation Awards for Scientists and Engineers with Disabilities (FASED) provide funding for special assistance or equipment to enable persons with disabilities (investigators and other staff, including student research assistants) to work on NSF projects. See the program announcement or contact the program coordinator at (703) 306-1636.

The National Science Foundation has TDD (Telephonic Device for the Deaf) capability, which enables individuals with hearing impairment to communicate with the Foundation about NSF programs, employment, or general information. To access NSF TDD, dial (703) 306-0090; for FIRS, 1-800-877-8339.
 
 
 

NSF 98-65