Survey Overview (FY 2016 survey cycle)

Top of Page.
Purpose.

The Nonprofit Research Activities (NPRA) Survey collects information on research and experimental development (R&D) performed or funded by nonprofit organizations in the United States.

Data collection authority.

The information is solicited under the authority of the National Science Foundation Act of 1950, as amended, and the America COMPETES Reauthorization Act of 2010. The Office of Management and Budget control number is 3145–0240 and expired on 28 February 2021. The survey was conducted by the National Center for Science and Engineering Statistics within the National Science Foundation.

Major changes to the recent survey cycle.

Not applicable.

Key Survey Information

top of page.
Frequency.

Occasional.

Initial survey year.

2018.

Reference period.

The nonprofit fiscal year ending in 2016.

Response unit.

Organization.

Sample or census.

Sample.

Population size.

A total of 117,539 nonprofit organizations.

Sample size.

A total of 6,071 nonprofit organizations.

Key variables.

Key variables of interest are listed below.

  • R&D expenditures by source of funds (federal government, state and local government, business, foundations, universities, other nonprofits, internal funds, individual donors, and other)

  • R&D expenditures by type of R&D (basic research, applied research, and experimental development)

  • R&D expenditures by field of R&D (agricultural sciences and natural resources and conservation; biological, biomedical, and health sciences; engineering; geosciences, atmospheric sciences, and ocean sciences; mathematics, statistics, and computer and information sciences; physical sciences; psychology and social sciences; humanities; and other fields)

  • Full-time equivalent R&D personnel (researchers vs. technicians and other support personnel)

  • Headcounts for contract employees and volunteers working on research activities

  • R&D funding provided to others, by type of funding (grants and contracts vs. subawards and subcontracts)

  • R&D funding provided to others, by type of recipient organization (universities, other nonprofit organizations, businesses, and other)

  • R&D funding by field of R&D (agricultural sciences and natural resources and conservation; biological, biomedical, and health sciences; engineering; geosciences, atmospheric sciences, and ocean sciences; mathematics, statistics, and computer and information sciences; physical sciences; psychology and social sciences; humanities; and other fields)

  • R&D funding provided to others, by source of funds (federal government, state and local government, business, foundations, universities, other nonprofits, internal funds, individual donors, and other)

  • R&D funding by type of R&D (basic research, applied research, and experimental development)

Survey Design

top of page.
Target population.

The target population for the FY 2016 NPRA Survey consisted of nonprofit organizations in the United States that performed or funded R&D activities in FY 2016.

Sampling frame.

The National Center for Charitable Statistics (NCCS) Core Files, which capture financial information from Internal Revenue Service (IRS) Form 990, Form 990-EZ, and Form 990-PF, served as the primary input to the sampling frame from which the NPRA Survey sample was selected. Organizations were excluded from the frame if they were considered outside the scope of the survey (e.g., churches, government organizations, educational institutions, or organizations located outside the United States). A financial threshold was also imposed to increase the efficiency of reaching organizations that perform or fund research.

Sample design.

NPRA used a stratified random sample design with primary strata defined by prior history or estimated likelihood of performing or funding R&D, National Taxonomy or Exempt Entities classification code groups (hospitals, research institutes, and other classifications), and IRS tax return type (Form 990 vs. 990-PF). Size strata defined by expenses were created within noncertainty primary strata (see below).

The primary strata are:

  • Stratum 1—Likely performers and funders
  • Stratum 2—Likely performers
  • Stratum 3—Likely funders
  • Stratum 4—Hospitals not included in the above strata
  • Stratum 5—Research institutes not included in the above strata
  • Stratum 6a—Nonprofits filing Form 990 not included above; high likelihood of R&D performance
  • Stratum 6b—Nonprofits filing Form 990 not included above; moderate likelihood of R&D performance
  • Stratum 6c—Nonprofits filing Form 990 not included above; low likelihood of R&D performance
  • Stratum 7a—Nonprofits filing Form 990-PF not included above; high likelihood of R&D funding
  • Stratum 7b—Nonprofits filing Form 990-PF not included above; moderate likelihood of R&D funding
  • Stratum 7c—Nonprofits filing Form 990-PF not included above; low likelihood of R&D funding

All nonprofits in strata 1–3 were selected with certainty. Sampling rates in the other strata varied by size stratum and were based on the relative cost per R&D active performer and variance of total expenses.

Data Collection and Processing

top of page.
Data collection.

Data collection for the NPRA Survey occurred in two general phases: a screening phase and the survey itself. In phase 1, organizations in strata 4–7c were sent a screener response card asking if they had performed or funded R&D in FY 2016. If an organization said that it had not performed or funded research in FY 2016, it was not contacted again.

Known R&D active organizations in strata 1–3 received their first communication about the survey in phase 2. Organizations from phase 1 that either responded that they performed or funded research or did not respond at all were also included in phase 2. There were two versions of the FY 2016 NPRA Survey questionnaire: the health version of the survey form was sent to organizations classified as a health organization by NCCS, and the standard survey form was sent to all other organizations. Respondents could choose to submit a paper survey or use a Web-based data collection system to respond. Every effort was made to maintain close contact with respondents throughout the process to ensure the accuracy of the resulting data.

Data processing.

All data submitted by respondent organizations were reviewed to ensure that data fields were complete and that data are internally consistent. Automated edit checks were applied to improve the efficiency of data review and correction. Edit checks were designed to catch arithmetic errors and logically inconsistent responses (balance edits). If additional information or data corrections were needed, respondents were contacted to clarify or correct data.

Estimation techniques.

Where possible, missing data were imputed using previous survey data or other publicly available documents such as annual reports and financial statements. Weights were used to compensate for unequal probabilities of selection and for nonresponse and to calibrate sample estimates of expenses to match total expenses on the frame.

Survey Quality Measures

top of page.
Sampling error.

Relative standard errors (RSEs) are produced for all estimates. The RSEs represent variability of the survey estimates as a result of random sampling error as well as nonresponse error assuming missing at random and item imputation. Tables presenting the estimated measures of variability corresponding to each data table are available as an appendix in Assessment of the FY 2016 Survey of Nonprofit Research Activities to Determine Whether Data Meet Current Statistical Standards for Publication https://ncses.nsf.gov/pubs/ncses22212.

The RSE for the estimated total R&D performed by nonprofit organizations in FY 2016 is 13%. The RSE for the estimated total R&D funded by nonprofit organizations in FY 2016 is 18%.

Nonsampling error.

All surveys and censuses have nonsampling errors. Nonsampling errors are attributable to various sources, such as the inability to obtain information for all cases in the universe, imputation for missing data, data errors and biases, mistakes in recording or keying data, errors in collection or processing, and coverage problems.

Data processing error.

It is believed that most of the important operational and data errors were detected and corrected through automated data edits and analyst inspections designed to review the data for reasonableness and consistency. Quality control techniques were used to verify that operating procedures were carried out as specified.

Coverage error.

Using the NCCS Core Files as a sampling frame excludes most public charities and other exempt organizations with gross receipts of less than $50,000. Although organizations with gross receipts of less than $50,000 represent 70% of public charities and other exempt organizations, they only represent approximately 0.1% of the total gross receipts generated by those organizations. Exclusion of these small organizations is an undercoverage risk and a limitation of the design because they will not be included in the sampling frame. However, due to their size, the impact on the estimate of total research expenditures is negligible.

Another source of undercoverage is new organizations that have not filed a Form 990, Form 990-EZ, or Form 990-PF because they did not meet the criteria or because they did not exist in 2013 when the frame was created. However, anticipated coverage errors are small, based on an increase of only 1.1% of organizations between the 2014 and 2015 IRS Exempt Organizations Business Master File Extract. Further, to the extent that R&D for new organizations is similar to that of organizations on the frame, the calibration adjustment reduces the impact of new organizations on the final estimates.

Overcoverage is caused by organizations that are not part of the target population (e.g., defunct organizations) being present in the sampling frame and potentially sampled. The sample cleaning and screening process (to remove organizations that do not perform or fund research) used by the NPRA Survey to verify performer and funder status was designed to remove these cases and therefore minimize overcoverage error in the final estimates. Organizations that did not respond to the screener questions could not be screened out.

Item nonresponse error.

There are several possible sources of item nonresponse by respondents, including inadvertently skipping data fields and skipping data fields when data are unavailable. Organizations were encouraged to estimate when actual data are unavailable. Variation due to the imputation of missing data is accounted for in the RSE estimates.

Unit nonresponse error.

The survey obtained a 48% unweighted and 61% weighted response rate. Response rates varied across strata. The overall unweighted response rates, inclusive of performers and funders, ranged from 35% among hospitals to 62% among organizations that were likely performers and funders of research. Response rates were slightly higher among funders than performers across most strata. A nonresponse bias analysis was conducted using each sampled organization's 2016 total expenses. Relative bias in mean total expenses was –.526 (i.e., 52.6%) overall, ranging from –0.946 to 0.196 across strata. Four of the 11 strata had significant relative bias in mean total expenses. In order of absolute value of the bias (from highest to lowest), the four strata are as follows:

  1. Stratum 6a (Form 990, high likelihood): –0.946
  2. Stratum 4 (hospitals): –0.689
  3. Stratum 6b (Form 990, moderate likelihood): –0.295
  4. Stratum 6c (Form 990, low likelihood): –0.143

The results of the nonresponse bias analysis were used to develop weighting cells to reduce the risk of nonresponse bias for total R&D performing and funding.

Measurement error.

The largest risk of measurement error is likely respondents' interpretation of the definition of R&D activities and variations in record-keeping procedures used by respondents to answer the survey questions. For example, the survey asks for an organization's expenditures for basic research, applied research, and experimental development. Some organizations said that these amounts were difficult to report because either they could not determine how best to allocate expenditures among the three categories or because they did not track information in that way.

Data Availability and Comparability

top of page.
Data availability.

Data are available for FY 2016.

Data comparability.

Not applicable.

Data Products

top of page.
Publications.

Data tables from this survey are published in the series Nonprofit Research Activities Survey. They are located at https://www.nsf.gov/statistics/srvynpra/.

Electronic access.

Not available.

Contact Information

top of page.

For additional information about this survey, or the methodology report, contact the Survey Manager.

Ronda Britt
Survey Manager
NCSES
Tel: (703) 292-7765
E-mail: rbritt@nsf.gov

National Center for Science and Engineering Statistics
Directorate for Social, Behavioral and Economic Sciences
National Science Foundation
2415 Eisenhower Avenue, Suite W14200
Alexandria, VA 22314
Tel: (703) 292-8780
FIRS: (800) 877-8339
TDD: (800) 281-8749
E-mail: ncsesweb@nsf.gov