Sampling

The target population for the NPRA Survey is nonprofit organizations in the United States. An organization is considered a nonprofit if it is categorized by the Internal Revenue Service (IRS) as a 501(c)(3) public charity, a 501(c)(3) private foundation, or another exempt organization, such as a 501(c)(4), 501(c)(5), or 501(c)(6).[2] As recorded in the IRS Exempt Organizations Business Master File Extract (EO BMF) dated December 2016, there were 1.6 million tax-exempt organizations in the United States. Of those, nearly 1.2 million filed an information return with the IRS during the previous 24 months. Certain nonprofit organizations are not required to file an information return (e.g., churches), but those that are must file Form 990, 990-N, 990-EZ, or 990-PF, based on their organization type and financial size.[3]

Small organizations, with gross receipts under $50,000, are allowed to file Form 990-N (the e-Postcard), which does not require the organization to report financial data. Nearly half the filing organizations in the December 2016 EO BMF filed Form 990-N. These organizations were excluded from the frame because of their relatively small size and lack of financial data. The remaining organizations filed Form 990, 990-EZ, or 990-PF, which all require financial reporting.

The financial information on Forms 990, 990-EZ, and 990-PF is captured in the National Center for Charitable Statistics (NCCS) Core Files. The 2013 Core Files were the latest available at the time of the sample selection (September 2017) for the NPRA Survey, which requested data for FY 2016. A total of 618,612 organizations had financial information recorded in the 2013 Core Files. Organizations were excluded from the frame if they were considered to be outside the scope of the survey. Specifically, organizations were excluded if they had an IRS 501(c) subsection code that did not equal 3, 4, 5, or 6; had a foundation code indicating that they were a school, church, or government; were located outside the United States; had a North American Industry Classification System (NAICS) code indicating that they were in the public administration sector; or had an NCCS code indicating that they were a government entity or otherwise out of scope. In addition, organizations were excluded if they were found to be inactive during the pilot NPRA Survey, which was conducted in late 2016 through early 2017.

A financial threshold was imposed to further reduce the frame size. For organizations filing Form 990 or Form 990-EZ, only those with $500,000 or more in expenses were included; for organizations filing Form 990-PF, only those with $2,750,000 or more in total assets were included.

For the 117,539 organizations remaining after exclusions and financial truncation, stratification was done based on frame variables associated with R&D. This included the NCCS's National Taxonomy of Exempt Entities (NTEE) Core Code, which categorizes the primary function of the organization (e.g., hospitals, research institutes), as well as a propensity score measuring the likelihood that an organization performed or funded research. The propensity score was developed from a model relating likely performers and likely funders to financial variables in the frame. The likely performers and likely funders were a subset of organizations identified from auxiliary sources that strongly indicated that they performed or funded research. The model produced a propensity score, where high values indicate a higher likelihood of performing or funding research. The propensity scores were grouped into high, moderate, and low likelihood.

The propensity score strata were combined with other stratifiers to form the final stratification. Based on the results of the pilot survey, size strata defined by the total amount of expenses were also added.

The sample was stratified as follows:

• Stratum 1—Likely performers and funders
• Stratum 2—Likely performers
• Stratum 3—Likely funders
• Stratum 4—Hospitals
• Stratum 5—Research institutes
• Stratum 6a—Form 990, high likelihood of R&D performance
• Stratum 6b—Form 990, moderate likelihood of R&D performance
• Stratum 6c—Form 990, low likelihood of R&D performance
• Stratum 7a—Form 990-PF, high likelihood of R&D funding
• Stratum 7b—Form 990-PF, moderate likelihood of R&D funding
• Stratum 7c—Form 990-PF, low likelihood of R&D funding

Strata 1–3 included organizations with a high likelihood of performing or funding research based on various auxiliary data, including past surveys, membership lists, and other government data collections. Organizations in these strata were sampled with certainty. Strata 4–7c included organizations whose performer or funder status was unknown. Stratum 4 included organizations identified as hospitals by their NTEE Core Code, if those organizations had not already been assigned to strata 1–3. Stratum 5 included organizations identified as research institutes that had not already been assigned to strata 1–3. Strata 6a–7c included organizations whose likelihood of performing or funding research was predicted using information on their Form 990 (strata 6a–6c) or Form 990-PF (strata 7a–7c). Within each stratum, organizations were grouped into six size classes based on total expenses. The sample was allocated to the strata to minimize the variability of total expenses, resulting in intentional oversampling in substrata that included large organizations.

The sample was a systematic (1-in-k) random sample of organizations within each stratum. The organizations were selected with equal probability. Before the systematic sample was selected, the organizations were stratified implicitly (sorted) by total expenditures to ensure that the sample was proportionately distributed by size. The stratum variances for the sample design calculations excluded 206 organizations with expenses larger than $900 million. It was assumed that these 206 organizations would be self-representing due to their size and have minimal impact on the sampling variance. The 206 organizations were selected with certainty. The final sample sizes by stratum are shown in table A.

Data Collection and Processing Methods

Data collection for the NPRA Survey occurred in two general phases. The first focused on determining whether the organizations in strata 4–7c (i.e., "unknown" organizations) performed or funded research during FY 2016. The second involved notifying organizations that they had been selected to participate in the survey and sending the survey materials to each organization's point of contact via mail and e-mail. One major challenge of the survey was the inability to locate e-mail addresses for 52% of the surveyed organizations, meaning these organizations only received the survey and reminders by mail.

In phase 1, organizations in strata 4–7c were sent a letter providing details about the survey, a screener response card, and a business reply envelope in late February 2018. They were asked to complete the screener card by indicating whether their organization had performed or funded research during FY 2016 and providing their contact information (i.e., contact name, title, e-mail address, and phone number). If an organization said that it had not performed or funded research in FY 2016, it was not contacted again. The last screener card was returned on May 25, and after a week during which no additional screener cards were received, phase 1 was considered closed on June 1.

Organizations in strata 1–3 (i.e., "known" organizations) received their first communication about the survey in phase 2. Organizations from phase 1 that either responded that they performed or funded research or did not respond at all were also included in phase 2. All organizations included in phase 2 received a link to the Web survey via mail (and e-mail for those with known e-mail addresses) in late April 2018. A second mailing was done in late May that included a paper version of the questionnaire. There were two versions of the FY 2016 NPRA Survey questionnaire: A health version of the survey form was sent to organizations classified as a health organization by NCCS, and the standard survey form was sent to all other organizations. Questions on the two surveys were the same, but during the pilot and preparations for the FY 2016 survey, NCSES learned that large health-focused nonprofits, such as hospitals, were less likely to discard a "health survey" form than they would a "nonprofit organization survey" form. Several more contacts (e-mail, mail, and phone) were made throughout the summer to encourage response, including mailing another copy of the questionnaire in late July. Every effort was made to maintain close contact with respondents throughout the process to ensure the accuracy of the resulting data. Questionnaires were carefully examined for completeness upon receipt, and respondents were sent personalized e-mails asking them to make any necessary revisions before the final processing and tabulation of data.

Response rates

Overall response was defined as completing the questionnaire or indicating that the organization does not perform or fund research, and response rates were calculated out of the number of eligible organizations in the sample—i.e., excluding organizations found to be ineligible for the survey because they were not a nonprofit, were covered through another NCSES data collection, or were defunct (i.e., out of business). The survey obtained a 48% unweighted and 61% weighted response rate (table B) across all strata (i.e., 2,919 organizations responding out of 6,071).

Unweighted response rates ranged from a low of 35% for hospitals not in stratum 1–3 (i.e., those not already identified as being likely to perform or fund research) to 62% for the likely performers and funders (stratum 1). When viewed as a whole, hospitals were much less likely to respond than were non-hospitals across all strata (35% versus 51%). A major challenge of surveying hospitals was getting past the organization's gatekeepers to an office or individual that would be willing to respond to the survey. Another challenge was the degree of consolidation among individual hospitals that was unable to be determined in advance from the information provided on the NCCS Core Files. Many hospitals viewed as separate entities according to their Form 990 information were in fact part of much larger health systems and sent their surveys to a parent organization responsible for centralized reporting.

Performer and funder response rates were both calculated to count organizations as complete if they answered all the questions asked of them. Performer response rates were calculated as the share of completed performer questionnaires plus organizations that indicated they only fund research plus organizations that indicated they do not perform or fund any research, out of all eligible organizations. Similarly, funder response rates were calculated as the share of completed funder questionnaires plus organizations that indicated they only perform research plus organizations that indicated they do not perform or fund any research, out of all eligible organizations.

The performer response rate was 49% unweighted (i.e., 2,985 organizations out of 6,071) and 61% weighted. The funder response was 51% unweighted (i.e., 3,074 organizations out of 6,071) and 62% weighted.

Among those who responded, the percentage of those reporting R&D activity varied considerably by stratum (table C). Across all strata, only 27% reported R&D activity. This ranged from a low of 4% for those in the 990-PF low likelihood stratum to a high of 83% for the likely performers and funders stratum.

For more detail including response by type of nonprofit and item response rates, see the technical tables in Appendix B.

Nonresponse bias analysis

The nonresponse bias analysis was conducted using each sampled organization's 2016 total expenses. Relative mean bias, which defines bias as a percentage of the weighted mean for respondents, was used so that bias estimates were comparable across strata and other groupings. Relative bias in mean total expenses was -.526 (i.e., 52.6%) overall, ranging from -0.946 to 0.196 across strata. Four of the 11 strata had significant relative bias in mean total expenses. In order of absolute value of the bias (from highest to lowest), the four strata are:

1. Stratum 6a (Form 990, high likelihood): -0.946
2. Stratum 4 (hospitals): -0.689
3. Stratum 6b (Form 990, moderate likelihood): -0.295
4. Stratum 6c (Form 990, low likelihood): -0.143

Organizations that received the health survey form also had a significant level of bias in total expenses (0.875). Organizations receiving the standard survey form showed less bias, although it was statistically significant (-0.115).

Estimated coverage for total expenses was defined as the proportion of total expenses on the sampling frame accounted for by sample weighted respondents, (i.e., a measure of how well sample-weighted responding organizations reflect the sampling frame). Coverage rates ranged from 25% to 58% across strata, with overall coverage of 39%. Again, hospitals (stratum 4) had the lowest coverage rate (25%) and hospital organizations (NTEE Core Codes E20–E24) had a coverage rate of 29%, compared with 47% for nonhospitals.

Two major trends appeared in the nonresponse bias analysis. First, hospitals and health organizations showed the highest nonresponse bias and lowest coverage rates for total expenses, fulfilling expectations about the difficulty of surveying these groups.

Second, although the certainty strata showed moderate to low levels of nonresponse bias as well as moderate levels of coverage for expenses, a different pattern appeared within the noncertainty stratum in which organizations' likelihood of performing or funding research was predicted. Organizations with a high likelihood of performing R&D (stratum 6a) showed higher levels of nonresponse bias and lower levels of coverage for expenses than did groups with a lower likelihood of performing R&D. However, the same pattern does not hold for organizations with a high likelihood of funding R&D and that file Form 990-PF (i.e., stratum 7a, private foundations). This stratum had very low nonresponse bias (0.196). It is interesting that organizations known to have higher levels of R&D (i.e., certainty strata) and private foundations predicted to have high levels of R&D have moderate to low nonresponse bias, whereas other types of nonprofit organizations predicted to have high levels of R&D exhibited very high nonresponse bias. This suggests that organizational characteristics beyond real or predicted R&D levels are important for understanding nonresponse bias and reinforces the importance of coverage evaluation and projection techniques performed.

To reduce the risk of bias in the survey estimates, nonresponse and calibration adjustments were made. These weights were performed as part of the overall weighting approach for the NPRA Survey (see next section). Weighting for nonresponse reduces the risk of nonresponse bias to the extent possible given the data currently available for nonrespondents.

Imputation, weighting, and estimation procedures

Missing data were imputed for organizations that (1) did not respond to the survey, but for which auxiliary data about the amounts spent performing or funding research were available, (2) did not respond to the survey, but for which information was available from the pilot survey about the amounts spent performing or funding research, and (3) reported that they performed or funded research (Questions 7 and 8, respectively) but did not provide information on the amounts spent performing or funding research (Questions 9 and 16, respectively). The last group included organizations that completed the screener card in phase 1 but did not complete the questionnaire in phase 2. Organizations that did not fall into one of these groups were accounted for in the nonresponse weighting adjustment described below. For those organizations performing R&D activities, imputation accounted for $3.1 billion of the $22.6 billion weighted total reported. For those funding R&D activities, imputation accounted for $792 million of the $10.5 billion weighted total reported.

For large organizations that did not respond to the survey, publicly available documents—such as annual reports and financial statements—were used to impute the amounts spent on research activities. In total, 83 organizations were included in the lookups, including the 40 largest nonresponding organizations from strata 1–3 (likely performers and funders) and the 10 largest organizations who reported either performing or funding but did not provide an amount. This auxiliary look-up method resulted in imputing research-performing status for 46 organizations, research-funding status for 33 organizations, total performance dollar amounts for 24 organizations, and total funding dollar amounts for 2 organizations.

Sixty-three organizations reported the amount spent on performing research on the pilot NPRA Survey but did not provide a response on the FY 2016 survey. Similarly, 44 organizations reported the amount spent on funding research on the pilot survey but not on the FY 2016 survey. The pilot survey was used to impute values for these organizations. The pilot survey amounts were adjusted by an imputation factor to account for inflation or deflation in the reported amounts from the previous year. Imputation factors are the ratio of the current survey data to the previous survey data for organizations that responded to both, and these factors reflect the average annual growth or decline in research expenditures. The imputation factor was applied to responses to the previous survey to estimate the amount for the current survey. The total imputed amount spent in each subcategory for performing research (Questions 10, 11, and 12) and funding research (Questions 17, 18, 19, and 20) was imputed by distributing the total amounts to the subcategories in the same proportions as reported on the pilot survey.

For organizations that reported performing or funding research (Questions 7 and 8, respectively) but did not provide the amount spent (Questions 9 and 16, respectively), amounts were imputed from a regression model using expenses and assets reported on Form 990 or Form 990-PF and significant classification variables such as NTEE Core Code. The source of the expenses and assets was the 2016 Statistics of Income (SOI) financial data extract downloaded from the IRS. If no 2016 data were available, data for 2015 or 2014 were substituted.

The total imputed amount spent in each subcategory for performing research (Questions 10, 11, and 12) and funding research (Questions 17, 18, 19, and 20) was imputed by distributing the total amounts to the subcategories, based on the average proportions for the responding organizations in the imputation classes based on type of organization (groupings by NTEE Core Codes).

The nonresponse weighting adjustment accounted for organizations that did not respond to the FY 2016 survey and for which information from other sources was not available to use for imputation. For these adjustments, the definition of respondents included organizations that performed or funded research (confirmed eligible) and those that reported that they did not perform or fund research (confirmed ineligible). Nonrespondents were the organizations for which eligibility had not been determined (unresolved eligibility status).

The nonresponse adjustment was a ratio adjustment where the respondents (r) were weighted to account for the nonrespondents (nr). The nonresponse adjustment calculation was weighted based on base weight and total expenses. The base weight was adjusted by the nonresponse factor, W2 = W1 * f1.

Nonresponse classes were determined through the nonresponse bias analysis that identified variables with differential nonresponse. This was done using a logistic regression model with survey response as the outcome. The outcome was modeled based on the frame data and auxiliary information available for both respondents and nonrespondents. The frame data included information from Form 990 and Form 990-PF at the time of frame creation (2013). Other information explored in the nonresponse analysis included expenses, assets, and revenue from the 2016 SOI data and organizations from the FY 2016 Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions (Federal Support Survey). Because the available data for nonresponse adjustment differed between Form 990 and Form 990-PF, the response models were calculated separately for organizations filing Form 990 and for organizations filing Form 990-PF. Nonresponse classes were based on significant variables within each group.

The last step in the weighting process was to calibrate the weighted expenses for the responding organizations to match the total expenses of nonprofit organizations in the population based on the 2016 SOI financial data extract. This adjustment corrected for changes in the population that may have occurred between frame development in 2013 and the reference year of the survey (FY 2016). The calibration adjustment was based on the respondents and the out-of-sample organizations found during the data collection. Although the out-of-sample organizations did not qualify for the NPRA Survey, they were still represented in the population.

Successive difference replication (SDR) was used to estimate the variance of the estimates. SDR was developed for systematic samples where the frame is ordered in such a way as to improve the sampling variance. The sample selection was sorted by expenses within each stratum. In comparison to the direct variance estimator, SDR has the advantage of capturing the variability resulting from the imputation and weighting process. The variance estimates were based on 80 replicates. For each replicate, every selected organization was weighted by a replicate factor of 1.0, 1.7, or 0.3. The replicate factor was applied to the sampling weight. For each sample replicate, nonresponse and calibration adjustments were recalculated as described above.

Refinement of the final estimates

There were a series of estimates developed for the FY 2016 NPRA Survey as the estimation methodology was refined. Each estimate was checked for errors and benchmarked against existing comparable data.

During the first run of the estimation procedures on the final data, two errors were discovered. First, two extremely large organizations were weighted twice due to duplicate representation in the sampled employer identification numbers (EINs). Second, many organizations found to perform R&D via the Federal Support Survey were not accurately coded at the time of sampling and were thus placed in the unknown strata which received higher weighting.

To correct these errors, the duplicated organizations were permanently removed from the dataset, and alternatives to handling the incorrectly stratified organizations were explored in a series of alternate estimates. Further refinements were also made to the estimation plan that would compare weighting by counts vs. expenses.

To gauge the sensitivity of the original estimates, four alternate approaches for estimating nonprofit R&D performance were considered: two based on counts of organizations and two based on expenses, each with and without attempts to correct the issue of the incorrectly classified organizations. Specifically, alternate estimate 1 used the original sampling strata with no revisions and was weighted to total expenses on the frame. Alternate estimate 2 adjusted the original sampling strata to move the inaccurately coded organizations to the known strata ("strata adjusted for Federal Support Survey misses") and was also weighted to total expenses. Alternate estimate 3 used the original sampling strata with no revisions and was weighted based on the count of organizations on the frame. Alternate estimate 4 adjusted the sampling strata as was done for alternate estimate 2 and was weighted to the count of organizations.

Following the evaluation of these alternate estimates, a final estimate was developed that allowed the original sampling strata to stay unaltered but accounted for the incorrect classification of the organizations by flagging them in the imputation and nonresponse adjustment models. The final estimate also based the nonresponse adjustment and final calibration on total expenses, which was determined to be the most appropriate measure to use after evaluating the previous estimates.

The four alternate estimates range from between $23.4 billion and $28.6 billion in total R&D performance, of which federal sources account for between $9.2 billion and $11.5 billion (table D). The final survey estimates show total nonprofit R&D performance at $22.6 billion, of which federal sources accounted for $8.3 billion.

There are no independent sources of data on R&D performance by nonprofit organizations, which is why an updated NPRA Survey was so important to undertake. However, one source of comparable data on R&D funding to the nonprofit sector is the NCSES Federal Funds Survey, which measures federal R&D obligations to nonprofits. These federal agencies' reported totals differ from the nonprofit reported R&D performance totals because the Federal Funds Survey measures obligations by federal fiscal year and the NPRA Survey measures expenditures by the nonprofit's fiscal year. However, the total from the Federal Funds Survey can be used for rough benchmarking of the federally funded R&D performance total on the FY 2016 NPRA Survey.

The Federal Funds Survey data show obligations to nonprofits of $6.7 billion in FY 2016 and $8.1 billion in FY 2017, for a 2-year average of $7.4 billion. The four alternate estimates of federal funded nonprofit R&D performance are higher, ranging between 24% and 55% higher than the total from the Federal Funds Survey.

All five federal estimates (four alternate estimates and the final estimate) are higher than the total from the Federal Funds Survey. However, at only 11% higher, the final nonprofit performer survey estimate is not significantly different from the total from the Federal Funds Survey.

The shares of the other sources of funds were relatively stable in each of the five estimates, pointing to the importance of benchmarking the federal estimate to increase accuracy of the overall number.

Another helpful comparison are the estimates of nonprofit R&D performance and funding from the NCSES publication National Patterns of R&D Resources. The final FY 2016 NPRA Survey estimate of $22.6 billion in R&D performed falls between the estimates from the National Patterns of R&D Resources for calendar year 2016 and 2017 ($21.3 and $23.3 billion respectively). However, the estimate for R&D funded by nonprofits was not as similar. The NPRA Survey estimate of funding for R&D performed both internally and externally was $14.2 billion in FY 2016. Estimates from the National Pattern of R&D Resources for calendar year 2016 and 2017 were almost 50% higher at $20.6 and $21.5 billion for 2016 and 2017, respectively. This may be due to the lower response rate from foundations and other likely funders impacting the quality of the final NPRA Survey estimates. More work needs to be done to ensure broader coverage and response of funding organizations in any future survey measuring nonprofit funding of R&D.

Notes

[2] See https://learn.guidestar.org/help/irs-subsection-codes for definitions of the subsection codes for tax-exempt organizations.

[3] See https://www.irs.gov/charities-non-profits/form-990-series-which-forms-do-exempt-organizations-file-filing-phase-in for more information on the different versions of Form 990.