Growth of High Quality Graphene by UCSB Researchers a Critical Discovery for Producing Next Generation Electronics.

NSF Guidelines for Research Security Analytics

Summary

The NSF Research Security Analytics Guidelines is a public document describing NSF's internal guidance for research security data-related practices. It includes a breakdown of which agency personnel may conduct research security-related activities; what monitoring activities are allowed and with what resources they are conducted; how information will be validated to ensure accuracy; and how information may be shared within NSF and externally (summarized in Figure 1 to the right).  

This report builds on extensive feedback from community stakeholders and establishes key principles for NSF's research security analytics activities:  

  1. Program staff are not permitted to use research security concerns as a determining factor in the merit review process.  

  1. All research security analytics activities at NSF will be conducted solely by the Office of the Chief of Research Security Strategy and Policy, or OCRSSP. 

  1. Program staff are not permitted to conduct intentional information querying activities related to research security. Concerns encountered during routine merit review activities (see "routine assessment" in definitions) are to be reported to OCRSSP.  

The guidelines are consistent with information laid out in the System of Records Notice on NSF-77 Data Analytics Application Suite, or SORN NSF-77, and are a part of NSF's efforts towards transparency of internal practices. The guidelines also relate OCRSSP and NSF requirements as described in the "CHIPS and Science Act of 2022," National Security Presidential Memorandum 33, or NSPM-33, and its accompanying implementation guidelines.  

Relevant authorities and supporting documentation

The NSF guidelines for research security analytics were created in response to the following policies requiring proper disclosure of appointments, affiliations and current and pending support for external funding sources:

Research security responsibilities and process of the Office of the Chief of Research Security Strategy and Policy

NSF is committed to safeguarding the integrity and security of science while also keeping fundamental research open and collaborative. This requires coordinated, transparent initiatives managed by NSF's Office of the Chief of Research Security Strategy and Policy. OCRSSP is the only office within NSF approved to conduct advanced monitoring and verification activities related to research security. These unique responsibilities of routinely assessing, verifying and reporting inconsistences are summarized in Figure 1.

NSF's OCRSSP is firmly committed to conducting its research security-related advanced monitoring and verification activities in strict accordance with federal law. As stated in in the 2022 "CHIPS and Science Act" Section 10637, "In carrying out requirements under this subtitle, each Federal research agency shall ensure that policies and activities developed and implemented pursuant to this subtitle are carried out in a manner that does not target, stigmatize, or discriminate against individuals on the basis of race, ethnicity, or national origin, consistent with title VI of the Civil Rights Act of 1964 (42 U.S.C. 2000d et seq.)." OCRSSP will ensure these requirements are upheld while carrying out its research security responsibilities.

Figure 2. NSF's Steps for Handling Potential Inconsistencies
Figure 2. NSF's Steps for Handling Potential Inconsistencies

Credit: NSF

Monitoring and reporting by NSF offices and staff

Terms and conditions compliance-monitoring responsibilities of program staff

As part of NSF's grant review and management process, program staff are responsible for checking proposal disclosure information and overseeing their award portfolio to ensure awardees complete their obligations (see routine assessment in Definitions). They are also expected to engage with the research community and keep abreast of scientific developments. Through these activities, program staff may encounter information that suggests an awardee is non-compliant with NSF's PAPPG disclosure requirements and NSF's terms and conditions, which span multiple topics, including budget expenditures, project reports, etc.

Vetting for employment

NSF may use analytics for vetting employment. The mechanism for that vetting and the responsibility of different NSF organizational elements is a current topic of discussion in NSF's "Intergovernmental Personnel Act" Vetting Working Group. Once the working group's efforts are complete, this section will be updated to reflect the agreed-on approach for vetting for employment.

​​​​​​Permissible approaches for research security analytics

As part of the responsibility to monitor and report research security-related concerns, the CRSSP is approved to apply the data-driven approaches listed below. OCRSSP staff are prohibited from conducting any analysis that selects for a particular national origin or racial identity (see Prohibited practices for research security analytics for complete list).

  1. Conducting analyses that compare all participants in agency programs.
  2. Comparing of self-reported information of PIs with open-source information by the same PI (see Individual matching criteria for validation and information sharing activities for matching requirements).
  3. Filtering and analyzing NSF's portfolio based on institutional characteristics (e.g., the institution's Carnegie Classification, amount awarded in funding, etc.) and research topics (e.g., quantum computing, artificial intelligence, biological sciences, etc.).

Prohibited practices for research security analytics

As stated in the NSPM-33 implementation guidance, "It is essential that policies and consequences must be applied without discrimination in any way, including with respect to national origin or identity." For research security-related analytics, data queries and analyses that are explicitly or implicitly designed to return the identities of individuals of a specific national origin or racial identity are prohibited. This includes querying NSF data and public information by:

  • Nationality.
  • Citizenship.
  • Common names for specific countries/ethnicities.

 

Individual matching criteria for validation and information sharing activities

Human oversight is a critical safeguard of the OCRSSP's monitoring, validating and reporting responsibilities.

Any validation and reporting activities of individuals with conflicting information in the public domain will first be checked against matching criteria listed below. NSF staff reporting inconsistencies to the CRSSP are instructed to apply similar oversight. A potential inconsistency will be considered "verified" if the institution, or OIG, confirms the inconsistency detected by NSF.

  • The first names without the middle names and the email addresses match in both datasets.
  • The last names or a portion of the last names (in case of hyphenated names) and the email addresses match in both datasets.
  • The first names (without the middle names) and the last names (or a portion of the last names, in the case of hyphenated names) match with either a) department affiliation, or b) the institutional affiliation in both datasets
  • Email addresses and department affiliation match in both datasets.
  • Persistent ID (e.g., ORCID) match in both datasets.

 
Figure 3. Guardrails in Place to Prevent Bias and Validate Information
Figure 3. Guardrails in Place to Prevent Bias and Validate Information

Credit: NSF

Data, services and methods used for research security analytics

Non-NSF data used in research security analyses

From 2021 to 2022, OCRSSP staff have used the following data sources to conduct research security-related analyses. These are widely used industry standards and are maintained and updated by dedicated staff. This list will be updated when and if new data sources are used.

  1. Elsevier SCOPUS.
    https://www.elsevier.com/
    Transfer method: private encrypted AWS download.
    Update rate: monthly, with option for weekly updates.
  2. Web of Science.
    https://clarivate.com/webofsciencegroup/solutions/web-of-science/
    Transfer method: private secure SFTP download.
    Update rate: weekly.
  3. U.S. Patent and Trademark Office Patent Database.
    https://bulkdata.uspto.gov/
    Transfer method: public bulk weekly downloads.
    Update rate: weekly.

Analysis criteria and purpose

is published and/or otherwise in the public domain. These criteria include, but are not limited to:

  1. Mismatches between institutional affiliations in published papers and disclosed/reported institution affiliations within proposals to NSF.
  2. Mismatches between funding sources in published papers and disclosed/reported Current and Pending funding sources within proposals to NSF.
  3. Mismatches between filed patents in the USPTO and self-reported intellectual property filings in NSF annual reports

Services used in research security analyses

NSF currently does not employ any off-the-shelf analysis tools or services for research security-related activities. Current analysis activities of the above databases are conducted using Apache SOLR, Carrot Search Lingo4G, Python and R. Information is shared through Excel spreadsheets and visualized through dashboard interfaces created with the Lucidworks fork (link is external) of the Kibana framework. (link is external)  

Human oversight

Safeguarding NSF's analytics process is a top priority for the agency, and human oversight is a paramount requirement for all information sharing activities. No information on individuals may be reported and no adverse action may be taken based solely on a potential inconsistency without human verification of the matching criteria.

Sharing of information with institutions

To clarify inconsistencies in reported information, NSF may share information with the organization or institution that originally submitted the proposal to help cross-reference and verify information (see SORN NSF-77 Routine Use 12).

Verification requirements: When sharing an individual's reporting inconsistencies with an institution (see SORN NSF-77 Routine Use 12), OCRSSP must attempt to verify the information according to the matching criteria listed in the guidelines matching requirements. Should any item not be successfully verified, this will be clearly communicated to the organization or institution.

Sharing of information by OCRSSP with inspector general or federal agencies

In accordance with the "CHIPS and Science Act" Section 10331 and NSPM-33 4.(e), information on validated incidents of violations of NSF's disclosure requirements for submitted proposals, the terms and conditions of an award, and project reports may be disclosed by OCRRSP to the appropriate federal agencies (including but not limited to OIG, law enforcement, intelligence agencies, and other relevant agency components) to inform efforts related to national and research security (See SORN NSF-77 Routine Use 13).

  • All reporting must have documentation showing that the verification process included human oversight.
  • Personally identifiable information, or PII, shared with agencies will be restricted to information about senior personnel only and coordinated with OIG, unless otherwise required by applicable law.
  • To the extent allowed by law, NSF will not share any PII information beyond an individual's name and affiliated institution. Self-defined ethnicity, gender, etc. will not be included. Relevant administrative information, such as PII-redacted proposal and award history and open-source information, may be shared. Only proposal content of verified inconsistencies may be shared with appropriate points of contact at other agencies.
    • Example of information sharing: An intelligence agency is investigating a PI and has asked for information regarding NSF-funded activities, including proposal information. CRSSP staff prepare the relevant information but will omit any PII-content from the proposal, including gender and ethnicity.

The CRSSP is responsible for approving all information shared by OCRSSP with OIG or federal agencies. Prior to consideration of further action, OCRSSP may consult with host institutions to verify potential inconsistences, and when appropriate, share the verified inconsistencies with the appropriate federal agencies.

Updates to guidelines

These guidelines will undergo iterative improvement based on the needs of the scientific community, the federal government, and the national security community. Updates made to the guidelines will be tracked here. This version was published in June 2023 and is the first version of the guidelines.  

 

Research Security Guidelines PDF