July 19, 2024

Dear Colleagues:

The U.S. National Science Foundation's (NSF) Directorates for Computer and Information Science and Engineering (CISE) and Technology, Innovation and Partnerships (TIP) are seeking new approaches to develop and evaluate novel artificial intelligence (AI) methods in real-world settings. Too often new AI systems are deployed before the interactions with and impacts on users can be fully evaluated or understood. Often, when AI is evaluated, it is with an inadequate number of samples that do not scale or generalize beyond a limited number of use cases.

This Dear Colleague Letter (DCL) seeks to address these limitations by encouraging the community to pursue Planning Grants designed to develop AI-Ready Test Beds that can be used by researchers to test novel AI methods in potential real-world application scenarios. Because of the current limits in scalability and scope, the AI community is encouraged to expand existing test beds and infrastructure to make them AI-ready and appropriate for use in evaluating the impact and effect of AI tools and systems on users. To do this, proposers are encouraged to create teams of researchers who have expertise in AI as well as domain experts and staff managing existing test beds. Existing test beds and infrastructure can be found in a range of settings, including, but not limited to: NSF-funded centers (e.g., Engineering Research Centers) and facilities, infrastructure funded by other federal agencies (e.g., Department of Energy, National Institute of Standards and Technology, National Oceanic and Atmospheric Administration, Department of Homeland Security, Department of Transportation, etc. or at https://www.nitrd.gov/apps/ai-rd-testbed-inventory/), state agencies (e.g., https://testbed.cityofnewyork.us/) and industry. These Planning Grants will support costs associated with the formation of teams, the planning of the desired or envisioned AI-Ready infrastructure, the collection of preliminary pilot data and creation of governance and management plans for scalable AI-Ready Test Beds.

AI-Ready Test Beds are envisioned as comprehensive services that will provide infrastructure to support researchers to bring innovative applications of AI to bear in high-impact settings. The AI-Ready Test Beds will include technical staff and domain experts to guide and enable new researchers to easily connect to the application domains and provide human and technology interfaces that embed with front-line service providers and domain experts. Planning Grant proposers should consider that the targeted AI-Ready Test Beds could include both co-located and remote research teams and that the AI-Ready Test Beds should support multiple and varied use cases, facilitating inter-project collaboration and nexus-building as appropriate. Planning Grant proposers should envision AI-Ready Test Beds that add AI components to test beds that are already suited to use case experimentation, adopter demonstrations, longer baseline validation testing, etc.

While building AI systems with static data is well-understood, building them or evolving them based on continuously changing or dynamic data, including real-time streaming data and data with significant spatio-temporal characteristics that are difficult to predict, is a significant issue. Hence, AI-Ready Test Beds should embrace a "living lab" model that aims to provide researchers with the ability to study societal impacts including harms to individuals and communities; assess various risks related to security, safety, privacy and fairness, and help stakeholders effectively deploy AI systems following well-established AI risk management frameworks; and experiment with and develop various red-teaming techniques as well as evaluate their effectiveness. AI-Ready Test Beds are envisioned to provide the infrastructure to lower the costs and reduce the communication barriers that researchers currently encounter upon entering an application domain, while allowing real-world tests of innovative AI at scale and in real time.

Funding is available for Planning Grants in Fiscal Year (FY) 2025 to provide support to teams so that they may later submit a full proposal for an AI-Ready Test Bed.

BACKGROUND

This initiative is responsive to the Executive Order (EO) on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence (the AI EO, October 2023), which states, "Artificial intelligence (AI) holds extraordinary potential for both promise and peril. Responsible AI use has the potential to help solve urgent challenges while making our world more prosperous, productive, innovative, and secure. At the same time, irresponsible use could exacerbate societal harms such as fraud, discrimination, bias, and disinformation; displace and dis-empower workers; stifle competition; and pose risks to national security."

Thus, to enhance the value and responsible use of AI, it is imperative to foster a healthy infrastructure ecosystem for evaluating novel AI methods, including how these systems interact with and impact their human users. NSF has long funded research and development in AI and in infrastructure for developing novel computational methods. The AI EO tasks NSF to expand on these efforts, to "ensure the availability of testing environments, such as test beds, to support the development of safe, secure, and trustworthy AI technologies." Through this DCL, NSF is providing funding for Planning Grants that support the science and engineering community to generate plans to enhance and expand existing test beds and infrastructure to make them "AI-ready" to support multi- and interdisciplinary research collaborations to test novel AI methods in real-time and real-world scenarios. To do this, Planning Grants should bring together teams of researchers who have expertise in AI with domain experts and staff managing existing test beds and testing infrastructure.

DESCRIPTION OF THE OPPORTUNITY

Planning Grants funded through this DCL are expected to cultivate research teams that actively address the expansion or enhancement of an existing test bed to evaluate the impact on and interaction with users of novel AI methods. Proposers supported through this DCL may use the funding to organize activities that help stimulate the formation of AI-Ready Test Bed teams [in terms of Principal Investigator (PI), co-PI, Senior/Key Personnel, and organization type] and crystalize the ideas and research plans to be presented in a future AI-Ready Test Bed proposal.

Examples of application domains cover all major areas, such as: urban and regional transportation networks; telecommunications networks; power grid control; development, monitoring, maintenance, and repair of urban infrastructure; multi-level healthcare delivery systems: criminal justice system; human and social services requiring coordination across several city or community offices; allocation of public services; public safety; automated farming and food distribution; autonomous scientific laboratories; and manufacturing including the industrial Internet of Things (IIOT), materials and labor pipelines, warehousing, and distribution.

Given the complexity of an AI-Ready Test Bed, NSF recognizes that many teams will identify important research priorities but may not have the full complement of skills needed to effectively address the challenge. The Planning Grant can be used to support team-formation activities that create opportunities for the development of partnerships between researchers, organizations and existing test beds or infrastructure that are bi-directional and mutually beneficial, thus engaging a wide array of perspectives and scientific talent to address the national needs and grand challenges presented in AI.

NSF seeks to broaden geographic and demographic participation in research programs. To that end, proposal submissions from organizations in Established Program to Stimulate Competitive Research (EPSCoR) jurisdictions (https://new.nsf.gov/funding/initiatives/epscor) are encouraged. Collaborative proposals led by an organization in an EPSCoR jurisdiction with subaward(s) to organization(s) in non-EPSCoR jurisdictions are also particularly welcomed.

AWARD SIZE AND DURATION

The budget for a planning proposal may be up to $100,000 per year, with total funding requested of up to $200,000 for up to 24 months.

PREPARATION AND SUBMISSION INFORMATION

To be considered for an AI-Ready Test Bed Planning Grant, planning proposals must be submitted by 5:00 p.m., submitting organization's local time, on November 13, 2024.

PIs must submit a Concept Outline to AI-Ready_testbeds@nsf.gov prior to submission of a planning proposal to aid in determining the appropriateness of the work for consideration under this opportunity. Guidance on Concept Outlines can be found in in Chapter I.D.1 of the NSF Proposal & Award Policies & Procedures Guide (PAPPG). Proposers must include the email from the NSF AI-Ready Test Beds team encouraging submission in the Program Officer Concurrence Email(s) section of the planning proposal.

Planning proposals must be prepared in accordance with the instructions in Chapter II.F.1 of the PAPPG. Per the guidance contained in PAPPG Chapter I.E, eligible proposing organizations include: Institutions of Higher Education, Non-profit, Non-academic Organizations and Tribal Nations. When preparing the planning proposal in Research.gov, proposers should select the PAPPG (NSF 24-1) as the funding opportunity and direct the proposal to the Human-Centered Computing Program, in the Information and Intelligent Systems Division of the CISE Directorate. The "Planning" type of proposal should be selected. The system will automatically insert the prepended title "Planning" and that should be followed by "AI-Ready". Please note that although the PAPPG accepts planning proposals at any time, proposals submitted in response to this DCL must be submitted by 5:00 p.m., submitting organization's local time, on November 13, 2024.

Planning proposals do not constitute any commitment on behalf of the PI/co-PI(s) or their organizations to submit a future proposal. Nor does award of a Planning Grant constitute any commitment on behalf of NSF to fund a future AI-Ready Test Bed proposal subsequently submitted by the Planning Grant team. Prospective PIs are encouraged to read this DCL and the PAPPG carefully for planning proposal preparation and submission requirements.

AI-Ready Test Bed planning proposals may be reviewed internally by NSF staff, reviewed in a panel, reviewed by ad hoc reviewers, or reviewed by any combination of these methods.

For further information, please contact: AI-Ready_testbeds@nsf.gov.

Sincerely,

Dr. Gregory Hager, Assistant Director
Directorate for Computer and Information Science and Engineering

Dr. Erwin Gianchandani, Assistant Director
Directorate for Technology, Innovation and Partnerships