
NSF Org: |
CMMI Division of Civil, Mechanical, and Manufacturing Innovation |
Recipient: |
|
Initial Amendment Date: | August 1, 2019 |
Latest Amendment Date: | August 1, 2019 |
Award Number: | 1928481 |
Award Instrument: | Standard Grant |
Program Manager: |
Alexandra Medina-Borja
amedinab@nsf.gov (703)292-7557 CMMI Division of Civil, Mechanical, and Manufacturing Innovation ENG Directorate for Engineering |
Start Date: | September 1, 2019 |
End Date: | August 31, 2023 (Estimated) |
Total Intended Award Amount: | $826,795.00 |
Total Awarded Amount to Date: | $826,795.00 |
Funds Obligated to Date: |
|
History of Investigator: |
|
Recipient Sponsored Research Office: |
107 S INDIANA AVE BLOOMINGTON IN US 47405-7000 (317)278-3473 |
Sponsor Congressional District: |
|
Primary Place of Performance: |
545 w. Michigan st indianapolis IN US 46202-2915 |
Primary Place of
Performance Congressional District: |
|
Unique Entity Identifier (UEI): |
|
Parent UEI: |
|
NSF Program(s): | FW-HTF Futr Wrk Hum-Tech Frntr |
Primary Program Source: |
|
Program Reference Code(s): |
|
Program Element Code(s): |
|
Award Agency Code: | 4900 |
Fund Agency Code: | 4900 |
Assistance Listing Number(s): | 47.041 |
ABSTRACT
The work setting of the future presents an opportunity for human-technology partnerships, where a harmonious connection between human-technology produces unprecedented productivity gains. A conundrum at this human-technology frontier remains - will humans be augmented by technology or will technology be augmented by humans? This project overcomes the conundrum of human and machine as separate entities and instead, treats them as an assemblage. As groundwork for the harmonious human-technology connection, this assemblage needs to learn to fit synergistically. This learning is called assemblage learning and it will be important for Artificial Intelligence (AI) applications in health care, where diagnostic and treatment decisions augmented by AI will have a direct and significant impact on patient care and outcomes. This project will also identify ways in which learning can be shared between assemblages, such that collective swarms of connected assemblages can be created. The project will create a new learning model that integrates and measures concepts from individuals learning to swarm learn. The project will help demonstrate a symbiotic learning assemblage, such that envisioned productivity gains from AI can be achieved without loss of human jobs. Even though the focus is on visual cognitive tasks in radiology, lessons from this project may be applicable to other domains where human intelligence will be augmented by machine intelligence.
Recent studies of human versus machine competitions have demonstrated that assemblages that combine human-technology partnerships are stronger than individual humans or machines. By building on these, this project will integrate state-of-the-art algorithms into the radiology workflow. The project will answer the following research questions: Q1: How to develop assemblages, such that human-technology partnerships produce a "good fit" for visually based cognition-oriented tasks in radiology? Q2: What level of training should pre-exist in the individual human (radiologist) and independent machine learning model for human-technology partnerships to thrive? Q3: Which aspects and to what extent does an assemblage learning approach lead to reduced errors, improved accuracy, faster turn-around times, reduced fatigue, improved self-efficacy, and resilience? A rigorous counterbalanced trial will be performed to assess individual radiologists interpreting images with and without the assemblage. Data on clinician engagement from EHR systems will be captured and analyzed, along with pre-test and post-test surveys and interviews. Deep and wide analysis of the quantitative and qualitative data from the trial will answer questions related to learning gains, task performance, emotional as well as behavioral aspects of learning in an assemblage. The project employs perspectives from Science & Technology Studies, Computer Science, Psychology, and Learning Sciences, to create and study assemblages that can produce gains in routine radiology work.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
PUBLICATIONS PRODUCED AS A RESULT OF THIS RESEARCH
Note:
When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external
site maintained by the publisher. Some full text articles may not yet be available without a
charge during the embargo (administrative interval).
Some links on this page may take you to non-federal websites. Their policies may differ from
this site.
PROJECT OUTCOMES REPORT
Disclaimer
This Project Outcomes Report for the General Public is displayed verbatim as submitted by the Principal Investigator (PI) for this award. Any opinions, findings, and conclusions or recommendations expressed in this Report are those of the PI and do not necessarily reflect the views of the National Science Foundation; NSF has not approved or endorsed its content.
This NSF-funded project investigated how radiologists and artificial intelligence (AI) systems can work together in a symbiotic relationship to improve diagnosis of medical images. The researchers integrated multiple state-of-the-art AI image analysis models into an open-source radiology workflow system called LibreHealth Radiology Information System (RIS). This allows radiologists to get real-time insights from the AI models as they are interpreting medical images.
The researchers then conducted rigorous studies where radiologists read sets of 60-120 medical images both with and without the AI assistance at Emory University and Indiana University. Quantitative performance metrics and qualitative feedback through surveys and interviews were collected. The studies found that the radiologists performed better on diagnostic accuracy metrics with the AI assistant. The radiologists also subjectively reported increased confidence, reduced fatigue, and greater efficiency when working alongside the AI.
Through corrections and feedback the radiologists provided, the AI models were also able to improve their performance. This mutually beneficial relationship between humans and AI was termed "assemblage learning" by the researchers. It demonstrates how AI does not have to replace human roles, but can augment human capabilities and cognition. The radiologist-AI assemblages outperformed either alone. This symbiotic paradigm has significant impacts for human-AI collaboration in many fields beyond radiology.
To facilitate further research, the project developed and released several open-source software tools. LibreHealth RIS allows integration of custom AI services into clinical workflows. The AI Model Service provides a standardized API for connecting different AI algorithms. These tools lower barriers for exploring human-AI interaction in real clinical environments.
The researchers also published multiple papers investigating bias in AI systems. They found that while high-performing models can encode racial, gender, and other biases, human-in-the-loop oversight during model development and application can combat this. Human interaction can guide AI systems to become fairer and more trustworthy.
Related works have looked at human-AI collaboration in domains like manufacturing, finance, and self-driving vehicles. However, this project uniquely combined real clinical systems, practicing radiologists, and rigorous mixed-methods studies. The findings contribute conceptual knowledge on designing AI agents that productively collaborate with human experts, not just replace them. This "AI augmentation" paradigm maintains human oversight, transparency, and trust.
With over 12 papers published, the project made critical contributions at the intersection of human-computer interaction, health informatics, and machine learning. The open-source platforms developed will enable future research to continue investigating safe and effective integration of AI in clinical practice. Overall, the project advanced knowledge and tools to design AI as an ally rather than adversary to human professionals.
Last Modified: 01/29/2024
Modified by: Saptarshi Purkayastha
Please report errors in award information by writing to: awardsearch@nsf.gov.