Award Abstract # 1928481
FW-HTF-RM: Measuring learning gains in man-machine assemblage when augmenting radiology work with artificial intelligence

NSF Org: CMMI
Division of Civil, Mechanical, and Manufacturing Innovation
Recipient: TRUSTEES OF INDIANA UNIVERSITY
Initial Amendment Date: August 1, 2019
Latest Amendment Date: August 1, 2019
Award Number: 1928481
Award Instrument: Standard Grant
Program Manager: Alexandra Medina-Borja
amedinab@nsf.gov
 (703)292-7557
CMMI
 Division of Civil, Mechanical, and Manufacturing Innovation
ENG
 Directorate for Engineering
Start Date: September 1, 2019
End Date: August 31, 2023 (Estimated)
Total Intended Award Amount: $826,795.00
Total Awarded Amount to Date: $826,795.00
Funds Obligated to Date: FY 2019 = $826,795.00
History of Investigator:
  • Saptarshi Purkayastha (Principal Investigator)
    saptpurk@iu.edu
  • Joshua Danish (Co-Principal Investigator)
  • Elizabeth Krupinski (Co-Principal Investigator)
  • Judy Gichoya (Co-Principal Investigator)
Recipient Sponsored Research Office: Indiana University
107 S INDIANA AVE
BLOOMINGTON
IN  US  47405-7000
(317)278-3473
Sponsor Congressional District: 09
Primary Place of Performance: Indiana University
545 w. Michigan st
indianapolis
IN  US  46202-2915
Primary Place of Performance
Congressional District:
07
Unique Entity Identifier (UEI): YH86RTW2YVJ4
Parent UEI:
NSF Program(s): FW-HTF Futr Wrk Hum-Tech Frntr
Primary Program Source: 01001920DB NSF RESEARCH & RELATED ACTIVIT
Program Reference Code(s): 063Z
Program Element Code(s): 103Y00
Award Agency Code: 4900
Fund Agency Code: 4900
Assistance Listing Number(s): 47.041

ABSTRACT

The work setting of the future presents an opportunity for human-technology partnerships, where a harmonious connection between human-technology produces unprecedented productivity gains. A conundrum at this human-technology frontier remains - will humans be augmented by technology or will technology be augmented by humans? This project overcomes the conundrum of human and machine as separate entities and instead, treats them as an assemblage. As groundwork for the harmonious human-technology connection, this assemblage needs to learn to fit synergistically. This learning is called assemblage learning and it will be important for Artificial Intelligence (AI) applications in health care, where diagnostic and treatment decisions augmented by AI will have a direct and significant impact on patient care and outcomes. This project will also identify ways in which learning can be shared between assemblages, such that collective swarms of connected assemblages can be created. The project will create a new learning model that integrates and measures concepts from individuals learning to swarm learn. The project will help demonstrate a symbiotic learning assemblage, such that envisioned productivity gains from AI can be achieved without loss of human jobs. Even though the focus is on visual cognitive tasks in radiology, lessons from this project may be applicable to other domains where human intelligence will be augmented by machine intelligence.

Recent studies of human versus machine competitions have demonstrated that assemblages that combine human-technology partnerships are stronger than individual humans or machines. By building on these, this project will integrate state-of-the-art algorithms into the radiology workflow. The project will answer the following research questions: Q1: How to develop assemblages, such that human-technology partnerships produce a "good fit" for visually based cognition-oriented tasks in radiology? Q2: What level of training should pre-exist in the individual human (radiologist) and independent machine learning model for human-technology partnerships to thrive? Q3: Which aspects and to what extent does an assemblage learning approach lead to reduced errors, improved accuracy, faster turn-around times, reduced fatigue, improved self-efficacy, and resilience? A rigorous counterbalanced trial will be performed to assess individual radiologists interpreting images with and without the assemblage. Data on clinician engagement from EHR systems will be captured and analyzed, along with pre-test and post-test surveys and interviews. Deep and wide analysis of the quantitative and qualitative data from the trial will answer questions related to learning gains, task performance, emotional as well as behavioral aspects of learning in an assemblage. The project employs perspectives from Science & Technology Studies, Computer Science, Psychology, and Learning Sciences, to create and study assemblages that can produce gains in routine radiology work.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

PUBLICATIONS PRODUCED AS A RESULT OF THIS RESEARCH

Note:  When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external site maintained by the publisher. Some full text articles may not yet be available without a charge during the embargo (administrative interval).

Some links on this page may take you to non-federal websites. Their policies may differ from this site.

(Showing: 1 - 10 of 13)
Purkayastha, Saptarshi and Trivedi, Hari and Gichoya, Judy Wawira "Failures Hiding in Success for Artificial Intelligence in Radiology" Journal of the American College of Radiology , v.18 , 2021 https://doi.org/10.1016/j.jacr.2020.11.008 Citation Details
Abid, A. and Sinha, P. and Harpale, A. and Gichoya, J. and Purkayastha, S. "Optimizing Medical Image Classification Models for Edge Devices" Distributed Computing and Artificial Intelligence, Volume 1: 18th International Conference , v.1 , 2021 https://doi.org/10.1007/978-3-030-86261-9_8 Citation Details
Banerjee, Imon and Bhattacharjee, Kamanasish and Burns, John L. and Trivedi, Hari and Purkayastha, Saptarshi and Seyyed-Kalantari, Laleh and Patel, Bhavik N. and Shiradkar, Rakesh and Gichoya, Judy "Shortcuts Causing Bias in Radiology Artificial Intelligence: Causes, Evaluation, and Mitigation" Journal of the American College of Radiology , v.20 , 2023 https://doi.org/10.1016/j.jacr.2023.06.025 Citation Details
Bhimireddy, Ananth and Sinha, Priyanshu and Oluwalade, Bolu and Gichoya, Judy W and Purkayastha, Saptarshi "Blood Glucose Level Prediction as Time-Series Modeling using Sequence-to-Sequence Neural Networks" CEUR workshop proceedings , 2020 https://doi.org/ Citation Details
Burns, John Lee and Zaiman, Zachary and Vanschaik, Jack and Luo, Gaoxiang and Peng, Le and Price, Brandon and Mathias, Garric and Mittal, Vijay and Sagane, Akshay and Tignanelli, Christopher and Chakraborty, Sunandan and Gichoya, Judy Wawira and Purkayast "Ability of artificial intelligence to identify self-reported race in chest x-ray using pixel intensity counts" Journal of Medical Imaging , v.10 , 2023 https://doi.org/10.1117/1.JMI.10.6.061106 Citation Details
Gichoya, Judy Wawira and Banerjee, Imon and Bhimireddy, Ananth Reddy and Burns, John L and Celi, Leo Anthony and Chen, Li-Ching and Correa, Ramon and Dullerud, Natalie and Ghassemi, Marzyeh and Huang, Shih-Cheng and Kuo, Po-Chih and Lungren, Matthew P and "AI recognition of patient race in medical imaging: a modelling study" The Lancet Digital Health , v.4 , 2022 https://doi.org/10.1016/S2589-7500(22)00063-2 Citation Details
Gichoya, Judy Wawira and Thomas, Kaesha and Celi, Leo Anthony and Safdar, Nabile and Banerjee, Imon and Banja, John D and Seyyed-Kalantari, Laleh and Trivedi, Hari and Purkayastha, Saptarshi "AI pitfalls and what not to do: mitigating bias in AI" The British Journal of Radiology , v.96 , 2023 https://doi.org/10.1259/bjr.20230023 Citation Details
Guo, Xiaoyuan and Gichoya, Judy W. and Purkayastha, Saptarshi and Banerjee, Imon "Margin-aware intraclass novelty identification for medical images" Journal of Medical Imaging , v.9 , 2022 https://doi.org/10.1117/1.JMI.9.1.014004 Citation Details
Guo, Xiaoyuan and Gichoya, Judy Wawira and Trivedi, Hari and Purkayastha, Saptarshi and Banerjee, Imon "MedShift: Automated Identification of Shift Data for Medical Image Dataset Curation" IEEE Journal of Biomedical and Health Informatics , v.27 , 2023 https://doi.org/10.1109/JBHI.2023.3275104 Citation Details
Kathiravelu, Pradeeban and Sharma, Puneet and Sharma, Ashish and Banerjee, Imon and Trivedi, Hari and Purkayastha, Saptarshi and Sinha, Priyanshu and Cadrin-Chenevert, Alexandre and Safdar, Nabile and Gichoya, Judy Wawira "A DICOM Framework for Machine Learning and Processing Pipelines Against Real-time Radiology Images" Journal of Digital Imaging , 2021 https://doi.org/10.1007/s10278-021-00491-w Citation Details
Sinha, Priyanshu and Gichoya, Judy W. and Purkayastha, Saptarshi "Leapfrogging Medical AI in Low-Resource Contexts Using Edge Tensor Processing Unit" 2022 IEEE Healthcare Innovations and Point of Care Technologies (HI-POCT) , 2022 https://doi.org/10.1109/HI-POCT54491.2022.9744071 Citation Details
(Showing: 1 - 10 of 13)

PROJECT OUTCOMES REPORT

Disclaimer

This Project Outcomes Report for the General Public is displayed verbatim as submitted by the Principal Investigator (PI) for this award. Any opinions, findings, and conclusions or recommendations expressed in this Report are those of the PI and do not necessarily reflect the views of the National Science Foundation; NSF has not approved or endorsed its content.

This NSF-funded project investigated how radiologists and artificial intelligence (AI) systems can work together in a symbiotic relationship to improve diagnosis of medical images. The researchers integrated multiple state-of-the-art AI image analysis models into an open-source radiology workflow system called LibreHealth Radiology Information System (RIS). This allows radiologists to get real-time insights from the AI models as they are interpreting medical images.

The researchers then conducted rigorous studies where radiologists read sets of 60-120 medical images both with and without the AI assistance at Emory University and Indiana University. Quantitative performance metrics and qualitative feedback through surveys and interviews were collected. The studies found that the radiologists performed better on diagnostic accuracy metrics with the AI assistant. The radiologists also subjectively reported increased confidence, reduced fatigue, and greater efficiency when working alongside the AI.

Through corrections and feedback the radiologists provided, the AI models were also able to improve their performance. This mutually beneficial relationship between humans and AI was termed "assemblage learning" by the researchers. It demonstrates how AI does not have to replace human roles, but can augment human capabilities and cognition. The radiologist-AI assemblages outperformed either alone. This symbiotic paradigm has significant impacts for human-AI collaboration in many fields beyond radiology.

To facilitate further research, the project developed and released several open-source software tools. LibreHealth RIS allows integration of custom AI services into clinical workflows. The AI Model Service provides a standardized API for connecting different AI algorithms. These tools lower barriers for exploring human-AI interaction in real clinical environments.

The researchers also published multiple papers investigating bias in AI systems. They found that while high-performing models can encode racial, gender, and other biases, human-in-the-loop oversight during model development and application can combat this. Human interaction can guide AI systems to become fairer and more trustworthy.

Related works have looked at human-AI collaboration in domains like manufacturing, finance, and self-driving vehicles. However, this project uniquely combined real clinical systems, practicing radiologists, and rigorous mixed-methods studies. The findings contribute conceptual knowledge on designing AI agents that productively collaborate with human experts, not just replace them. This "AI augmentation" paradigm maintains human oversight, transparency, and trust.

With over 12 papers published, the project made critical contributions at the intersection of human-computer interaction, health informatics, and machine learning. The open-source platforms developed will enable future research to continue investigating safe and effective integration of AI in clinical practice. Overall, the project advanced knowledge and tools to design AI as an ally rather than adversary to human professionals.


Last Modified: 01/29/2024
Modified by: Saptarshi Purkayastha

Please report errors in award information by writing to: awardsearch@nsf.gov.

Print this page

Back to Top of page