Award Abstract # 1928614
FW-HTF-RL: Collaborative Research: Future expert work in the age of "black box", data-intensive, and algorithmically augmented healthcare
NSF Org: |
ECCS
Division of Electrical, Communications and Cyber Systems
|
Recipient: |
NEW YORK UNIVERSITY
|
Initial Amendment Date:
|
August 1, 2019 |
Latest Amendment Date:
|
November 17, 2022 |
Award Number: |
1928614 |
Award Instrument: |
Standard Grant |
Program Manager: |
Richard Nash
rnash@nsf.gov
(703)292-5394
ECCS
Division of Electrical, Communications and Cyber Systems
ENG
Directorate for Engineering
|
Start Date: |
September 1, 2019 |
End Date: |
August 31, 2025 (Estimated) |
Total Intended Award
Amount: |
$1,500,000.00 |
Total Awarded Amount to
Date: |
$1,548,000.00 |
Funds Obligated to Date:
|
FY 2019 = $1,500,000.00
FY 2020 = $16,000.00
FY 2021 = $16,000.00
FY 2023 = $16,000.00
|
History of Investigator:
|
-
Oded
Nov
(Principal Investigator)
on272@nyu.edu
-
Maurizio
Porfiri
(Co-Principal Investigator)
-
Batia
Wiesenfeld
(Co-Principal Investigator)
-
Yindalon
Aphinyanaphongs
(Co-Principal Investigator)
-
Yvonne
Lui
(Co-Principal Investigator)
|
Recipient Sponsored Research
Office: |
New York University
70 WASHINGTON SQ S
NEW YORK
NY
US
10012-1019
(212)998-2121
|
Sponsor Congressional
District: |
10
|
Primary Place of
Performance: |
New York University
5 Metrotech Center
New York
NY
US
10012-1019
|
Primary Place of
Performance Congressional District: |
10
|
Unique Entity Identifier
(UEI): |
NX9PXMKW5KW8
|
Parent UEI: |
|
NSF Program(s): |
FW-HTF Futr Wrk Hum-Tech Frntr, EPMD-ElectrnPhoton&MagnDevices
|
Primary Program Source:
|
01002324DB NSF RESEARCH & RELATED ACTIVIT
01001920DB NSF RESEARCH & RELATED ACTIVIT
01002021DB NSF RESEARCH & RELATED ACTIVIT
01002122DB NSF RESEARCH & RELATED ACTIVIT
|
Program Reference
Code(s): |
063Z,
116E,
1517,
9102,
9178,
9231,
9251
|
Program Element Code(s):
|
103Y00,
151700
|
Award Agency Code: |
4900
|
Fund Agency Code: |
4900
|
Assistance Listing
Number(s): |
47.041
|
ABSTRACT

The nature of expert work is changing. Technological advances such as artificial intelligence and data science increasingly enable new computerized tools and products that make predictions and recommendations which were previously made by human experts. However, many of these new tools are "black boxes" whose inner workings are often not understood by their users, place demands that create cognitive load, and de-emphasize abstract problem solving. As these technologies are being deployed, there is little understanding of how they affect experts' work practices, perceptions of the value of work, and the expert-client relationship. Foundational research is needed in order to understand and improve work in an age of data-intensive enhanced cognition, especially in healthcare where such new technologies are rapidly changing expert work. This project is expected to transform the future of expert work through a combined redesign of technology, workflow, and interactions. It will lead to: a healthier and better-informed population; efficient deployment of human capabilities in restructured healthcare occupations; healthcare providers reducing the proportion of time spent on repetitive tasks while increasing time devoted to value-adding, meaningful activities; guidelines on design and delivery of cognition-augmenting expert advice; and students who are well versed in cross-disciplinary research on cognition-augmenting technologies in the workplace.
The project's goals are: i) to study the relationships between experts, patients, and technologies in a multidisciplinary way; ii) to develop new ways for these technologies to serve experts and clients; and iii) to make expert work more responsive, value-adding, and meaningful. The project includes two strands. In the "Understand" strand, the interactions between experts, clients and cognition-augmenting technologies are examined. In the "Shape" strand, the project lays the foundations for technological and organizational interventions that will make the interactions between experts, clients, and technology more effective and empowering. With a multidisciplinary team including researchers in computer science, human-computer interaction, dynamical systems, and organization alongside with medical clinicians, the project will contribute: i) scalable approaches toward quantifying the benefits and drawbacks of cognition-augmented interactions, as well as measuring information flow in relationships between experts, clients, and cognition augmenting technologies; ii) insights into when, why, and how cognition-augmenting technologies are experienced as expertise enhancing, rather than degrading; iii) data-driven methodologies to predict the effects of technical and organizational interventions on experts' work and experts' interaction with patients; iv) novel tools and workflows for experts and clients to interact with black-box cognition-augmenting technologies; v) modeling how representation of problems can be embedded in expert; and vi) systematic exploration of explanation and dialogue interventions with regard to how they affect experts' work and expert-client relationship.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
PUBLICATIONS PRODUCED AS A RESULT OF THIS RESEARCH

Note:
When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external
site maintained by the publisher. Some full text articles may not yet be available without a
charge during the embargo (administrative interval).
Some links on this page may take you to non-federal websites. Their policies may differ from
this site.
(Showing: 1 - 10 of 43)
(Showing: 1 - 43 of 43)
Barak Ventura, Roni and Stewart Hughes, Kora and Nov, Oded and Raghavan, Preeti and Ruiz Marín, Manuel and Porfiri, Maurizio
"Data-Driven Classification of Human Movements in Virtual RealityBased Serious Games: Preclinical Rehabilitation Study in Citizen Science"
JMIR Serious Games
, v.10
, 2022
https://doi.org/10.2196/27597
Citation
Details
Bell, Andrew and Nov, Oded and Stoyanovich, Julia
"The Algorithmic Transparency Playbook: A Stakeholder-first Approach to Creating Transparency for Your Organizations Algorithms"
2023 CHI Conference on Human Factors in Computing Systems
, 2023
https://doi.org/10.1145/3544549.3574169
Citation
Details
Bell, Andrew and Solano-Kamaiko, Ian and Nov, Oded and Stoyanovich, Julia
"Its Just Not That Simple: An Empirical Study of the Accuracy-Explainability Trade-off in Machine Learning for Public Policy"
ACM Conference on Fairness, Accountability, and Transparency 2022 Pages 248266
, 2022
https://doi.org/10.1145/3531146.3533090
Citation
Details
Chunara, Rumi and Zhao, Yuan and Chen, Ji and Lawrence, Katharine and Testa, Paul A and Nov, Oded and Mann, Devin M
"Telemedicine and Healthcare Disparities: A cohort study in a large healthcare system in New York City during COVID-19"
Journal of the American Medical Informatics Association
, 2020
https://doi.org/10.1093/jamia/ocaa217
Citation
Details
De_Lellis, Pietro and Ruiz_Marín, Manuel and Porfiri, Maurizio
"Inferring directional interactions in collective dynamics: a critique to intrinsic mutual information"
Journal of Physics: Complexity
, v.4
, 2022
https://doi.org/10.1088/2632-072X/acace0
Citation
Details
Dove, Graham and Balestra, Martina and Mann, Devin and Nov, Oded
"Good for the Many or Best for the Few?: A Dilemma in the Design of Algorithmic Advice"
Proceedings of the ACM on Human-Computer Interaction
, v.4
, 2020
https://doi.org/10.1145/3415239
Citation
Details
Dove, Graham and Fernando, Adelle and Hertz, Kim and Kim, Jin and Rizzo, John-Ross and Seiple, William H. and Nov, Oded
"Digital Technologies in Orientation and Mobility Instruction for People Who are Blind or Have Low Vision"
Proceedings of the ACM on Human-Computer Interaction
, v.6
, 2022
https://doi.org/10.1145/3555622
Citation
Details
Dove, Graham and Guthmann, Marina Roos and Charvet, Leigh and Nov, Oded and Pilloni, Giuseppina
""Data Is One Thing, But I Want To Know The Story Behind": Designing For Self-Tracking and Remote Patient Monitoring In The Context Of Multiple Sclerosis Care"
, 2024
https://doi.org/10.1145/3643834.3661499
Citation
Details
Feng, Junchi and Beheshti, Mahya and Philipson, Mira and Ramsaywack, Yuvraj and Porfiri, Maurizio and Rizzo, John-Ross
"Commute Booster: A Mobile Application for First/Last Mile and Middle Mile Navigation Support for People With Blindness and Low Vision"
IEEE Journal of Translational Engineering in Health and Medicine
, v.11
, 2023
https://doi.org/10.1109/JTEHM.2023.3293450
Citation
Details
Gan, Tian and Das, Rishita and Porfiri, Maurizio
"Network Modeling of Consumers' Selection of Providers Based on Online Reviews"
IEEE Transactions on Network Science and Engineering
, v.11
, 2024
https://doi.org/10.1109/TNSE.2023.3348654
Citation
Details
Lawrence, Katharine and Levine, Defne L
"The Digital Determinants of Health: A Guide for Competency Development in Digital Care Delivery for Health Professions Trainees"
JMIR Medical Education
, v.10
, 2024
https://doi.org/10.2196/54173
Citation
Details
Lawrence, Katharine and Nov, Oded and Mann, Devin and Mandal, Soumik and Iturrate, Eduardo and Wiesenfeld, Batia
"The Impact of Telemedicine on Physicians After-hours Electronic Health Record Work Outside Work During the COVID-19 Pandemic: Retrospective Cohort Study"
JMIR Medical Informatics
, v.10
, 2022
https://doi.org/10.2196/34826
Citation
Details
Lawrence, Katharine and Singh, Nina and Jonassen, Zoe and Groom, Lisa L and Alfaro Arias, Veronica and Mandal, Soumik and Schoenthaler, Antoinette and Mann, Devin and Nov, Oded and Dove, Graham
"Operational Implementation of Remote Patient Monitoring Within a Large Ambulatory Health System: Multimethod Qualitative Case Study"
JMIR Human Factors
, v.10
, 2023
https://doi.org/10.2196/45166
Citation
Details
Lotan, E. and Zhang, B. and Dogra, S. and Wang, W.D. and Carbone, D. and Fatterpekar, G. and Oermann, E.K. and Lui, Y.W.
"Development and Practical Implementation of a Deep LearningBased Pipeline for Automated Pre- and Postoperative Glioma Segmentation"
American Journal of Neuroradiology
, v.43
, 2022
https://doi.org/10.3174/ajnr.A7363
Citation
Details
Major, Vincent J. and Jones, Simon A. and Razavian, Narges and Bagheri, Ashley and Mendoza, Felicia and Stadelman, Jay and Horwitz, Leora I. and Austrian, Jonathan and Aphinyanaphongs, Yindalon
"Evaluating the Effect of a COVID-19 Predictive Model to Facilitate Discharge: A Randomized Controlled Trial"
Applied Clinical Informatics
, v.13
, 2022
https://doi.org/10.1055/s-0042-1750416
Citation
Details
Mandal, Soumik and Wiesenfeld, Batia M and Mann, Devin and Lawrence, Katharine and Chunara, Rumi and Testa, Paul and Nov, Oded
"Evidence for Telemedicines Ongoing Transformation of Health Care Delivery Since the Onset of COVID-19: Retrospective Observational Study"
JMIR Formative Research
, v.6
, 2022
https://doi.org/10.2196/38661
Citation
Details
Mandal, Soumik and Wiesenfeld, Batia M. and Mann, Devin M. and Szerencsy, Adam C. and Iturrate, Eduardo and Nov, Oded
"Quantifying the impact of telemedicine and patient medical advice request messages on physicians' work-outside-work"
npj Digital Medicine
, v.7
, 2024
https://doi.org/10.1038/s41746-024-01001-2
Citation
Details
Mann, Devin M and Chen, Ji and Chunara, Rumi and Testa, Paul A and Nov, Oded
"COVID-19 transforms health care through telemedicine: Evidence from the field"
Journal of the American Medical Informatics Association
, v.27
, 2020
10.1093/jamia/ocaa072
Citation
Details
Méndez Méndez, Ana Elisa and Cartwright, Mark and Bello, Juan Pablo and Nov, Oded
"Eliciting Confidence for Improving Crowdsourced Audio Annotations"
Proceedings of the ACM on Human-Computer Interaction
, v.6
, 2022
https://doi.org/10.1145/3512935
Citation
Details
Nakayama, Shinnosuke and Richmond, Samuel and Nov, Oded and Porfiri, Maurizio
"The gold miner's dilemma: Use of information scent in cooperative and competitive information foraging"
Computers in Human Behavior
, v.109
, 2020
https://doi.org/10.1016/j.chb.2020.106352
Citation
Details
Nov, Oded and Aphinyanaphongs, Yindalon and Lui, Yvonne W. and Mann, Devin and Porfiri, Maurizio and Riedl, Mark and Rizzo, John-Ross and Wiesenfeld, Batia
"The transformation of patient-clinician relationships with AI-based medical advice"
Communications of the ACM
, v.64
, 2021
https://doi.org/10.1145/3417518
Citation
Details
Nov, Oded and Singh, Nina and Mann, Devin
"Putting ChatGPTs Medical Advice to the (Turing) Test: Survey Study"
JMIR Medical Education
, v.9
, 2023
https://doi.org/10.2196/46939
Citation
Details
Porfiri, Maurizio
"Validity and Limitations of the Detection Matrix to Determine Hidden Units and Network Size from Perceptible Dynamics"
Physical Review Letters
, v.124
, 2020
10.1103/PhysRevLett.124.168301
Citation
Details
Razavian, Narges and Major, Vincent J. and Sudarshan, Mukund and Burk-Rafel, Jesse and Stella, Peter and Randhawa, Hardev and Bilaloglu, Seda and Chen, Ji and Nguy, Vuthy and Wang, Walter and Zhang, Hao and Reinstein, Ilan and Kudlowitz, David and Zenger,
"A validated, real-time prediction model for favorable outcomes in hospitalized COVID-19 patients"
npj Digital Medicine
, v.3
, 2020
https://doi.org/10.1038/s41746-020-00343-x
Citation
Details
Ricci, Fabiana Sofia and Boldini, Alain and Beheshti, Mahya and Rizzo, John-Ross and Porfiri, Maurizio
"A virtual reality platform to simulate orientation and mobility training for the visually impaired"
Virtual Reality
, 2022
https://doi.org/10.1007/s10055-022-00691-x
Citation
Details
Ricci, Fabiana Sofia and Boldini, Alain and Ma, Xinda and Beheshti, Mahya and Geruschat, Duane R. and Seiple, William H. and Rizzo, John-Ross and Porfiri, Maurizio
"Virtual reality as a means to explore assistive technologies for the visually impaired"
PLOS Digital Health
, v.2
, 2023
https://doi.org/10.1371/journal.pdig.0000275
Citation
Details
Ruiz Marín, Manuel and Villegas Martínez, Irene and Rodríguez Bermúdez, Germán and Porfiri, Maurizio
"Integrating old and new complexity measures toward automated seizure detection from long-term video EEG recordings"
iScience
, v.24
, 2021
https://doi.org/10.1016/j.isci.2020.101997
Citation
Details
Seals, Ayanna and Pilloni, Giuseppina and Kim, Jin and Sanchez, Raul and Rizzo, John-Ross and Charvet, Leigh and Nov, Oded and Dove, Graham
"Are They Doing Better In The Clinic Or At Home?: Understanding Clinicians Needs When Visualizing Wearable Sensor Data Used In Remote Gait Assessments For People With Multiple Sclerosis"
Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems
, 2022
https://doi.org/10.1145/3491102.3501989
Citation
Details
Singh, Nina and Lawrence, Katharine and Andreadis, Katerina and Twan, Chelsea and Mann, Devin M
"Developing and Scaling Remote Patient Monitoring Capacity in Ambulatory Practice"
NEJM Catalyst
, v.5
, 2024
https://doi.org/10.1056/CAT.23.0417
Citation
Details
Small, William R and Malhotra, Kiran and Major, Vincent J and Wiesenfeld, Batia and Lewis, Marisa and Grover, Himanshu and Tang, Huming and Banerjee, Arnab and Jabbour, Michael J and Aphinyanaphongs, Yindalon and Testa, Paul and Austrian, Jonathan S
"The First Generative AI Prompt-A-Thon in Healthcare: A Novel Approach to Workforce Engagement with a Private Instance of ChatGPT"
PLOS Digital Health
, v.3
, 2024
https://doi.org/10.1371/journal.pdig.0000394
Citation
Details
Small, William R and Wiesenfeld, Batia and Brandfield-Harvey, Beatrix and Jonassen, Zoe and Mandal, Soumik and Stevens, Elizabeth R and Major, Vincent J and Lostraglio, Erin and Szerencsy, Adam and Jones, Simon and Aphinyanaphongs, Yindalon and Johnson, S
"Large Language ModelBased Responses to Patients In-Basket Messages"
JAMA Network Open
, v.7
, 2024
https://doi.org/10.1001/jamanetworkopen.2024.22399
Citation
Details
Succar, Rayan and Boldini, Alain and Porfiri, Maurizio
"Detecting hidden states in stochastic dynamical systems"
Physical Review Research
, v.6
, 2024
https://doi.org/10.1103/PhysRevResearch.6.013149
Citation
Details
Wiesenfeld, Batia Mishan and Aphinyanaphongs, Yin and Nov, Oded
"AI model transferability in healthcare: a sociotechnical perspective"
Nature Machine Intelligence
, v.4
, 2022
https://doi.org/10.1038/s42256-022-00544-x
Citation
Details
Woo, Kar-mun_C and Simon, Gregory_W and Akindutire, Olumide and Aphinyanaphongs, Yindalon and Austrian, Jonathan_S and Kim, Jung_G and Genes, Nicholas and Goldenring, Jacob_A and Major, Vincent_J and Pariente, Chloé_S and Pineda, Edwin_G and Kang, Stella_
"Evaluation of GPT-4 ability to identify and generate patient instructions for actionable incidental radiology findings"
Journal of the American Medical Informatics Association
, v.31
, 2024
https://doi.org/10.1093/jamia/ocae117
Citation
Details
Yang, Anbang and Beheshti, Mahya and Hudson, Todd E. and Vedanthan, Rajesh and Riewpaiboon, Wachara and Mongkolwat, Pattanasak and Feng, Chen and Rizzo, John-Ross
"UNav: An Infrastructure-Independent Vision-Based Navigation System for People with Blindness and Low Vision"
Sensors
, v.22
, 2022
https://doi.org/10.3390/s22228894
Citation
Details
Yang, Elisabeth and Aphinyanaphongs, Yindalon and Punjabi, PV and Austrian, Jonathan and Wiesenfeld, Batia
"Quantitative and Qualitative Evaluation of Provider Use of a Novel Machine Learning Model for Favorable Outcome Prediction"
, 2023
Citation
Details
(Showing: 1 - 10 of 43)
(Showing: 1 - 43 of 43)
Please report errors in award information by writing to: awardsearch@nsf.gov.