Award Abstract # 2222129
Collaborative Research: FW-HTF-R: Toward an Ecosystem of Artificial Intelligence-Powered Music Production (TEAMuP)

NSF Org: IIS
Division of Information & Intelligent Systems
Recipient: UNIVERSITY OF ROCHESTER
Initial Amendment Date: September 12, 2022
Latest Amendment Date: December 23, 2022
Award Number: 2222129
Award Instrument: Standard Grant
Program Manager: Scott Robertson
sroberts@nsf.gov
 (703)292-2971
IIS
 Division of Information & Intelligent Systems
CSE
 Directorate for Computer and Information Science and Engineering
Start Date: October 1, 2022
End Date: September 30, 2026 (Estimated)
Total Intended Award Amount: $1,413,858.00
Total Awarded Amount to Date: $1,413,858.00
Funds Obligated to Date: FY 2022 = $1,413,858.00
History of Investigator:
  • Raffaella Borasi (Principal Investigator)
    rborasi@warner.rochester.edu
  • Zhiyao Duan (Co-Principal Investigator)
  • Jonathan Herington (Co-Principal Investigator)
  • Rachel Roberts (Co-Principal Investigator)
Recipient Sponsored Research Office: University of Rochester
910 GENESEE ST
ROCHESTER
NY  US  14611-3847
(585)275-4031
Sponsor Congressional District: 25
Primary Place of Performance: University of Rochester
500 JOSEPH C WILSON BLVD
ROCHESTER
NY  US  14627-0140
Primary Place of Performance
Congressional District:
25
Unique Entity Identifier (UEI): F27KDXZMF9Y8
Parent UEI:
NSF Program(s): FW-HTF Futr Wrk Hum-Tech Frntr
Primary Program Source: 01002223DB NSF RESEARCH & RELATED ACTIVIT
Program Reference Code(s): 063Z
Program Element Code(s): 103Y00
Award Agency Code: 4900
Fund Agency Code: 4900
Assistance Listing Number(s): 47.041, 47.070

ABSTRACT

This project builds the foundations of a new ecosystem for music production to empower future musicians to better leverage Artificial Intelligence (AI) tools in the creation, performance, and dissemination of their music, while also accelerating audio AI research. This involves the creation of both an open-access software framework enabling musicians and researchers to collaborate in the development and use of ever-better AI-powered tools for music creation, and a set of initiatives to enable a critical mass of musicians to use these tools in transformative ways. Musicians are expected to use these tools to produce lower-cost, higher-quality music products, which meet growing demand for digital music content for videos, websites, advertising, audio recordings, and other new media. Enabling musicians to be more self-sufficient in their music creation has the potential to increase the number of musically talented individuals that will be able to make a living with their art, especially from currently under-represented populations. To enable growing musicians to make full use of AI tools, a set of innovative learning experiences to acquire the needed mindsets and skills will be developed and field tested in a 2-semester course for students with music interests and a ?Summer Camp? for pre-college under-represented youth, along with the creation of online instructional materials to support specific learning experiences in a variety of settings.

The project team possesses complementary disciplinary expertise in music, audio-engineering, AI, learning sciences/ education, business/ entrepreneurship, ethics, and inclusion. These skills will be brought to bear on developing a framework for a commonly-used free and open-source digital audio platform that will allow: (a) audio AI researchers to easily deploy their new AI models into the platform; and, b) musicians who use these AI tools to share their music productions with AI researchers so they can refine their models. Interviews and surveys will also be conducted with diverse musicians to better understand key factors that may affect their adoption of AI music production tools and how those tools may transform their work, as well as the implications of the pandemic and other barriers that may be experienced by under-represented populations in music production. Together, the project will generate a better understanding of factors that may affect musicians? adoption and transformative use of AI in their work, understanding which could be generalized to other occupations at the human-technology frontier. Finally, the team will develop pedagogical principles and practices that can inform the design of effective educational interventions to better prepare future musicians and other domain experts to leverage technology.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

PUBLICATIONS PRODUCED AS A RESULT OF THIS RESEARCH

Note:  When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external site maintained by the publisher. Some full text articles may not yet be available without a charge during the embargo (administrative interval).

Some links on this page may take you to non-federal websites. Their policies may differ from this site.

Yang, Qiaoyu and Duan, Zhiyao "Harmonic analysis with neural semi-CRF" , 2023 Citation Details
Zang, Yongyi and Zhong, Yi and Cwitkowitz, Frank and Duan, Zhiyao "SynthTab: Leveraging Synthesized Data for Guitar Tablature Transcription" , 2024 https://doi.org/10.1109/ICASSP48485.2024.10447902 Citation Details
Zang, Yongyi and Zhang, You and Heydari, Mojtaba and Duan, Zhiyao "SingFake: Singing Voice Deepfake Detection" , 2024 https://doi.org/10.1109/ICASSP48485.2024.10448184 Citation Details
Zang, Yongyi and Shi, Jiatong and Zhang, You and Yamamoto, Ryuichi and Han, Jionghao and Tang, Yuxun and Xu, Shengyuan and Zhao, Wenxiao and Guo, Jing and Toda, Tomoki and Duan, Zhiyao "CtrSVDD: A Benchmark Dataset and Baseline Analysis for Controlled Singing Voice Deepfake Detection" , 2024 Citation Details
Zang, Yongyi and Benetatos, Christodoulos and Duan, Zhiyao "Euterpe: A Web Framework for Interactive Music Systems" Journal of the Audio Engineering Society , 2023 Citation Details
Yu, Huiran and Duan, Zhiyao "Note-Level Transcription of Choral Music" , 2024 Citation Details
Yan, Yujia and Duan, Zhiyao "Scoring Time Intervals using Non-Hierarchical Transformer for Automatic Piano Transcription" , 2024 Citation Details
Garcia, Hugo Flores and Benetatos, Christodoulos and O'Reilly, Patrick and Aguilar, Aldo and Duan, Zhiyao and Pardo, Bryan "HARP: Bringing Deep Learning to the DAW with Hosted, Asynchronous, Remote Processing" , 2023 Citation Details

Please report errors in award information by writing to: awardsearch@nsf.gov.

Print this page

Back to Top of page