ABSTRACT

Online persistent and shared multi-user virtual environments (MUVEs), with thousands or even millions of users, constitute an emerging and rapidly growing field that is likely to dramatically impact higher education in the near future. Direct player-to-player interaction, and the networks that players develop in the virtual world, are central to the unique experience and success of these MUVEs. However, despite their increased visual realism, the immersive "social functionality" in current MUVEs is still rudimentary at best, since real-world conversations and social interactions have not been mimicked and modeled. This is because it is technically challenging to extend existing one-to-one conversation modeling approaches to digitally mediated multi-party conversations and interactions in virtual worlds, due to the significant differences in nonverbal behavior and interaction patterns. The automated generation of digitally mediated multi-party communication and interaction has thus become a major technical barrier that restricts the depth and usefulness of various online virtual worlds and virtual reality applications. In this research, the PI will tackle this issue by designing new algorithms and systems driven by live speech from users in different locations, which can automatically generate synchronized multi-modal conversational gestures on embodied avatars, including head/eye movement, lip movement, hand gesture, and body posture. Project outcomes will facilitate the widespread adoption of useful avatar and tele-immersion technology in applications where computer-mediated communication plays a role, including education, commerce, health and engineering. The PI will make the acquired high-fidelity multi-modal multi-party conversational behavior datasets available to the scientific community at large, so they can be used in future research.

This ambitious project will focus on three inter-related research thrusts that are aligned with the PI's research expertise in computer animation, virtual humans, and human computer interaction. Automated generation of realistic talking avatars based on live speech input alone; the PI will design efficient and automated schemes to generate on-the-fly talking avatars based on live speech input, by fusing established social exchange rules with data-driven statistical modeling. Automated generation of believable listening avatars with immersive social exchanges; based on in-depth statistical analysis of real life multiparty conversation data, the PI will design data-driven schemes for generating tightly coordinated gazes, head movements, and body posture shifts on listening avatars, as well as social gaze exchanges between listening peers. Comparative evaluation of the proposed avatar-mediated multi-party conversation and interaction approach in an in-house built research testbed; the robustness and effectiveness of the proposed framework will be evaluated by integrating it into an in-house built research testbed (i.e., a simplified MUVE prototype).

Please report errors in award information by writing to: awardsearch@nsf.gov.