This project advances and democratizes deep-learning based conversational virtual assistant technology.  We have created new virtual assistant development methodologies and open-source tools that help developers (without being AI experts) create effective conversational agents for their chosen domains for different languages. We have validated our research with powerful privacy-preserving assistants and social chatbots. 

Technical Contributions.

Despite recent dramatic advances in large language models (LLMs), they tend to hallucinate and are hence untrustworthy as virtual assistants. We show that we can eliminate hallucination from LLMs by grounding them with external corpora, which can comprise databases, knowledge bases, free-text documents, APIs on the internet, and theorem provers. 

We introduce a new formal, executable notation that supports inter-operation of these different types of resources. Unlike existing support for APIs in virtual assistants, our notation captures the full signature of the API calls, which includes the results, and not just the input parameters. This is necessary to support compositionality, the key to automating high-level functions.  

We reduce the prohibitively high cost of data collection for training semantic parsers, which translate natural language into code, with a data synthesizer that covers all constructs and properties in schemas. 

To ground LLMs in open-text documents, we create a multi-stage LLM system with few-shot examples that inject factuality into the natural-sounding responses from LLMs. 

We distill large language models for virtual assistants into models small enough to run on a personal device. This not only improves the speed of the chatbot, but also privacy, affordability, and accessibility to many more developers.  

To promote equity across people speaking low-resource languages, we develop methodology and tools to internationalize and localize task-oriented agents to new languages. 

Our contributions to multi-modal assistants and robotic process automation include: We show that a virtual assistant microphone can be used to automatically determine the user position and gaze direction. We propose a novel way to combine touch and verbal commands so users can easily perform cross-application tasks on mobile devices. We allow non-programmers to automate sophisticated web routines by demonstrating them along with natural verbal instructions to incorporate programming concepts such as function composition, conditional, and iterative evaluation. We show how to create a universal call agent that follows natural language instructions to automatically interact with users and operate the web.

To support privacy, we have created a federated virtual assistant architecture so users can use natural language to share with each other privately, with fine-grain access control, with the help of a satisfiability-modulo-theory prover. 

Information Dissemination.

We have published extensively and given numerous keynotes and interviews on our research (https:/oval.cs.stanford.edu). We evangelized open-source assistants through organizing two workshops at Stanford. The first invitation-only workshop, held in 2019, was attended by practitioners and researchers from 30 different organizations. The second, a hybrid online/offline workshop, had over 400 registrants, from 60 educational institutions, 15 nonprofit/government organizations, and 200 companies in many different sectors. All our software is available at https://github.com/stanford-oval.

We have developed a new course, Conversational Virtual Assistant with Deep Learning, which offers students an opportunity to participate in state-of-the-art virtual assistant research. It was attended by a class consisting of 40% female and 10% black/hispanic students last year.

Impact.

1. We have created the first open-source, conversational virtual assistant capable of the top ten most popular skills, including voice control of 15 different kinds of IoT devices. It can run privately on users’ own devices, and has been awarded Popular Science's “Best of What's New Award in Security” in 2019. It was also released as an open-source privacy-preserving voice interface for Home Assistant, a popular, open-source, crowdsourced IoT platform.
2. Our Chirpy Cardinal Socialbot won second place in Alexa Prize Socialbot Grand Competitions,  2020 and 2021. It is currently featured as part of AI: More than Human, an art exhibition about the relationship between humans and AI which has been seen by over 260,000 visitors in Barcelona, Guangzhou, and Liverpool. 
   
   
3. In collaboration with the School of Medicine, we have created a coach for autistic individuals that improves their social skills through drills on conversations. 
   
   
4. In collaboration with an international team, we created a new multilingual few-shot benchmark, X-RiSAWOZ, that covers languages spoken by 3.5B people (China, English, French, Hindi, Hindi-English, Korean) using a tool we created to facilitate data curation.   Our few-shot agents achieve between 61-85% accuracy on dialogue state tracking across all the different languages.
    
5. We have developed the first non-hallucinating LLM-based open-domain chatbot that is informational and conversational, and grounded on Wikipedia and Wikidata.  This unlocks the power of LLMs as a knowledge acquisition tool. 
   
6. Our open-source Genie virtual assistant generation framework is uniquely designed to eliminate hallucination, handle large data schemas along with free-text retrieval, and support multilinguality.
   
   
7. Our multimodal virtual assistant techniques have been commercialized by at least two startups. 

Last Modified: 08/07/2023
Modified by: Monica S Lam