Cyber-physical systems (CPS) such as autonomous vehicles and medical devices must quickly identify and adapt to novel environments. Artificial intelligence (AI) and machine learning (ML) enable CPS to be adaptive and self-improving, capable of non-preprogrammed behaviors. However, the lack of generalization of ML models and the safety-critical nature of CPS requires a trustworthy, resilient, and interpretable learning paradigm that enables CPS to operate in an open world where the environment might be far from the training distribution. We developed a model-centric framework for high-assurance learning-enabled autonomous systems whose behavior automatically adapts via learning and improves its performance and safety to new environments. The functions of an autonomous system can generally be partitioned into those concerned with perception and those concerned with action. Perception builds and maintains an internal model of the world (i.e., the system's environment) that is used to plan and execute actions to accomplish a goal established by human supervisors. Accordingly, the assurance argument for the safety of CPS decomposes into two parts: a) ensuring that the model is an accurate representation of the world as it changes through time and b) ensuring that the actions are safe (and effective), given the model. Both perception and action may employ AI/ML, which presents challenges to assurance. However, it is usually feasible to guard the actions with traditionally engineered and assured monitors, thereby ensuring safety, given the model. Thus, the model becomes the central focus for assurance and safety certification. We developed a new TrinityAI architecture for trustworthy, resilient, and interpretable AI that complements the usual bottom-up learning in CPS, where sensor data is used to make decisions using AI and ML models, with top-down inference wherein a larger context model is used to predict the observations. Minor prediction errors indicate that the world is evolving as expected, and the model is updated accordingly. Significant prediction errors indicate surprise, which may be due to errors in sensing or interpretation of the sensor observation or unexpected changes in the world (e.g., a pedestrian steps into the road). The former initiates error masking or recovery, while the latter requires revision to the model. Higher-level AI functions assist in the diagnosis and execution of these tasks. Although this two-level architecture, where the lower level does "predictive processing" and the upper performs more reflective tasks, both focused on the maintenance of a world model, is derived by engineering considerations, it also matches widely accepted theory of human cognition, namely, "predictive coding" and "dual process theory". The development and implementation of this TrinityAI architecture for safe self-improving CPS led to several technical milestones that were published in top-tier peer-reviewed formal methods, machine learning, and control theory venues such as ICML, IJCAI, NeurIPS, JAR, and ACC. This project involved training and collaboration with several graduate students who interned with the PI at SRI - a non-profit research institute. The developed technology has been made available as open-source software, and public datasets, maintained at https://nusci.csl.sri.com/project/nsf-sicps/ .

Last Modified: 01/02/2023
Modified by: Susmit Jha