This project aimed to generalize and certify the performance of reinforcementlearning algorithms for control of cyber-physical systems (CPS). In particular,outcomes of the project including new theory, metrics, and methods for certifying algorithms that compute optimal control policies in systems with non-classical dynamics and non-classical costs.  Broadly speaking, reinforcement learning applied to physical systems is concerned with making predictions from data to control the system to extremize a performance criterion. In this context, this project  particularly focused on developing theory and algorithms applicable to hybrid and multi-agent control systems, that is, systems with continuous and discrete elements and systems with multiple decision-making agents, which are ubiquitous in CPS acrosss patio-temporal scales and application domains. While arguably reinforcement learning algorithms are not yet mature enough to guarantee performance when applied to control of complex CPS with learning enable components, this project made great strides in laying the theoretical and computational foundation to certify reinforcement learning algorithms so that they may be deployed in society with high confidence.

 To achieve this goal, the research team generalized convergent algorithms originally designed for purely continuous systems to apply in hybrid controlsystems whose states undergo a mixture of discrete and continuous transitions. Computationally efficient methods for ensuring additional desiderata such as saftey were developed. Moreover, the team developed techniques that targeted  societal-scale CPS inwhich multiple agents, some of which may be humans, interact directly with the CPS. The developed algorithms and theoretical models were experimentally validated on three testbeds that represent a range of hybrid and multi-agent phenomena arising in CPS. The first testbed focuses on the performance of ouralgorithms on societal-scale traffic flow networks via simulation. The second testbed concerns heterogeneous teams of aerial and terrestrial mobile robots collaborating with human partners to perform construction, inspection, and maintenance tasks on scale facsimiles of infrastructure like bridges, andtunnels. The third testbed comprises both simulation and real-world experimental studies focused on the closed-loop interaction between individual humans and learning based systems. The latter test bed is motivated by CPS suchas teleoperated robots that perform dynamic locomotion and manipulation behaviors, human-in-the-loop societal scale CPS such as intelligent transportation, and multi-agent reinforcement learning broadly. Overall the project resulted in a wide range of results from fundamental new theory on non-smooth dynamical systems and multi-agent interactions to practically implementable algorithms to experimental validation in a variety of CPS context. 

The research team included not just the PIs and graduate students, but a numberof undergraduate researchers funded through REUs who contributed significantlyto the project, especially the simulation-based test beds which are now publiclyavailable for the community to use. Finally, the team disseminated the results to a range of different communities including industry, policy makers, governing bodies, and academia. 

Last Modified: 01/14/2023
Modified by: Lillian Ratliff