Convergence Between Model - and Data – driven Design for Cyber-Physical Systems

Bio: Dr. Janos Sztipanovits is currently the E. Bronson Ingram Distinguished Professor of Engineering at Vanderbilt University and John Von Neumann professor of the Budapest University of Technology and Economy. He served as founding director of the Institute for Software Integrated Systems between 1998 and 2022, and currently he is member of the Executive Council. Between 1999 and 2002, he worked as program manager and deputy director of DARPA Information Technology Office. He was member of the US Air Force Science Advisory Board between 2006 and 2010 and the Board on Army RDT&E, Systems Acquisition, and Logistics (BARSL) between 2019 and 2021. He co-authored two books and over 350 papers in model-based design, model-integrated computing, design automation for cyber-physical systems, security and autonomous systems. He is Fellow of the IEEE and external member of the Hungarian Academy of Sciences.

Abstract: Cyber-Physical Systems (CPS) give rise to a heterogeneous but tightly coupled engineering design domain. CPS design requires engineering processes that span multiple design disciplines, complex design flows and extensive tool suites. One of the challenges of model-based design automation of CPS is that design trade-offs across traditionally isolated design domains require the deep integration of models, design flows and tool chains. The first part of the talk covers the evaluation of model-based methods gained along the implementation of an experimental design automation tool suite, OpenMETA developed for DARPA’s Adaptive vehicle Make (AVM) program. Experience with OpenMETA showed fundamental benefits as well as practical limitations of model-based design. Cost of developing component models and reusable component model libraries, the semantic complexity of compositional design of systems using heterogeneous components and scalability concerns of design space exploration represent challenges that slow down progress. Recent advances in data-driven methods that has been inspired by the successes of machine learning and AI applications offer partial answer to these challenges while introducing others. The second part of the talk presents recent results in introducing Learning-Enabled Components (LECs) in CPS design and new methods in design space exploration using surrogate models. Progress in the assurance of CPS designs incorporating LECs and developing surrogate models that merge symbolic and data-driven elements show that the convergence of model- and data-driven design is a promising direction that has the potential of accelerating industrial impact. The talk will conclude with the impact of this convergence on tool suites supporting the design of mission and safety critical systems CPS.

Autonomous and Cost-effective Defect Detection System for Molded Pulp Products

• Haochen Wang, Shandong University
• Zhiwei Shi, Zhejiang DeepCode Robotics Co. Ltd.
• Yafei Qiao, Shandong University
• Fan Yang, Zhejiang DeepCode Robotics Co. Ltd.
• Yuzhe He, Zhejiang DeepCode Robotics Co. Ltd.
• Dong Xuan, Shandong University
• Wei Zhao, Shenzhen Institutes of Advanced Technology

Molded pulp products, such as dinnerware, containers, packaging boxes, etc., have gained increasing popularity due to their eco-friendly features. One critical step in their production process is detecting their defects. In this paper, we present an autonomous defect detection system for such products. In the system design, we face four challenges: first, molded pulp products come in various forms and sizes; second, defects are typically small and appear in different forms; third, detection must be fast enough to achieve desired high production rates; fourth, low cost is a key consideration in the pulp molding industry. To overcome these challenges, we have designed a defect detection system with an enhanced YOLOV5s + DeepLabV3Plus backbone and specific modules. Particularly, we design a lightweight YOLOV5s network with an attention mechanism to improve YOLOV5s’ accuracy and speed, for roughly detecting and identifying the type and position of defects. We then utilize the DeepLabV3Plus segmentation model for precise detection. We deploy multiple cameras to handle the products of different sizes and forms, and design a spatial-information based method to eliminate the duplication in detection by different cameras. We have implemented our detection system in a real-world pulp molding manufacturing line, using cost-effective hardware. We have conducted extensive evaluation on our system, demonstrating that our system can meet molded pulp production requirements.

Digital-Twin-Based Patient Evaluation during Stroke Rehabilitation

• Yilun Chen, ShanghaiTech University, China
• Wentao Wang, ShanghaiTech University, China
• Junyu Diao, ShanghaiTech University, China
• Daoyu Wang, ZD Medtech
• Zhuo Jian, ZD Medtech
• Yixi Wang, ZD Medtech
• Zhihao Jiang, ShanghaiTech University and Shanghai Engineering Research Center of Intelligent Vision and Imaging, China

Individuals who experience motor impairment after stroke are able to partially restore motor control through rehabilitation, which achieves long-term recovery through repeated short-term adaptation. The customization of rehabilitation tasks is crucial for enhancing the effectiveness of rehabilitation by promoting the patient's awareness of motor impairments and reducing compensatory behaviors, which is currently dependent on the expertise of physiotherapists. The development of rehabilitation robots aims to alleviate the workload of physiotherapists and has the potential to offer accurate assessments of both short-term adaptation and long-term recovery in stroke patients. In this paper, we propose a framework for automated patient evaluation and task planning during robotic rehabilitation. A motor control model was proposed to capture the patient's motor control process. By adjusting its state and parameters, a digital twin of the patient can be generated and updated, providing insight into the level of adaptation and rehabilitation progress. The digital twin is then utilized to plan customized rehabilitation tasks, which can effectively reduce uncertainty and ambiguities during patient evaluation, and improves patient's adaptation during rehabilitation. The digital twin framework and the task planning algorithms were validated using human subject and simulation experiments.

BubCam: A Vision System for Automated Quality Inspection at Manufacturing Lines

• Jiale Chen, Nanyang Technological University
• Duc Van Le, Nanyang Technological University
• Rui Tan, Nanyang Technological University
• Daren Ho, HP Inc.

Visual sensing has been widely adopted for quality inspection in production processes. This paper presents the design and implementation of a smart collaborative camera system, called BubCam, for automated quality inspection of manufactured ink bags in Hewlett-Packard (HP) Inc.'s factories. Specifically, BubCam estimates the volume of air bubbles in an ink bag, which may affect the printing quality. The design of BubCam faces challenges due to the dynamic ambient light reflection, motion blur effect, and data labeling difficulty. As a starting point, we design a single-camera system which leverages various deep learning based image segmentation and depth fusion techniques. New data labeling and training approaches are proposed to utilize prior knowledge of the production system for training the segmentation model with a small dataset. Then, we design a multi-camera system which additionally deploys multiple wireless cameras to achieve better accuracy via multi-view sensing. To save power of the wireless cameras, we formulate a configuration adaptation problem and develop a deep reinforcement learning (DRL)-based solution to adjust each wireless camera's operation mode and frame rate in response to the changes of presence of air bubbles and light reflection. Extensive evaluation on a lab testbed and real factory trial shows that BubCam outperforms six baseline solutions including the current manual inspection and existing bubble detection and camera configuration adaptation approaches. In particular, BubCam achieves 1.34x accuracy improvement and 260x latency reduction, compared with the manual inspection approach.

Non-invasive bladder volume sensing via bio-impedance spectroscopy: feasibility demonstration in ex-vivo bladder models

• Ata Vafi, University of California, Davis
• Kourosh Vali, University of California, Davis
• Begum Kasap, University of California, Davis
• Jonathan C Hu, University of California, Davis
• Eric Kurzrock, University of California, Davis
• Soheil Ghiasi, University of California at Davis

A bladder volume sensing method based on Bio-Impedance Spectroscopy (BIS) is presented in this paper. The 10 kHz to 0.5 MHz BIS is performed using a Vector Network Analyzer (VNA) on an ex-vivo porcine bladder. The bio-impedance response of the bladder is measured for a saline solution from 0 to 600 ml in increments of 100 ml. The measured data was further post-processed to establish a correlation between the change in bio-impedance data and the amount of change in bladder volume. The measurement was validated across five different bladders with three iterations per bladder for further assessment of data reliability. All experiments showed a decreasing pattern in bio-impedance magnitude with respect to the increase in the bladder volume, which indicates an inverse relationship between the bio-impedance magnitude and the bladder volume. In this regard, the Impedance Change Ratio (ICR) is proposed as a metric to quantitatively characterize the change in the measured impedance associated with the change in the bladder volume. The ICR showed the impedance decrease pattern for the volume increase.

DOME: Drone-assisted Monitoring of Emergent Events For Wildland Fire Resilience

• Fangqi Liu, University of California Irvine
• Janine Ann Baijnath-Rodino, University of California Irvine
• Tung-Chun Chang, University of California Irvine
• Tirtha Banerjee, University of California Irvine
• Nalini Venkatasubramanian, University of California Irvine

We develop a Drone-assisted Monitoring system, DOME, that gathers real-time data for situational awareness in emergent and evolving events. The driving use case for this work is a prescribed burn event (Rx fire), often used to reduce hazardous fuels in forests. DOME coordinates the use of multiple heterogeneous drone platforms to support the observation of emergent physical phenomena (e.g., fire spread) by leveraging domain expert input and physics-based modeling/simulation methods. We propose an executable rule-based system for drone task generation; here, a high-level mission specification utilizes physics-based models for fire spread prediction and automatically generates monitoring instructions with locations, periods, and frequency for drones. DOME integrates algorithms for task allocation (mapping tasks to drones) and flight path planning while considering trade-offs between sensing coverage and accuracy. In addition, DOME will guide in-flight drones to store and upload data under challenged communication settings (out of transmission range, external signal blocking by trees). We evaluate the performance of DOME in real events (with expert-developed burn plans). We test the applicability of the DOME system using simulated Rx burns at the Blodgett Forest Research Station and evaluate our proposed algorithms by comparing their performance with multiple baseline algorithms. Our experiments show the effectiveness of the composite mechanisms in DOME that outperforms other approaches with higher rewards (capturing data of higher quality) and coverage (reduction of missed tasks).

Offline Learning of Closed-Loop Deep Brain Stimulation Controllers for Parkinson Disease Treatment

• Qitong Gao, Duke University
• Stephen L. Schmidt, Duke University
• Afsana Chowdhury, Duke University
• Guangyu Feng, Duke University
• Jennifer J. Peters, Duke University
• Katherine Genty, Duke University
• Warren M. Grill, Duke University
• Dennis A. Turner, Duke University
• Miroslav Pajic, Duke University

Deep brain stimulation (DBS) has shown great promise toward treating motor symptoms caused by Parkinson’s disease (PD), by delivering electrical pulses to the Basal Ganglia (BG) region of the brain. However, DBS devices approved by the U.S. Food and Drug Administration (FDA) can only deliver continuous DBS (cDBS) stimuli at a fixed amplitude; this energy inefficient operation reduces battery lifetime of the device, cannot adapt treatment dynamically for activity, and may cause significant side-effects (e.g., gait impairment). In this work, we introduce an offline reinforcement learning (RL) framework, allowing the use of past clinical data to train an RL policy to adjust the stimulation amplitude in real time, with the goal of reducing energy use while maintaining the same level of treatment (i.e., control) efficacy as cDBS. Moreover, clinical protocols require the safety and performance of such RL controllers to be demonstrated ahead of deployments in patients. Thus, we also introduce an offline policy evaluation (OPE) method to estimate the performance of RL policies using historical data, before deploying them on patients. We evaluated our framework on four PD patients equipped with the RC+S DBS system, employing the RL controllers during monthly clinical visits, with the overall control efficacy evaluated by severity of symptoms (i.e., bradykinesia and tremor), changes in PD biomakers (i.e., local field potentials), and patient ratings. The results from clinical experiments show that our RL-based controller maintains the same level of control efficacy as cDBS, but with significantly reduced stimulation energy. Further, the OPE method is shown effective in accurately estimating and ranking the expected returns of RL controllers.

Towards Ambient Intelligence for Healthcare: A CPS Perspective

Bio: Professor John A. Stankovic is the BP America Professor in the Computer Science Department at the University of Virginia and Director of the (CPS) Link Lab. He is a Fellow of both the IEEE and the ACM. He has been awarded an Honorary Doctorate from the University of York for his work on real-time systems. In 2022, he was elected to the Virginia Academy of Science, Engineering, and Medicine. He won the IEEE Real-Time Systems Technical Committee's Award for Outstanding Technical Contributions and Leadership. He also received the IEEE Technical Committee on Distributed Processing's Distinguished Achievement Award (inaugural winner), and the IEEE TC on CPS’s Technical Achievement Award. He has two test-of-time paper awards. Stankovic has an h-index of 122 and over 65,000 citations. Prof. Stankovic received his PhD from Brown University.

Abstract: Ambient Intelligence has been a goal for more than 20 years. Are we getting close? What if we focus ambient intelligence on smart healthcare, are we getting close? What role does CPS play in ambient intelligence? This talk is motivated by these questions. Various challenges, research directions, and research results from my group’s work will be used to (partially) address these themes for smart healthcare. The talk includes discussions of the vision, the role of CPS, cognitive assistants on wearables, solutions supporting mental health, and lessons learned from real deployments. There is also a brief discussion on two key challenges: the need for robust models and dealing with uncertainties due to the environment and human behaviors.

Learning Spatio-Temporal Aggregations for Large-Scale Capacity Expansion Problems

• Aron Brenner, Massachusetts Institute of Technology
• Rahman Khorramfar, Massachusetts Institute of Technology
• Saurabh Amin, Massachusetts Institute of Technology

Effective investment planning decisions are crucial to ensure that critical cyber-physical infrastructures satisfy performance requirements over an extended time horizon. Computing these decisions often requires solving Capacity Expansion Problems (CEPs). In the context of regional-scale energy systems, these problems are prohibitively expensive to solve owing to large network sizes, heterogeneous node characteristics (e.g., electric power and natural gas energy vectors), and a large number of operational periods. To maintain tractability, traditional approaches resort to aggregating network nodes and/or selecting a set of representative time periods. Often, these reductions do not capture the supply-demand variations that crucially impact the CEP costs and constraints, leading to suboptimal decisions. Here, we propose a novel graph convolutional autoencoder approach for spatiotemporal aggregation of a generic CEP with heterogeneous nodes (CEPHN). Our autoencoder architecture leverages graph pooling to identify nodes with similar characteristics and minimizes a multi-objective loss function. This loss function is specifically tailored to induce desirable spatial and temporal aggregations in terms of tractability and optimality of CEPHN. In particular, the output of the graph pooling provides a spatial aggregation while clustering the low-dimensional encoded representations yields a temporal aggregation. We apply our approach to generation expansion planning of coupled power and natural gas system in New England. The resulting spatiotemporal aggregation leads to a simpler CEPHN with 6 nodes (as opposed to 88 nodes in the original system) and a small set of representative days selected from a full year. We evaluate aggregation outcomes over a range of hyperparameters governing the loss function, and compare resulting upper bounds on the original problem with those obtained using previously known methods. The results from our case study show that this approach provides solutions that are 33% (resp. 10%) better those than obtained from standard spatial (resp. temporal) aggregation approaches.

FedAR+: A Federated Learning Approach to Appliance Recognition with Mislabeled Data in Residential Environments

• Ashish Gupta, Missouri University of Science and Technology, Rolla, USA
• Hari Prabhat Gupta, Indian Institute of Technology (BHU
• Sajal K. Das, Missouri University of Science and Technology

With the enhancement of people's living standards and the rapid evolution of cyber-physical systems, residential environments are becoming smart and well-connected, causing a significant raise in overall energy consumption. As household appliances are major energy consumers, their accurate recognition becomes crucial to avoid unattended usage and minimize peak-time load on the smart grids, thereby conserving energy and making smart environments more sustainable. Traditionally, an appliance recognition model is trained at a central server (service provider) by collecting electricity consumption data via smart plugs from the clients (consumers), causing a privacy breach. Besides that, the data are susceptible to noisy labels that may appear when an appliance gets connected to a non-designated smart plug. While addressing these issues jointly, we propose a novel federated learning approach to appliance recognition, called FedAR+, enabling decentralized model training across clients in a privacy-preserving way even with mislabeled training data. FedAR+ introduces an adaptive noise handling method, essentially a joint loss function incorporating weights and label distribution, to empower the appliance recognition model against noisy labels. By deploying smart plugs in an apartment complex, we collect a labeled dataset that, along with two existing datasets, are utilized to evaluate the performance of FedAR+. Experimental results show that our approach can effectively handle up to $30\%$ concentration of noisy labels while outperforming the prior solutions by a large margin on accuracy.

Pishgu: Universal Path Prediction Network Architecture for Real-time Cyber-physical Edge Systems

• Ghazal Alinezhad Noghre, University of North Carolina at Charlotte
• Vinit Katariya, University of North Carolina Charlotte
• Armin Danesh Pazho, University of North Carolina at Charlotte
• Christopher Neff, University of North Carolina at Charlotte
• Hamed Tabkhi, University of North Carolina Charlotte

Path prediction is an essential task for many real-world Cyber-Physical Systems (CPS) applications, from autonomous driving and traffic monitoring/management to pedestrian/worker safety. These real-world CPS applications need a robust, lightweight path prediction that can provide a universal network architecture for multiple subjects (e.g., pedestrians and vehicles) from different perspectives. However, most existing algorithms are tailor-made for a unique subject with a specific camera perspective and scenario. This article presents Pishgu, a universal lightweight network architecture, as a robust and holistic solution for path prediction. Pishgu's architecture can adapt to multiple path prediction domains with different subjects (vehicles, pedestrians), perspectives (bird's-eye, high-angle), and scenes (sidewalk, highway). Our proposed architecture captures the inter-dependencies within the subjects in each frame by taking advantage of Graph Isomorphism Networks and the attention module. We separately train and evaluate the efficacy of our architecture on three different CPS domains across multiple perspectives (vehicle bird's-eye view, pedestrian bird's-eye view, and human high-angle view). Pishgu outperforms state-of-the-art solutions in the vehicle bird's-eye view domain by 42% and 61% and pedestrian high-angle view domain by 23% and 22% in terms of ADE and FDE, respectively. Additionally, we analyze the domain-specific details for various datasets to understand their effect on path prediction and model interpretation. Finally, we report the latency and throughput for all three domains on multiple embedded platforms showcasing the robustness and adaptability of Pishgu for real-world integration into CPS applications.

A Neurosymbolic Approach to the Verification of Temporal Logic Properties of Learning-enabled Control Systems

• Navid Hashemi, University of Southern California
• Bardh Hoxha, Toyota Research Institute of North America
• Tomoya Yamaguchi, Toyota Research Institute of North America
• Danil Prokhorov, Toyota Research Institute of North America
• Georgios Fainekos, Toyota Research Institute of North America
• Jyotirmoy Deshmukh, University of Southern California

Signal Temporal Logic (STL) has become a popular tool for expressing formal requirements of Cyber-Physical Systems (CPS). The problem of verifying STL properties of neural network-controlled CPS remains a largely unexplored problem. In this paper, we present a model for the verification of Neural Network (NN) controllers for general STL specifications using a custom neural architecture where we map an STL formula into a feed-forward neural network with ReLU activation. In the case where both our plant model and the controller are ReLU-activated neural networks, we reduce the STL verification problem to reachability in ReLU neural networks. We also propose a new approach for neural network controllers with general activation functions; this approach is a sound and complete verification approach based on computing the Lipschitz constant of the closed-loop control system. We demonstrate the practical efficacy of our techniques on a number of examples of learning-enabled control systems.

Self-Preserving Genetic Algorithms for Safe Learning in Discrete Action Spaces

• Preston K. Robinette, Vanderbilt University
• Nathaniel Hamilton, Parallax Advanced Research
• Taylor T. Johnson, Vanderbilt University

Self-Preserving Genetic Algorithms (SPGA) combine the evolutionary strategy of a genetic algorithm with safety assurance methods commonly implemented in safe reinforcement learning (SRL), a branch of reinforcement learning (RL) that accounts for safety in the exploration and decision-making process of the agent. Safe learning approaches are especially important in safety-critical environments, where failure to account for the safety of the controlled system could result in the loss of millions of dollars in hardware or bodily harm to people working nearby, as is true of many cyber-physical systems. While SRL is a viable approach to safe learning, there are many challenges that must be taken into consideration when training agents, such as sample efficiency, stability, and exploration---an issue that is easily addressed by the evolutionary strategy of a genetic algorithm. By combining GAs with the safety mechanisms used with SRL, SPGA offers a safe learning alternative that is able to explore large areas of the solution space, addressing SRL's challenge of exploration. This work implements SPGA with both action masking and run time assurance safety strategies to evolve safe controllers for three types of discrete action space environments applicable to cyber physical systems (control, routing, and operations) and under various safety conditions. Training and testing evaluation metrics are compared with results from SRL trained controllers to validate results. SPGA and SRL controllers are trained across 5 random seeds and evaluated on 500 episodes to calculate average wall time to train, average expected return, and percentage of safe action evaluation metrics. SPGA achieves comparable reward and safety performance results with significantly improved training efficiency (55x faster on average), demonstrating the effectiveness of this safe learning approach.

CODiT: Conformal Out-of-Distribution Detection in Time-Series Data for Cyber-Physical Systems

• Ramneet Kaur, University of Pennsylvania
• Kaustubh Sridhar, University of Pennsylvania
• Sangdon Park, Georgia Institute of Technology
• Yahan Yang, University of Pennsylvania
• Susmit Jha, SRI International
• Anirban Roy, SRI International
• Oleg Sokolsky, University of Pennsylvania
• Insup Lee, University of Pennsylvania

Uncertainty in the predictions of learning enabled components hinders their deployment in safety-critical cyber-physical systems (CPS). A shift from the training distribution of a learning enabled component (LEC) is one source of uncertainty in the LEC’s predictions. Detection of this shift or out-of-distribution (OOD) detection on individual datapoints has therefore gained attention recently. But in many applications, inputs to CPS form a temporal sequence. Existing techniques for OOD detection in time-series data for CPS either do not exploit temporal relationships in the sequence or do not provide any guarantees on detection. We propose using deviation from the in-distribution temporal equivariance as the non-conformity measure in conformal anomaly detection framework for OOD detection in time-series data for CPS. Computing independent predictions from multiple conformal detectors based on the proposed measure and combining these predictions by Fisher’s method leads to the proposed detector CODiT with bounded false alarms. We illustrate the efficacy of CODiT by achieving state-of-the-art results in autonomous driving systems with perception (or vision) LEC. We also perform experiments on medical CPS for GAIT analysis where physiological (non-vision) data is collected with force-sensitive resistors attached to the subject’s body. Code, data, and trained models are available at https://github.com/kaustubhsridhar/time-series-OOD

David Corman, Program Director

Bio: Dr. David Corman is the Program Director leading Cyber Physical Systems (CPS), Smart and Connected Communities (S&CC), and CIVIC Innovation Challenge Programs for the National Science Foundation. The CPS program is a cross-disciplinary and inter-agency program and seeks to reveal cross-cutting, fundamental scientific and engineering principles that underpin the integration of cyber and physical elements across all application domains including autonomous systems, manufacturing, energy, civil and mechanical engineering, and agriculture. The Smart and Connected Communities(S&CC) program was started by NSF in 2016. The goal of this program is to support strongly interdisciplinary, integrative research and research capacitybuilding activities that will improve understanding of smart and connected communities and lead to discoveries that enable sustainable change to enhance community functioning. The focus here is not simply on cities – but on cities, towns, and rural regions. Whereas S&CC looks to develop foundational research, CIVIC looks to accelerate the transition of the research through community partnership and impactful pilots. Dr. Corman joined NSF 2013. He previously worked for McDonnell Douglas / Boeing in a variety of research positions. Dr. Corman was chief scientist in the Network Systems Technology for Boeing Research and Technology during the period from 2007 – 2013. His responsibilities also included development and leadership of research projects in cybersecurity for airplane and avionics systems. Dr. Corman obtained a dual BS degree in System Science and Mathematics and Applied Mathematics and Computer Science from Washington University in St. Louis. He then obtained a dual MS degree in SSM and Mechanical Engineering from Washington University. He completed his graduate education at the University of Maryland – College Park and obtained a PhD in Electrical Engineering with a major in controls and minor in communications and applied mechanics.

Linda Bushnell, Program Director

Bio: Linda Bushnell is a Research Professor in ECE at the University of Washington. She received her Ph.D. in EECS from UC Berkeley in 1994, her M.A. in Mathematics from UC Berkeley in 1989, her M.S. in EE from UConn in 1987, and her B.S. in EE from UConn in 1985. She also received her MBA from the UW Foster School of Business in 2010. Her research interests include networked control systems and cyber-physical systems. She is a Fellow of the IEEE for contributions to networked control systems. She is a Fellow of IFAC for contributions to the analysis and design of networked control systems. She is a recipient of the US Army Superior Civilian Service Award, NSF ADVANCE Fellowship, and IEEE Control Systems Society Distinguished Member Award. She has been a member of the IEEE since 1985, a member of the IEEE CSS since 1990, and a member of the IEEE Women in Engineering since 2013. She is Treasurer of the American Automatic Control Council (AACC) and Member of the Technical Board for the International Federation on Automatic Control (IFAC).

John E. Taylor, Program Director

Bio: John E. Taylor is the Frederick Law Olmsted Professor. Taylor studies the dynamics where human and engineered networks meet, making him an ideal fit for an endowed professorship named for the father of landscape architecture and a designer who believed engineered infrastructure should be both functional and aesthetically appealing, serving society’s needs while also creating more livable and healthy communities. Taylor has been an entrepreneur and worked as a project manager before starting his career in higher education. He taught most recently at Virginia Tech, where he was a dean’s faculty fellow in the College of Engineering and a Preston and Catharine White fellow in the College of Architecture and Urban Studies.

Pavithra Prabhakar, Program Director

Bio: Dr. Pavithra Prabhakar is professor in the department of computer science, and holds the Peggy and Gary Edwards Chair in Engineering. She is currently serving the National Science Foundation as a Program Director in the Software and Hardware Foundations Cluster in the Computer and Information Science and Engineering Directorate. She obtained her doctorate in computer science and a master's degree in applied mathematics from the University of Illinois at Urbana-Champaign, followed by a CMI postdoctoral fellowship for a year at the California Institute of Technology. Prior to coming to K-State, she spent four years at the IMDEA Software Institute in Spain as a tenure-track assistant professor. She previously interned at Bell Labs, Murray Hill, while working toward her doctorate. Her main research interest is in formal analysis of intelligent, autonomous, and cyber-physical systems with emphasis on both foundational and practical aspects related to automated and scalable techniques for verification and synthesis of hybrid control systems. Her research borrows ideas from automata theory, control and dynamical systems theory, formal methods and logics. She has lead several project on safety analysis of cyber-physical systems, including novel methods that combine counter-example guided abstraction refinement and hybridization. She has pioneered a novel approach for stability analysis of hybrid control systems based on ideas from formal methods which has appeared in the form of several invited papers and best paper award nominations. She is currently pursuing projects on robust analysis and design of autonomous and cyber-physical systems with artificial intelligence and machine learning components and applications in automotive, aerospace and robotics systems, and agricultural automation.

Efficiently Enabling Rich and Trustworthy Inferences at the Extreme Edge

Bio: Mani Srivastava is on the faculty at UCLA where he is a Distinguished Professor in the ECE Department with a joint appointment in the CS Department and is the Vice Chair for Computer Engineering. His research is broadly in the area of human-cyber-physical and IoT systems that are learning-enabled, resource-constrained, and trustworthy. It spans problems across the entire spectrum of applications, architectures, algorithms, and technologies in the context of systems and applications for mHealth, sustainable buildings, and smart built environments. He is a Fellow of both the ACM and the IEEE.

Abstract: Computing systems intelligently performing perception-cognition-action (PCA) loops are essential to interfacing our digitized society with the analog world it is embedded in. They employ distributed edge-cloud computing hierarchies and deep learning methods to make sophisticated inferences and decisions from high-dimensional unstructured sensory data in our personal, social, and physical spaces. While the adoption of deep learning has resulted in considerable advances in accuracy and richness, they have also resulted in challenges such as generalizing to novel situations, assuring robustness in the face of uncertainty, engendering trust in opaque modes, reasoning about complex spatiotemporal events, and implementing in ultra resource-constrained edge devices. This talk presents ideas for addressing these challenges with physics-aware neuro-symbolic models, automatic platform-aware architecture search, and sharing of edge resources, and describes our experience in applying them in varied application domains such as mobile health, agricultural robotics, etc.

Joint Differentiable Optimization and Verification for Certified Reinforcement Learning

• Yixuan Wang, Northwestern University
• Simon Zhan, UC Berkeley
• Zhilu Wang, Northwestern University
• Chao Huang, University of Liverpool
• Zhaoran Wang, Northwestern University
• Zhuoran Yang, Yale University
• Qi Zhu, Northwestern University

Model-based reinforcement learning has been widely studied for controller synthesis in cyber-physical systems (CPSs). In particular, for safety-critical CPSs, it is important to formally certify system properties (e.g., safety, stability) under the learned RL controller. However, as existing methods typically conduct formal verification \emph{after} the controller has been learned, it is often difficult to obtain any certificate, even after many iterations between learning and verification. To address this challenge, we propose a framework that jointly conducts reinforcement learning and formal verification by formulating and solving a novel bilevel optimization problem, which is end-to-end differentiable by the gradients from the value function and certificates formulated by linear programs and semi-definite programs. In experiments, our framework is compared with a baseline model-based stochastic value gradient (SVG) method and its extension to solve constrained Markov Decision Processes (CMDPs) for safety. The results demonstrate the significant advantages of our framework in finding feasible controllers with certificates, i.e., barrier functions and Lyapunov functions that formally ensure system safety and stability.

Conformal Prediction for STL Runtime Verification

• Lars Lindemann, University of Pennsylvania
• Xin Qin, University of Southern California
• Jyotirmoy V. Deshmukh, University of Southern California
• George J. Pappas, University of Pennsylvania

We are interested in predicting failures of cyber-physical systems during their operation. Particularly, we consider stochastic systems and signal temporal logic specifications, and we want to calculate the probability that the current system trajectory violates the specification. The paper presents two predictive runtime verification algorithms that predict future system states from the current observed system trajectory. As these predictions may not be accurate, we construct prediction regions that quantify prediction uncertainty by using conformal prediction, a statistical tool for uncertainty quantification. Our first algorithm directly constructs a prediction region for the satisfaction measure of the specification so that we can predict specification violations with a desired confidence. The second algorithm constructs prediction regions for future system states first, and uses these to obtain a prediction region for the satisfaction measure. To the best of our knowledge, these are the first formal guarantees for a predictive runtime verification algorithm that applies to widely used trajectory predictors such as RNNs and LSTMs, while being computationally simple and making no assumptions on the underlying distribution. We present numerical experiments of an F-16 aircraft and a self-driving car.

Monitoring Signal Temporal Logic in Distributed Cyber-physical Systems

• Anik Momtaz, Michigan State University
• Houssam Abbas, Oregon State University
• Borzoo Bonakdarpour, Michigan State University

This paper solves the problem of runtime verification for signal temporal logic in distributed cyber-physical systems (CPS). We assume a partially synchronous setting, where a clock synchronization algorithm guarantees a bound on clock drifts among all signals. We introduce a formula progression and a signal retiming technique that allow reasoning about the correctness of formulas among continuous-time and continuous-valued signals that do not share a global view of time. The resulting problem is encoded as an SMT solving problem, and we introduce techniques to solve the SMT encoding efficiently. We also conduct two case studies on monitoring a network of aerial vehicles and a water distribution system.

Design and Deployment of Resilient Control Execution Patterns: A Prediction, Mitigation Approach

• Ipsita Koley, Indian Institute of Technology Kharagpur
• Sunandan Adhikary, Indian Institute of Technology Kharagpur
• Arkaprava Sain, Indian Institute of Technology Kharagpur
• Soumyajit Dey, Indian Institute of Technology Kharagpur

Modern Cyber-Physical Systems (CPSs) are often designed as networked, software-based controller implementations which have been found to be vulnerable to network-level and physical-level attacks. A number of research works have proposed CPS-specific attack detection schemes as well as techniques for attack-resilient controller design. However, such schemes also incur platform-level overheads. In this regard, some recent works have leveraged the use of skips in control execution to enhance the resilience of a CPS against false data injection (FDI) attacks. However, skipping the control executions may degrade the performance of the controller. In this paper, we provide an analytical discussion on when and how skipping a control execution can improve the system’s resilience against FDI attacks while maintaining the control performance requirement. Our proposed method i) synthesizes a library of such optimal control execution patterns offline, and ii) executes one of them in run-time judging the intent of the attacker. To the best of our knowledge, no previous work has provided any quantitative analysis about the trade-off between attack resilience and control performance for such aperiodic control execution. Finally, we evaluate the proposed method on several safety-critical CPS benchmarks.

Dynamic Simplex: Balancing Safety and Performance in Autonomous Cyber Physical Systems 

• Baiting Luo, Vanderbilt University
• Shreyas Ramakrishna, Vanderbilt University
• Ava Pettet, Vanderbilt University
• Christopher Kuhn, TUM
• Gabor Karsai, Vanderbilt University
• Ayan Mukhopadhyay, Vanderbilt University

Learning Enabled Components (LEC) have greatly assisted cyber-physical systems in achieving higher levels of autonomy. However, LEC’s susceptibility to dynamic and uncertain operating conditions is a critical challenge for the safety of these systems. Redundant controller architectures have been widely adopted for safety assurance in such contexts. These architectures augment LEC “performant” controllers that are difficult to verify with “safety” controllers and the decision logic to switch between them. While these architectures ensure safety, we point out two limitations. First, they are trained offline to learn a conservative policy of always selecting a controller that maintains the system's safety, which limits the system’s adaptability to dynamic and non-stationary environments. Second, they do not support reverse switching from the safety controller to the performant controller, even when the threat to safety is no longer present. To address these limitations, we propose a dynamic simplex strategy with an online controller switching logic that allows two-way switching. We consider switching as a sequential decision-making problem and model it as a semi-Markov decision process. We leverage a combination of a myopic selector using surrogate models (for the forward switch) and a non-myopic planner (for the reverse switch) to balance safety and performance. We evaluate this approach using an autonomous vehicle case study in the CARLA simulator using different driving conditions, locations, and component failures. We show that the proposed approach results in fewer collisions and higher performance than state-of-the-art alternatives.

EnergyShield: Provably-Safe Offloading of Neural Network Controllers for Energy Efficiency

• Mohanad Odema, University of California, Irvine
• James Ferlez, University of California, Irvine
• Goli Vaisi, University of California, Irvine
• Yasser Shoukry, University of California, Irvine
• Mohammad Al Faruque, UC Irvine

To mitigate the high energy demand of Neural Network (NN) based Autonomous Driving Systems (ADSs), we consider the problem of offloading NN controllers from the ADS to nearby edge computing infrastructure, but in such a way that formal vehicle safety properties are guaranteed. In particular, we propose the EnergyShield framework, which repurposes a controller “shield” as a low-power runtime safety monitor for the ADS vehicle. Specifically, the shield in EnergyShield provides not only safety interventions but also a formal, state-based quantification of the tolerable edge response time before vehicle safety is compromised. Using EnergyShield, an ADS can then save energy by wirelessly offloading NN computations to edge computers, while still maintaining a formal guarantee of safety until it receives a response (on-vehicle hardware provides a just-in-time fail safe). To validate the benefits of EnergyShield, we implemented and tested it in the Carla simulation environment. Our results show that EnergyShield maintains safe vehicle operation while providing significant energy savings compared to on-vehicle NN evaluation: from 24% to 54% less energy across a range of wireless conditions and edge delays.

sat2pc: Generating Building Roof’s Point Cloud from a Single 2D Satellite Images

• Yoones Rezaei, University of Pittsburgh
• Stephen Lee, University of Pittsburgh

Three-dimensional (3D) urban models have gained interest because of their applications in many use-cases such as urban planning, damage detection, transportation, and virtual reality. However, generating these 3D representations requires LiDAR data, which is usually expensive to collect. Because it is expensive, the lidar data are not frequently updated and are not widely available for many regions in the US. As such, 3D models based on these lidar data are either outdated or limited to those locations where the data is available. In contrast, satellite images are freely available and frequently updated. To take advantage of this availability, we propose sat2pc, a deep learning-based approach that predicts the point cloud of a building roof from a single 2D satellite image. Our technique integrates two different loss functions, namely Chamfer Distance and Earth Mover's Distance loss, resulting in a 3D output that balances the overall structure and detail. We extensively evaluate our model and perform ablation studies on a building roof dataset. Our results show that sat2pc outperforms the existing baselines by at least 18.6%. Moreover, we show that our refinement module improves the overall performance, resulting in fine-grained 3D output. Finally, we show that the predicted point cloud captures more detail and geometric characteristics than other baselines.

TIM: A Novel Quality of Service Metric for Tactile Internet

• Kees Kroep, Delft University of Technology
• Vineet Gokhale, Delft University of Technology
• Ashutosh Simha, Delft University of Technology
• R Venkatesha Prasad, Delft University of Technology
• Vijay S Rao, Cognizant Technologies Limited

Tactile Internet (TI) envisions communicating haptic sensory information and kinesthetic feedback over the network and is expected to transfer human skills remotely. For mission-critical TI applications, the network latency is commonly mandated to be between 1-10 ms, due to the sensitivity of human touch, and the packet delivery ratio to be 99.99999%, failing which can lead to catastrophic outcomes. However, with humans-in-the-loop, their dexterity and adaptability to varying responses to stimuli under different network conditions, measuring the performance of a TI session only with latency and packet losses are insufficient and presents an incorrect representation of the experience of the TI application. To develop an objective measure of the quality of TI sessions, we propose a framework that models TI applications as networked control systems, including humans-in-the-loop. We derive a closed-form expression for measuring the difference between the application performance in ideal and non-ideal network conditions. Based on Weber’s law of Just Noticeable Difference, we provide a metric called TIM to estimate the impact of the network on haptic feedback. We implemented TIM on multiple applications on a TI testbed to show that our approach is feasible and TIM strongly follows real subjective measurements. Further, we propose a channel compensation spring based on TIM, to alleviate the network conditions’ negative effects. We demonstrate the efficacy of the channel compensation spring in improving the user experience. We also present implementation notes for TI application developers.

AVstack: An Open-Source, Reconfigurable Platform for Autonomous Vehicle Development

• R. Spencer Hallyburton, Duke University
• Shucheng Zhang, Duke University
• Miroslav Pajic, Duke University

Pioneers of autonomous vehicles (AVs) promised to revolutionize the driving experience and driving safety. However, milestones in AVs have materialized slower than forecast. Culprits include (1) the lack of verifiability of proposed state-of-the-art AV components, and (2) stagnation of pursuing next-level evaluations, e.g., vehicle-to-infrastructure (V2I) and multi-agent collaboration. In part, progress has been hampered by: the large volume of software in AVs, the multiple disparate conventions, the difficulty of testing across datasets and simulators, and the inflexibility of state-of-the-art AV components. To address these challenges, we present AVstack1,2, an open-source, reconfigurable software platform for AV design, implementation, test, and analysis. AVstack solves the validation problem by enabling first-of-a-kind trade studies on datasets and physics-based simulators. AVstack addresses the stagnation problem as a reconfigurable AV platform built on dozens of open-source AV components in a high-level programming language. We demonstrate the power of AVstack through longitudinal testing across multiple benchmark datasets and V2I-collaboration case studies that explore trade-offs of designing multi-sensor, multi-agent algorithms.