Model-based Design of Closed-Loop Deep Brain Stimulation Controllers using Reinforcement Learning
Abstract – Parkinson’s disease (PD) currently influences around one million people in the US. Deep brain stimulation(DBS) is one of the most effective treatments that can minimize the effects of PD with different severity levels, by employing low-voltage electrical stimulation to the basal ganglia (BG) region of the brain. Most existing commercial DBS devices employ stimulation based only on fixed-frequency periodic pulses. While such periodic high-frequency DBS controllers provide effective PDtherapy that alleviates PD symptoms, they are very inefficient in terms of energy consumption; thus, severely limiting lifetime of these battery-operated devices. Consequently, there is a need to move beyond (1) fixed stimulation pulse controllers, and(2) ‘one-size-fits-all’ patient-agnostic treatments, in order to provide energy-efficient and effective(in terms of PD treatment)DBS controllers. In this work, to the best of our knowledge, we propose a first deep reinforcement learning (RL)-based approach that can derive patient-specific DBS controllers that are both effective and energy-efficient. Specifically, we model the BG as a Markov decision process (MDP) and define the state and action space as state of the neurons in the BGregions and the stimulation patterns, respectively. Thereafter, we define the reward functions over the state space, and the learning objective is set to maximize the accumulated reward over a finite horizon (i.e., the treatment duration), while bounding average stimulation frequency. We evaluate the performance of our methodology using a Brain-on-Chip(BoC)FPGA platform that implements the physiologically-relevant basal ganglia model (BGM). We show that our RL-basedDBS controllers significantly outperform existing periodical controllers in terms of energy efficiency (e.g., by 70% over commonly used periodic controllers), while providing suitable levels of PD treatment.