Towards Deployment-Efficient Reinforcement Learning: Lower Bound and Optimality | HTML5

ar5iv.labs.arxiv.org · 25,060 words · saved by 1 readers

Deployment efficiency is an important criterion for many real-world applications of reinforcement learning (RL). Despite the community’s increasing interest, there lacks a formal theoretical formulation for the problem. In this paper, we propose such a formulation for deployment-efficient RL (DE-RL) from an “optimization with constraints” perspective: we are interested in exploring an MDP and obtaining a near-optimal policy within minimal deployment complexity, whereas in each deployment the policy can sample a large batch of data. Using finite-horizon linear MDPs as a concrete structural model, we reveal the fundamental limit in achieving deployment efficiency by establishing information-theoretic lower bounds, and provide algorithms that achieve the optimal deployment efficiency. Moreover, our formulation for DE-RL is flexible and can serve as a building block for other practically relevant settings; we give “Safe DE-RL” and “Sample-Efficient DE-RL” as two examples, which may be wort

[2202.06450] Towards Deployment-Efficient Reinforcement Learning: Lower Bound and Optimality Towards Deployment-Efficient Reinforcement Learning: Lower Bound and Optimality Jiawei Huang † † ~{}\dagger , Jinglin Chen † † \dagger , Li Zhao ‡ ‡ \ddagger , Tao Qin ‡ ‡ \ddagger , Nan Jiang † † \dagger , Tie-Yan Liu ‡ ‡ \ddagger † † \dagger Department of Computer Science, University of Illinois at Urbana-Champaign {jiaweih, jinglinc, nanjiang}@illinois.edu ‡ ‡ \ddagger Microsoft Research Asia {lizo, taoqin, tyliu}@microsoft.com Work done during the internship at Microsoft Research Asia. Abstract Dep

Towards Deployment-Efficient Reinforcement Learning: Lower Bound and Optimality | HTML5

related reading