flâneur

Value estimation with finite data

mcgill.scholaris.ca · 465 words · saved by 1 readers

This thesis investigates the intersection of reinforcement learning (RL) with function approximation and limited data and aims to develop practical, broadly applicable algorithms. Our contributions center around improving the quality and reliability of approximate value functions in off-policy RL algorithms. Central to the accuracy of value functions is overestimation bias, a key source of optimization error in value estimation. Demonstrating its presence and impact on gradient-based optimization, we propose an algorithmic correction for standard off-policy value learning with substantial performance improvements on benchmark tasks.A critical challenge in applying RL is the dependence on online trial-and-error learning. In principle, this limitation can be circumvented by offline learning, where the RL agent is supplied with data instead of actively collecting it. However, we show that existing off-policy RL algorithms struggle to learn in the offline regime due to errors in value estimation from unmitigated uncertainty. From this insight, we establish the first class of offline deep RL algorithms.Lastly, we study general techniques concerning the optimization and learning of approximate value functions. We develop theoretical foundations for the non-uniform sampling of prior experiences and introduce a representation learning method grounded in the interaction between state and action. These insights contribute to the development of a high-performance RL algorithm for both the online and offline settings

This thesis investigates the intersection of reinforcement learning (RL) with function approximation and limited data and aims to develop practical, broadly applicable algorithms. Our contributions center around improving the quality and reliability of approximate value functions in off-policy RL algorithms. Central to the accuracy of value functions is overestimation bias, a key source of optimization error in value estimation. Demonstrating its presence and impact on gradient-based optimization, we propose an algorithmic correction for standard off-policy value learning with substantial…

saved by

related reading