[2306.01157] Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confounding

Abstract:A prominent challenge of offline reinforcement learning (RL) is the issue of hidden confounding: unobserved variables may influence both the actions taken by the agent and the observed outcomes. Hidden confounding can compromise the validity of any causal conclusion drawn from data and presents a major obstacle to effective offline RL. In the present paper, we tackle the problem of hidden confounding in the nonidentifiable setting. We propose a definition of uncertainty due to hidden confounding bias, termed delphic uncertainty, which uses variation over world models compatible with the observations, and differentiate it from the well-known epistemic and aleatoric uncertainties. We derive a practical method for estimating the three types of uncertainties, and construct a pessimistic offline RL algorithm to account for them. Our method does not assume identifiability of the unobserved confounders, and attempts to reduce the amount of confounding bias. We demonstrate through extensive experiments and ablations the efficacy of our approach on a sepsis management benchmark, as well as on electronic health records. Our results suggest that nonidentifiable hidden confounding bias can be mitigated to improve offline RL solutions in practice.

Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confounding Alizée Pace 1,2,3 Hugo Yèche 2 Bernhard Schölkopf 3 Gunnar Rätsch 2 Guy Tennenholtz 4 arXiv:2306.01157v1 [cs.LG] 1 Jun 2023 1 2 ETH AI Center Department of Computer…

saved by

• Akira Yoshiyama

related reading

• [2506.22401] Exploration from a Primal-Dual Lens: Value-Incentivized Actor-Critic Methods for Sample-Efficient Online RLarxiv.org
• [2602.05999] On the Role of Computation in Reinforcement Learningarxiv.org
• [2606.05555] Representation Learning Enables Scalable Multitask Deep Reinforcement Learningarxiv.org
• [2602.11399] Can We Really Learn One Representation to Optimize All Rewards?arxiv.org
• Value estimation with finite datamcgill.scholaris.ca
• [1709.06560] Deep Reinforcement Learning that Mattersarxiv.org
• [2602.19362] LLMs Can Learn to Reason Via Off-Policy RLarxiv.org
• [2602.15515] The Obfuscation Atlas: Mapping Where Honesty Emerges in RLVR with Deception Probesarxiv.org
• [2103.06326] S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningarxiv.org
• Reinforcement Learning in Newcomblike Problemsproceedings.neurips.cc
• 2109.10813.pdfarxiv.org
• Exploration for the Efficient Deployment of Reinforcement Learning Agentsopenreview.net