On Value Functions and the Agent-Environment Boundary | HTML5

When function approximation is deployed in reinforcement learning (RL), the same problem may be formulated in different ways, often by treating a pre-processing step as a part of the environment or as part of the agent. As a consequence, fundamental concepts in RL, such as (optimal) value functions, are not uniquely defined as they depend on where we draw this agent-environment boundary. This causes further problems in theoretical analyses that provide optimality guarantees, as the same analysis may yield different bounds in equivalent formulations of the same problem. We address this issue via a simple and novel boundary-invariant analysis of Fitted Q-Iteration, a representative RL algorithm, where the assumptions and the guarantees are invariant to the choice of boundary. We also discuss closely related issues on state resetting, deterministic vs stochastic systems, imitation learning, and the verifiability of theoretical assumptions from data. A large part of RL theory—including tha

On Value Functions and the Agent-Environment Boundary Nan Jiang Abstract When function approximation is deployed in reinforcement learning (RL), the same problem may be formulated in different ways, often by treating a pre-processing step as a part of the environment or as part of the agent. As a consequence, fundamental concepts in RL, such as (optimal) value functions, are not uniquely defined as they depend on where we draw this agent-environment boundary . This causes further problems in theoretical analyses that provide optimality guarantees, as the same analysis may yield different bound

related reading

• A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
• Reinforcement Learning in Newcomblike Problemsproceedings.neurips.cc
• c21f4ce780c5c9d774f79841b81fdc6d-Paper.pdfproceedings.neurips.cc
• [2211.05311] When is Realizability Sufficient for Off-Policy Reinforcement Learning?ar5iv.labs.arxiv.org
• Value estimation with finite datamcgill.scholaris.ca
• [2506.22401] Exploration from a Primal-Dual Lens: Value-Incentivized Actor-Critic Methods for Sample-Efficient Online RLarxiv.org
• 2203.00543arxiv.org
• [2602.11399] Can We Really Learn One Representation to Optimize All Rewards?arxiv.org
• rltheorybook_ABJKS.pdfrltheorybook.github.io
• [1710.10044] Distributional Reinforcement Learning with Quantile Regressionarxiv.org
• [2106.10316] Proper Value Equivalencearxiv.org
• [1709.06560] Deep Reinforcement Learning that Mattersarxiv.org