University of Applied Sciences St. Pölten Better Safe Than Sorry : A Systematic Literature Review on Avoiding Negative Side-Effects in Reinforcement Learning Reinforcement Learning (RL) has achieved remarkable success across complex tasks, from mastering games such as Go, chess, and Dota2 to operating in stochastic and partially observed environments. Despite these advances, ensuring the safety and reliability of RL agents remains a fundamental challenge within AI alignment research. This master's thesis focuses on avoiding negative side effects (NSEs): preventing agents from causing unintended harm to their environment while pursuing their assigned objectives. To capture the current state of research, a systematic literature review was conducted following the PRISMA framework, guided by two research questions: (i) What approaches have been taken in previous studies to ensure that RL agents avoid negative side effects while completing their tasks? (ii) What areas within this domain require further exploration? A multi-pronged search strategy combining database searches, targeted workshops, and author-specific queries was employed for publications between 2019 and 2023, complemented by preprints from reputable sources. Retrieved records were independently scored by four reviewers, and included studies were classified along a proposed taxonomy of underlying agent assumptions: fully observable states, availability of a no-op action, known dynamics, linear feature rewards, action-independent rewards, and known past states. The review identifies five principal methodological categories for mitigating NSEs. Reward regularization approaches, such as Occupancy-Regularized Policy Optimization, augment the reward function with penalties for harmful behavior. Impact regularization methods, including relative reachability and attainable utility preservation, penalize deviations from a baseline state to avoid irreversible changes to the environment. Inverse reinforcement learning infers human-aligned reward functions from expert demonstrations or the initial configuration of the environment. Human feedback and model-based penalty prediction approaches, such as minimax-regret querying and the parenting algorithm, incorporate human input or learned environment models to anticipate and avoid unsafe actions. Finally, initial work on safety in multi-agent systems addresses emergent side effects arising from agent interactions. The classification reveals a strong reliance on idealized assumptions, most notably fully observable states, known dynamics, and the availability of a no-op action as an impact baseline. These assumptions rarely hold in real-world settings, where agents face partial observability, noisy sensory data, and unpredictable or evolving dynamics. Furthermore, the analysis highlights a persistent trade-off between task performance and safety: regularization-based methods frequently yield overly conservative agents, while evaluations remain largely confined to simulated gridworld environments such as AI Safety Gridworlds and SafeLife, limiting real-world transferability. Based on these findings, the master's thesis identifies several gaps requiring further exploration: handling partial observability and non-Markovian side effects whose consequences depend on historical trajectories, integrating real-time human feedback into agent decision-making, establishing standardized and generalizable safety benchmarks, providing long-term safety guarantees, extending safety mechanisms to multi-agent settings, and addressing fairness, accountability, and transparency of safety mechanisms. Closing these gaps is essential for the safe deployment of RL agents in complex, real-world scenarios. Reinforcement learning (RL) has made significant progress in solving complex tasks across various domains, yet ensuring the safety and reliability of RL agents remains a critical challenge. This thesis focuses on avoiding negative side effects (NSEs) in RL, a key aspect of AI safety that seeks to prevent agents from causing unintended harm while pursuing their goals. A systematic literature review was conducted to explore existing methodologies aimed at mitigating NSEs, with a particular emphasis on approaches such as reward regularization, impact regularization, inverse reinforcement learning, and model-based penalty prediction. The review reveales a strong reliance on assumptions such as fully observable states and known dynamics, which limit the applicability of current safety mechanisms in real-world, partially observable environments. The thesis also identifies gaps in the literature, including the need for more robust methods to handle non-Markovian side effects, better integration of real-time human feedback, and the development of standardized safety benchmarks. Addressing these gaps is crucial for advancing the safe deployment of RL agents in real-world scenarios, where complexity and unpredictability pose significant challenges. https://phaidra.ustp.at/o:7859 en TEXT thesis Hochschulschrift Tesi Text Hochschulschrift Documento PDF Tesi Text thesis Machine Learning ÖFOS 2012 -- Artificial Intelligence ÖFOS 2012 -- Machine learning ÖFOS 2012 -- Artificial intelligence ÖFOS 2012 -- Reinforcement Learning AI Safety Negative Side Effects Safe Reinforcement Learning Impact Regularization Reward Regularization Inverse Reinforcement Learning Markov Decision Process Reward Specification Safe Exploration AI Alignment Survey Systematic Literature Review 2024 2024-09-10 Fabian Kovac Machine Learning Machine learning Artificial Intelligence Artificial intelligence