{"@context":{"aiiso":"http:\/\/purl.org\/vocab\/aiiso\/schema#","arm":"https:\/\/ld4p.github.io\/arm\/core\/ontology\/0.1\/","bf":"http:\/\/id.loc.gov\/ontologies\/bibframe\/","bib":"https:\/\/bibliotek-o.org\/","bibo":"http:\/\/purl.org\/ontology\/bibo\/","cito":"http:\/\/purl.org\/spar\/cito\/","classSchemes":"http:\/\/id.loc.gov\/vocabulary\/classSchemes","dbo":"http:\/\/dbpedia.org\/ontology\/","dce":"http:\/\/purl.org\/dc\/elements\/1.1\/","dcmitype":"http:\/\/dublincore.org\/documents\/2000\/07\/11\/dcmi-type-vocabulary\/#","dcterms":"http:\/\/purl.org\/dc\/terms\/","ebucore":"http:\/\/www.ebu.ch\/metadata\/ontologies\/ebucore\/ebucore#","edm":"http:\/\/www.europeana.eu\/schemas\/edm\/","foaf":"http:\/\/xmlns.com\/foaf\/0.1\/","frap":"http:\/\/purl.org\/cerif\/frapo\/","identifiers":"http:\/\/id.loc.gov\/vocabulary\/identifiers\/","ids":"http:\/\/id.loc.gov\/vocabulary\/identifiers\/","oaire":"http:\/\/namespace.openaire.eu\/schema\/oaire\/","oefos2012":"https:\/\/vocabs.acdh.oeaw.ac.at\/oefosdisciplines\/","opaque":"http:\/\/opaquenamespace.org\/","pcdm":"http:\/\/pcdm.org\/models#","phaidra":"https:\/\/phaidra.org\/ontology\/","rdaa":"http:\/\/rdaregistry.info\/Elements\/a\/","rdam":"http:\/\/rdaregistry.info\/Elements\/m\/","rdau":"http:\/\/rdaregistry.info\/Elements\/u\/","rdax":"http:\/\/rdaregistry.info\/Elements\/x\/","rdf":"http:\/\/www.w3.org\/1999\/02\/22-rdf-syntax-ns#","rdfs":"http:\/\/www.w3.org\/2000\/01\/rdf-schema#","relators":"http:\/\/id.loc.gov\/vocabulary\/relators\/","role":"http:\/\/id.loc.gov\/vocabulary\/relators\/","role:res":{"@container":"@list","@id":"http:\/\/id.loc.gov\/vocabulary\/relators\/res"},"schema":"http:\/\/schema.org\/","skos":"http:\/\/www.w3.org\/2004\/02\/skos\/core#","skosxl":"http:\/\/www.w3.org\/2008\/05\/skos-xl#","vra":"http:\/\/purl.org\/vra\/"},"@id":"https:\/\/phaidra.ustp.at\/o:7859","bf:note":[{"@type":"bf:Note","skos:prefLabel":[{"@language":"eng","@value":"Reinforcement Learning (RL) has achieved remarkable success across complex tasks, from mastering games such as Go, chess, and Dota2 to operating in stochastic and partially observed environments. Despite these advances, ensuring the safety and reliability of RL agents remains a fundamental challenge within AI alignment research. This master's thesis focuses on avoiding negative side effects (NSEs): preventing agents from causing unintended harm to their environment while pursuing their assigned objectives.\n\nTo capture the current state of research, a systematic literature review was conducted following the PRISMA framework, guided by two research questions: (i) What approaches have been taken in previous studies to ensure that RL agents avoid negative side effects while completing their tasks? (ii) What areas within this domain require further exploration? A multi-pronged search strategy combining database searches, targeted workshops, and author-specific queries was employed for publications between 2019 and 2023, complemented by preprints from reputable sources. Retrieved records were independently scored by four reviewers, and included studies were classified along a proposed taxonomy of underlying agent assumptions: fully observable states, availability of a no-op action, known dynamics, linear feature rewards, action-independent rewards, and known past states.\n\nThe review identifies five principal methodological categories for mitigating NSEs. Reward regularization approaches, such as Occupancy-Regularized Policy Optimization, augment the reward function with penalties for harmful behavior. Impact regularization methods, including relative reachability and attainable utility preservation, penalize deviations from a baseline state to avoid irreversible changes to the environment. Inverse reinforcement learning infers human-aligned reward functions from expert demonstrations or the initial configuration of the environment. Human feedback and model-based penalty prediction approaches, such as minimax-regret querying and the parenting algorithm, incorporate human input or learned environment models to anticipate and avoid unsafe actions. Finally, initial work on safety in multi-agent systems addresses emergent side effects arising from agent interactions.\n\nThe classification reveals a strong reliance on idealized assumptions, most notably fully observable states, known dynamics, and the availability of a no-op action as an impact baseline. These assumptions rarely hold in real-world settings, where agents face partial observability, noisy sensory data, and unpredictable or evolving dynamics. Furthermore, the analysis highlights a persistent trade-off between task performance and safety: regularization-based methods frequently yield overly conservative agents, while evaluations remain largely confined to simulated gridworld environments such as AI Safety Gridworlds and SafeLife, limiting real-world transferability.\n\nBased on these findings, the master's thesis identifies several gaps requiring further exploration: handling partial observability and non-Markovian side effects whose consequences depend on historical trajectories, integrating real-time human feedback into agent decision-making, establishing standardized and generalizable safety benchmarks, providing long-term safety guarantees, extending safety mechanisms to multi-agent settings, and addressing fairness, accountability, and transparency of safety mechanisms. Closing these gaps is essential for the safe deployment of RL agents in complex, real-world scenarios."}]},{"@type":"bf:Summary","skos:prefLabel":[{"@language":"eng","@value":"Reinforcement learning (RL) has made significant progress in solving complex tasks across various\ndomains, yet ensuring the safety and reliability of RL agents remains a critical challenge. This thesis\nfocuses on avoiding negative side effects (NSEs) in RL, a key aspect of AI safety that seeks to prevent\nagents from causing unintended harm while pursuing their goals. A systematic literature review was\nconducted to explore existing methodologies aimed at mitigating NSEs, with a particular emphasis on\napproaches such as reward regularization, impact regularization, inverse reinforcement learning, and\nmodel-based penalty prediction. The review reveales a strong reliance on assumptions such as fully\nobservable states and known dynamics, which limit the applicability of current safety mechanisms in\nreal-world, partially observable environments. The thesis also identifies gaps in the literature, including\nthe need for more robust methods to handle non-Markovian side effects, better integration of real-time\nhuman feedback, and the development of standardized safety benchmarks. Addressing these gaps is\ncrucial for advancing the safe deployment of RL agents in real-world scenarios, where complexity and\nunpredictability pose significant challenges."}]}],"bf:physicalLocation":[{"@language":"eng","@value":"Library, University of Applied Sciences St. Pölten, Campus-Platz 1, St. Pölten, Austria"}],"bf:tableOfContents":[{"@type":"bf:TableOfContents","skos:prefLabel":[{"@language":"eng","@value":"1 Introduction\n1.1 Research Questions\n1.2 Related Work\n2 Methodology\n2.1 Eligibility Criteria\n2.2 Search Criteria\n2.3 Search Strategy\n3 Background\n3.1 Markov Decision Process\n3.1.1 Partially Observable Markov Decision Process\n3.1.2 Constrained Markov Decision Process\n3.1.3 Factored Markov Decision Process\n3.1.4 Lexicographic Markov Decision Process\n3.2 Value Methods\n3.2.1 Policy and Value Functions\n3.2.2 Tabular Methods\n3.2.3 Approximate Methods\n3.3 Direct Policy Methods\n3.3.1 Policy Gradient\n3.3.2 Actor-Critic\n3.4 Inverse Reinforcement Learning\n3.4.1 Maximum Entropy IRL\n3.4.2 Maximum Causal Entropy IRL\n4 Systematic Literature Review\n4.1 Agents without safety mechanisms\n4.1.1 Tabular Solution Methods\n4.1.2 Deep Reinforcement Learning\n4.1.3 Text-based Reinforcement Learning\n4.2 Reward Regularization\n4.2.1 Avoiding Negative Side Effects by Considering Others\n4.2.2 Avoiding Epistemic Side Effects\n4.2.3 Occupancy-Regularized Policy Optimization\n4.3 Impact Regularization\n4.3.1 (Relative) Reachability Measures\n4.3.2 Attainable Utility Preservation\n4.3.3 Safety Aware Reinforcement Learning\n4.3.4 Future Task Approach\n4.3.5 Potential-based Multiobjective Reinforcement Learning\n4.3.6 Challenges using Impact Regularization\n4.3.7 Theoretical Considerations\n4.4 Model-based Penalty Prediction\n4.4.1 Human Feedback\n4.4.2 Constrained Models\n4.4.3 Planning with Reduced or Incomplete Models\n4.4.4 Symbolic Planning\n4.4.5 Environment Shaping\n4.4.6 Safety Wrappers\n4.5 Inverse Reinforcement Learning\n4.5.1 MDP Solver\n4.5.2 Implicit Preference Information\n5 Results\n5.1 Classification of Agents\n5.1.1 Agents without safety mechanisms\n5.1.2 Reward Regularization\n5.1.3 Impact Regularization\n5.1.4 Model-based Penalty Prediction\n5.1.5 Inverse Reinforcement Learning\n5.2 Gaps in the Literature and Future Directions\n5.2.1 Handling Partial Observability and Non-Markovian Side-Effects\n5.2.2 Real-Time Human Feedback and Interaction\n5.2.3 Lack of Standardized Benchmarks and Real-World Transferability\n5.2.4 Challenges in Balancing Task Performance and Safety\n5.2.5 Safety in Multi-Agent Systems and Emergent Side-Effects\n5.2.6 Long-Term Safety Guarantees\n5.2.7 Ethical Considerations and Consistent Definitions of Side-Effects\n6 Discussion\n6.1 Assumptions and Limitations of Current Safety Mechanisms\n6.2 Performance-Safety Trade-offs\n6.3 Challenges in Real-World Applications\n7 Conclusion\n7.1 Answering the Research Questions\n7.2 Limitations and Mitigation Strategies\n7.3 Future Work\nList of Figures\nList of Tables\nGlossary\nReferences"}]}],"dce:subject":[{"@type":"skos:Concept","skos:prefLabel":[{"@language":"eng","@value":"Reinforcement Learning"}]},{"@type":"skos:Concept","skos:prefLabel":[{"@language":"eng","@value":"AI Safety"}]},{"@type":"skos:Concept","skos:prefLabel":[{"@language":"eng","@value":"Negative Side Effects"}]},{"@type":"skos:Concept","skos:prefLabel":[{"@language":"eng","@value":"Safe Reinforcement Learning"}]},{"@type":"skos:Concept","skos:prefLabel":[{"@language":"eng","@value":"Impact Regularization"}]},{"@type":"skos:Concept","skos:prefLabel":[{"@language":"eng","@value":"Reward Regularization"}]},{"@type":"skos:Concept","skos:prefLabel":[{"@language":"eng","@value":"Inverse Reinforcement Learning"}]},{"@type":"skos:Concept","skos:prefLabel":[{"@language":"eng","@value":"Markov Decision Process"}]},{"@type":"skos:Concept","skos:prefLabel":[{"@language":"eng","@value":"Reward Specification"}]},{"@type":"skos:Concept","skos:prefLabel":[{"@language":"eng","@value":"Safe Exploration"}]},{"@type":"skos:Concept","skos:prefLabel":[{"@language":"eng","@value":"AI Alignment"}]},{"@type":"skos:Concept","skos:prefLabel":[{"@language":"eng","@value":"Survey"}]},{"@type":"skos:Concept","skos:prefLabel":[{"@language":"eng","@value":"Systematic Literature Review"}]}],"dce:title":[{"@type":"bf:Title","bf:mainTitle":[{"@language":"eng","@value":"Better Safe Than Sorry"}],"bf:subtitle":[{"@language":"eng","@value":"A Systematic Literature Review on Avoiding Negative Side-Effects in Reinforcement Learning"}]}],"dcterms:created":["2024-09-10"],"dcterms:language":["eng"],"dcterms:subject":[{"@type":"skos:Concept","rdfs:label":[{"@language":"deu","@value":"ÖFOS 2012 --"},{"@language":"eng","@value":"ÖFOS 2012 --"}],"skos:exactMatch":["oefos2012:102019"],"skos:notation":["102019"],"skos:prefLabel":[{"@language":"deu","@value":"Machine Learning"},{"@language":"eng","@value":"Machine learning"}]},{"@type":"skos:Concept","rdfs:label":[{"@language":"deu","@value":"ÖFOS 2012 --"},{"@language":"eng","@value":"ÖFOS 2012 --"}],"skos:exactMatch":["oefos2012:102001"],"skos:notation":["102001"],"skos:prefLabel":[{"@language":"deu","@value":"Artificial Intelligence"},{"@language":"eng","@value":"Artificial intelligence"}]}],"dcterms:type":[{"@type":"skos:Concept","skos:exactMatch":["https:\/\/pid.phaidra.org\/vocabulary\/69ZZ-2KGX"],"skos:prefLabel":[{"@language":"eng","@value":"Text"},{"@language":"deu","@value":"Text"},{"@language":"ita","@value":"Documento PDF"}]}],"ebucore:filename":["thesis.pdf"],"ebucore:hasMimeType":["application\/pdf"],"edm:hasType":[{"@type":"skos:Concept","skos:exactMatch":["https:\/\/pid.phaidra.org\/vocabulary\/62DN-RZ7V"],"skos:prefLabel":[{"@language":"eng","@value":"thesis"},{"@language":"deu","@value":"Hochschulschrift"},{"@language":"ita","@value":"Tesi"}]}],"edm:rights":["http:\/\/creativecommons.org\/licenses\/by\/4.0\/"],"role:res":[{"@type":"schema:Person","schema:birthDate":["1993-01-25"],"schema:familyName":[{"@value":"Kovac"}],"schema:givenName":[{"@value":"Fabian"}],"skos:exactMatch":[{"@type":"ids:orcid","@value":"0000-0002-1611-2598"}]}],"schema:numberOfPages":["126"]}