All Methods¶

RobustRLlib

The Algorithm Book

22 algorithms in four groups

Every method has one spread: what it is on the left, how it is configured and run on the right. The two variants of RARL share one spread.

• Turn the page with the buttons, the arrow keys or a swipe.
• A label on the edge of the book opens a group.
• Open the full page leads to the complete description of a method.

1

Standard · 1 of 6

IQL

Implicit Q-Learning

Standard reference

Expectile-regression value learning that never queries out-of-distribution actions.

IQL avoids querying out-of-distribution actions altogether. A value function is fitted to the critic by expectile regression, the twin critics bootstrap from that value function, and the actor is extracted by advantage-weighted regression.

Original paperKostrikov et al. Offline Reinforcement Learning with Implicit Q-Learning. arXiv:2110.06169, 2021.

3

Standard · 2 of 6

TD3+BC

TD3 with Behaviour Cloning

Standard reference

TD3 with a behaviour-cloning term balanced against the critic.

TD3+BC adds a behaviour-cloning term to the TD3 actor loss and normalises states. The term is balanced against the critic by dividing by the mean absolute Q-value over the batch.

Original paperFujimoto and Gu A Minimalist Approach to Offline Reinforcement Learning. NeurIPS, 2021.

5

Standard · 3 of 6

MOPO

Model-based Offline Policy Optimization

Standard referenceBase · SAC

SAC on real and model data, with rollout rewards penalised by model uncertainty.

MOPO learns an ensemble dynamics model, branches short rollouts from dataset states, and penalises the rollout reward by the model's predicted uncertainty before running SAC on the mixture of real and model data.

Original paperYu et al. MOPO: Model-based Offline Policy Optimization. NeurIPS, 2020.

7

Standard · 4 of 6

SynthER

Synthetic Experience Replay

Standard referenceBase · IQL

A diffusion model enlarges the dataset; the offline learner itself is unchanged.

SynthER fits a diffusion model to logged transitions and trains an unmodified offline learner on a much larger sample from it.

Original paperLu et al. Synthetic Experience Replay. NeurIPS, 2023.

9

Standard · 5 of 6

PPO

Proximal Policy Optimization

Standard reference

The on-policy reference: a clipped probability-ratio surrogate with GAE.

PPO is the on-policy reference: a clipped probability-ratio surrogate on trajectories from the current policy, with generalised advantage estimation. The library uses the Stable-Baselines3 implementation.

Original paperSchulman et al. Proximal Policy Optimization Algorithms. arXiv:1707.06347, 2017.

11

Standard · 6 of 6

SAC

Soft Actor-Critic

Standard reference

The off-policy reference: twin critics and an entropy-regularised stochastic actor.

SAC is the off-policy reference: twin critics with a clipped double-Q target and a stochastic actor whose objective adds an entropy bonus with automatically tuned temperature. The library uses the Stable-Baselines3 implementation.

Original paperHaarnoja et al. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. ICML, 2018.

13

Robust Online · 1 of 5

ATLA

Alternating Training with Learned Adversaries

Learner-centricBase · PPOClaims · observation shift

A policy trained against an observation adversary that is itself an RL agent.

ATLA trains a policy against an optimal observation adversary in the state-adversarial MDP.

Original paperZhang et al. Robust Reinforcement Learning on State Observations with Learned Optimal Adversary. ICLR, 2021.

15

Robust Online · 2 of 5

ATLA-SA

ATLA with state-adversarial regularisation and a recurrent policy

Learner-centricBase · PPOClaims · observation shift

ATLA plus a state-adversarial KL regulariser and a 100-step recurrent policy.

ATLA-SA adds the SA-PPO regulariser to ATLA: a KL penalty between the policy at a state and at the worst-case neighbour found by stochastic gradient Langevin dynamics.

Original paperZhang et al. Robust Reinforcement Learning on State Observations with Learned Optimal Adversary. ICLR, 2021.

17

Robust Online · 3 of 5

RSC

Robust RL against Spurious Correlation

Learner-centricBase · SACClaims · semantic shift

Causal counterfactual replay that removes correlations a policy would exploit.

RSC learns a causal graph and a dynamics model over the state variables from replayed experience, then generates counterfactual transitions by intervening on the variables the graph marks as non-causal for the reward.

Original paperDing et al. Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlation. NeurIPS, 2023.

19

Robust Online · 4 of 5

RARL

Robust Adversarial Reinforcement Learning

Environment-centricBase · PPO / TRPOClaims · dynamic shift

A two-player zero-sum game against an adversary that applies external forces.

RARL casts robustness as a two-player zero-sum game. An adversary applies bounded external forces to the robot and is trained to minimise the protagonist's return, and the two are updated in alternating blocks.

Original paperPinto et al. Robust Adversarial Reinforcement Learning. ICML, 2017.

21

Robust Online · 5 of 5

DR

Domain Randomization

Environment-centricBase · SAC / PPOClaims · dynamic shift

Simulator parameters resampled every episode from fixed ranges.

Domain randomisation samples simulator parameters from fixed ranges at every episode, so that the policy meets a distribution of dynamics during training.

Original paperTobin et al. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. IROS, 2017.

23

Robust Offline · 1 of 8

RFQI

Robust Fitted Q-Iteration

Learner-centricBase · FQIClaims · dynamic shift

Worst-case Bellman update over a total-variation uncertainty set.

RFQI replaces the fitted-Q target by a worst-case Bellman backup over a total-variation uncertainty set of a fixed radius around the empirical transitions.

Original paperPanaganti et al. Robust Reinforcement Learning using Offline Data. NeurIPS, 2022.

25

Robust Offline · 2 of 8

RORL

Robust Offline RL via Conservative Smoothing

Learner-centricBase · SACClaims · observation shift

Local policy and value smoothing, with an ensemble penalty at perturbed states.

RORL makes an SAC learner conservative through smoothing.

Original paperYang et al. RORL: Robust Offline Reinforcement Learning via Conservative Smoothing. NeurIPS, 2022.

27

Robust Offline · 3 of 8

ATLA-IQL

Offline adaptation of ATLA on an IQL backbone

Learner-centricBase · IQLClaims · observation shift

A learned observation adversary, trained through the critic instead of by RL.

The offline adaptation keeps ATLA's principle, a learned observation adversary alternated with the learner.

Original paperZhang et al. Robust Reinforcement Learning on State Observations with Learned Optimal Adversary. ICLR, 2021.

29

Robust Offline · 4 of 8

RSC-IQL

Causal counterfactual rewriting on an IQL backbone

Data-centricBase · IQLClaims · semantic shift

Rewrites logged transitions along the dimensions a causal mask marks as non-causal.

RSC-IQL applies RSC's counterfactual rewriting to logged data. A causal mask over state dimensions is fitted from the dataset, and transitions are rewritten by permuting or perturbing the dimensions the mask marks as non-causal.

Original paperDing et al. Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlation. NeurIPS, 2023.

31

Robust Offline · 5 of 8

RAMBO

Robust Adversarial Model-Based Offline RL

Data-centricBase · SACClaims · dynamic shift

Model rollouts from an ensemble that is trained adversarially against the policy.

RAMBO augments the dataset with short rollouts from a learned dynamics ensemble and replaces the uncertainty penalty by adversarial training of the model.

Original paperRigter et al. RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning. NeurIPS, 2022.

33

Robust Offline · 6 of 8

ROMB

Policy-driven world-model adaptation

Data-centric · generativeBase · IQLClaims · dynamic shift

A world model adapted against the policy under a constrained maximin objective.

ROMB adapts a learned world model against the policy under a constrained maximin objective.

Original paperChen et al. Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learning. arXiv:2505.13709, 2025.

35

Robust Offline · 7 of 8

FWM

Flow-matching World Model

Data-centric · generativeBase · IQLClaims · dynamic shift

Multi-step critic targets whose future is sampled from a flow-matching world model.

FWM transfers the multi-step future modelling of the Diffusion World Model to a flow-matching world model.

Original paperDing et al. Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learning. arXiv:2402.03570, 2024.

37

Robust Offline · 8 of 8

PLR-PVL

Prioritised world-model selection by positive value loss

Data-centric · generativeBase · IQLClaims · dynamic shift

An ensemble of world models, prioritised per sample by the value loss they induce.

PLR-PVL trains a collection of world models consistent with the offline data, treats each as a level in the sense of unsupervised environment design, and prioritises them during policy learning by the value loss they induce.

Original paperBerdica et al. Robust Offline Learning via Adversarial World Models. NeurIPS Workshop on Open-World Agents, 2024.

39

Robust Safe · 1 of 2

RAMU

Risk-Averse Model Uncertainty

Robust safeBase · PPOClaims · dynamic shift

A risk measure over sampled next-state perturbations, applied to reward and cost targets.

RAMU applies a risk measure over sampled next-state perturbations to both the reward and the cost targets.

Original paperQueeney and Benosman Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learning. NeurIPS, 2023.

41

Robust Safe · 2 of 2

SPiDR

Zero-shot safety under domain randomisation

Robust safeBase · PPOClaims · dynamic shift

Constrained learning under domain randomisation with a pessimistic cost penalty.

SPiDR trains a constrained policy under domain randomisation and adds a pessimistic cost penalty proportional to the disagreement of next-state predictions across sampled dynamics.

Original paperAs et al. SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer. NeurIPS, 2026.

43