Skip to the text
RobustRLlib Tutorial Project Page

Robust Offline Algorithms

Robust offline methods learn from a fixed dataset and never interact with the environment, so they meet every shift without having seen it. They are organised by where robustness enters: the learner, the data, or a generative model of the data.

Features

Feature Robust Offline Algorithms
Interface One experiment file per run, launched with a training script
Method selection The algorithm card named by the experiment file
Training data A fixed dataset, named by the task card
Training environment None. The environment is used for evaluation only
Budget Gradient updates, native to each method
Evaluation The frozen checkpoint, on the grid named by the eval card

Supported methods

Method Family Base Claimed
robustness		 Shifted-env
rollout		 Adversarial
network		 Learned
model
RFQI Learner-centric FQI Dynamic
RORL Learner-centric SAC Observation
ATLA-IQL Learner-centric IQL Observation ✓
RSC-IQL Data-centric IQL Semantic ✓
RAMBO Data-centric SAC Dynamic ✓ ✓
ROMB Generative IQL Dynamic ✓ ✓
FWM Generative IQL Dynamic ✓
PLR-PVL Generative IQL Dynamic ✓

Claimed robustness is the shift the original paper targets.

Run a method

Every method is launched from an experiment file. RORL on Hopper:

python baselines/train.py \
    -c robustrllib/configs/experiment/rorl_hopper.yaml --seed 0
python baselines/evaluate.py --run runs/rorl_hopper/seed0

Every method page gives the command for that method. Train an algorithm explains the experiment file, the overrides and the run directory.

Configuration

Method Implementation Experiment file Training budget
RFQI baselines/rfqi rfqi_hopper.yaml 500k updates, batch 1000
RORL baselines/rorl rorl_hopper.yaml 3M updates on MuJoCo and Door, 1M elsewhere
ATLA-IQL baselines/atla_iql atla_iql_hopper.yaml 1M updates, batch 256
RSC-IQL baselines/rsc_iql rsc_iql_hopper.yaml 1M updates, batch 256
RAMBO baselines/rambo rambo_hopper.yaml 2M updates on MuJoCo and Door, 1M elsewhere
ROMB baselines/romb romb_hopper.yaml 1M updates, batch 256
FWM baselines/fwm fwm_hopper.yaml 1M updates, batch 256
PLR-PVL baselines/plr_pvl plr_pvl_hopper.yaml 1M updates, batch 256

Experiment files are in robustrllib/configs/experiment/. Each names the algorithm card, the task card and the evaluation grid of the run. Every method keeps its native training recipe and budget; what is shared is the evaluation.

Robust performance

Normalized score of the frozen checkpoint. Q1 to Q4 are the severity quartiles of each shift factor's scales, ordered by displacement from the nominal setting, and All covers every shifted condition.

Method Nominal Q1 Q2 Q3 Q4 All
RFQI 79.3 58.3 50.4 36.5 29.7 43.6
RORL 79.3 60.3 48.7 40.8 31.4 45.3
ATLA-IQL 108.0 81.4 61.3 44.3 40.4 56.9
RSC-IQL 100.4 80.3 56.2 42.2 33.3 53.0
RAMBO 95.5 73.5 57.4 45.9 37.8 53.7
ROMB 106.6 79.7 59.7 44.0 36.2 54.9
FWM 105.9 81.8 62.1 45.3 37.0 56.6
PLR-PVL 102.7 76.9 60.2 45.4 37.3 55.0

Scores are not clipped: 0 and 100 are reference points, not bounds. The protocol is described in Evaluation Protocol.

Method pages

Learner-centric

  • RFQI: Worst-case Bellman update over a total-variation uncertainty set.
  • RORL: Local policy and value smoothing, with an ensemble penalty at perturbed states.
  • ATLA-IQL: A learned observation adversary, trained through the critic instead of by RL.

Data-centric

  • RSC-IQL: Rewrites logged transitions along the dimensions a causal mask marks as non-causal.
  • RAMBO: Model rollouts from an ensemble that is trained adversarially against the policy.

Data-centric · generative

  • ROMB: A world model adapted against the policy under a constrained maximin objective.
  • FWM: Multi-step critic targets whose future is sampled from a flow-matching world model.
  • PLR-PVL: An ensemble of world models, prioritised per sample by the value loss they induce.