Skip to the text
RobustRLlib Tutorial Project Page

Standard Algorithms

Standard algorithms carry no robustness mechanism. They are the base learners that the robust methods are built on, and the reference that every robust method is measured against. The library holds four offline and two online standard algorithms.

Features

Feature Standard Algorithms
Role Base learner of the robust methods, and reference for comparison
Interface One experiment file per run, launched with a training script
Method selection The algorithm card named by the experiment file
Training On the nominal task, with the native recipe of each algorithm
Evaluation The frozen checkpoint, on the grid named by the eval card

Supported methods

Method Family Setting Base Claimed
robustness		 Shifted-env
rollout		 Adversarial
network		 Learned
model
IQL Standard reference Offline — —
TD3+BC Standard reference Offline — —
MOPO Standard reference Offline SAC —
SynthER Standard reference Offline IQL —
PPO Standard reference Online — —
SAC Standard reference Online — —

Run a method

Every method is launched from an experiment file. IQL on Hopper:

python baselines/train.py \
    -c robustrllib/configs/experiment/iql_hopper.yaml --seed 0
python baselines/evaluate.py --run runs/iql_hopper/seed0

Every method page gives the command for that method. Train an algorithm explains the experiment file, the overrides and the run directory.

Configuration

Method Implementation Experiment file Training budget
IQL baselines/iql iql_hopper.yaml 1M updates, batch 256
TD3+BC baselines/td3bc td3bc_hopper.yaml 1M updates, batch 256
MOPO baselines/mopo mopo_hopper.yaml 1M updates, batch 256
SynthER baselines/synther synther_hopper.yaml 1M updates, batch 256
PPO baselines/sb3 ppo_hopper.yaml 2M environment steps
SAC baselines/sb3 sac_hopper.yaml 1M environment steps

Experiment files are in robustrllib/configs/experiment/. Each names the algorithm card, the task card and the evaluation grid of the run. Every method keeps its native training recipe and budget; what is shared is the evaluation.

Robust performance

Normalized score of the frozen checkpoint. Q1 to Q4 are the severity quartiles of each shift factor's scales, ordered by displacement from the nominal setting, and All covers every shifted condition.

Method Nominal Q1 Q2 Q3 Q4 All
IQL 102.5 77.5 57.0 42.1 35.4 53.1
TD3+BC 89.4 70.5 57.1 44.5 32.9 51.3
MOPO 24.6 20.2 13.8 12.6 9.6 14.0
SynthER 104.0 74.2 56.2 43.5 38.5 53.2
PPO 71.2 58.8 47.7 41.2 37.2 46.2
SAC 78.6 70.6 64.6 49.0 43.0 56.7

Scores are not clipped: 0 and 100 are reference points, not bounds. The protocol is described in Evaluation Protocol.

Method pages

  • IQL: Expectile-regression value learning that never queries out-of-distribution actions.
  • TD3+BC: TD3 with a behaviour-cloning term balanced against the critic.
  • MOPO: SAC on real and model data, with rollout rewards penalised by model uncertainty.
  • SynthER: A diffusion model enlarges the dataset; the offline learner itself is unchanged.
  • PPO: The on-policy reference: a clipped probability-ratio surrogate with GAE.
  • SAC: The off-policy reference: twin critics and an entropy-regularised stochastic actor.