Overview¶
RobustRLlib is a library and benchmark for robust reinforcement learning that takes the algorithm as its unit of comparison. It holds 16 robust and robust safe methods and 6 standard algorithms behind one interface, a toolbox that declares deployment shifts, and one evaluation protocol that is applied to all of them.
Why an algorithm-centric library?¶
Robust RL has produced many mechanisms, but each is developed, trained and reported inside its own setting. Two barriers follow.
- Algorithms are not reusable. Methods are spread over separate implementations and training pipelines. There is no single interface through which a broad collection of robust methods can be tried on a new problem.
- The capability boundary is unclear. Methods are reported under different tasks, budgets and shift definitions, sometimes under the very shift their mechanism targets.
Algorithm groups¶
The library has four groups. Robust methods are split further by where robustness enters.
- Standard algorithms. The base learners that the robust methods are built on, and the reference for every comparison.
- Robust online algorithms. Learner-centric methods train against an adversary or on counterfactual replay, and environment-centric methods collect their rollouts in a perturbed environment.
- Robust offline algorithms. Learner-centric methods change the backup or the regulariser, data-centric methods reshape the training distribution, and generative methods train on futures sampled from a learned model.
- Robust safe algorithms. Methods that add a cost constraint and make it hold under shifted dynamics.
| Group | Methods |
|---|---|
| Standard | IQL, TD3+BC, MOPO, SynthER, PPO, SAC |
| Robust online | ATLA, ATLA-SA, RSC, RARL-T, RARL-P, DR |
| Robust offline | RFQI, RORL, ATLA-IQL, RSC-IQL, RAMBO, ROMB, FWM, PLR-PVL |
| Robust safe | RAMU, SPiDR |
Library features¶
- One interface for 22 algorithms. Every method is launched from an experiment file and keeps its native training recipe and budget.
- Attribution built in. Every method records the mechanism it adds, the shift it claims to address and the base algorithm that realises it.
- Six shift sources and five modes. Shifts are declared as data and stacked in any order. See Shift Sources and Modes.
- One evaluation protocol. One frozen checkpoint of every method is evaluated on the same grid. See Evaluation Protocol.
- Extensible. A new algorithm implements four methods, and a new simulator implements one adapter.
Comparison with related benchmarks¶
RobustRLlib extends existing RL libraries and robustness benchmarks by combining a library of robust algorithms with shifts on every component of the interaction loop, under one interface.
| Feature | RLlib | RRLS | ODRL | Robust-Gymnasium | RWRL Suite | RoAd-RL | RobustRLlib |
|---|---|---|---|---|---|---|---|
| Robust algorithms | 0 | 4 | 0 | 4 | 0 | 0 | 16 |
| MDP shifts | 0/4 | 2/4 | 1/4 | 4/4 | 3/4 | 1/4 | 4/4 |
| Latency shift | |||||||
| Semantic shift | |||||||
| Compound shift | |||||||
| Task expansion |
supported and evaluated partial absent or not demonstrated
RobustRLlib integrates 16 robust and robust safe methods next to 6 standard algorithms. It evaluates shift on all four MDP components: observations, actions, transitions, and reward or cost. It adds Latency shift, Semantic shift and compound shift, and carries the same interface to additional simulators.
Robust algorithms counts the robust or robust safe methods that a benchmark evaluates, without standard algorithms. MDP shifts counts, out of four, the components on which shift is evaluated. Compound shift requires several shift sources to be active at once. Task expansion is an interface for carrying the benchmark to further simulators.
Code structure¶
The code has four parts. Configuration selects what runs, Algorithms and the Shifted environment interact during training, and Evaluation scores the result.
What each part provides¶
| Part | Location | Interface | Meaning |
|---|---|---|---|
| Public interface | robustrllib |
make_robust, build_pipeline, ShiftSpec, make_env, load_config, bind_actor |
Everything a user script imports |
| Shift declaration | robustrllib.spec |
ShiftSpec(target, mode, params, schedule, seed) |
One shift, written as data |
| Pipeline | robustrllib.pipeline |
make_robust(env_id, shifts, seed) |
Creates the task and stacks one wrapper per shift, in list order |
| Shift wrappers | robustrllib.shifts |
ShiftWrapper, one subclass per target |
Applies a shift at reset and at step |
| Schedules | robustrllib.schedule |
make_schedule(spec) |
The Non-stationary mode: a multiplier that changes within the episode |
| Simulator adapters | robustrllib.adapters |
DynamicsAdapter: list_params, get_nominal, set_param, reset_all |
Reads and writes the parameters of one simulator |
| Algorithm interface | robustrllib.algos |
Algo: train, predict, save, load |
The contract a method implements to enter the library |
| Task environments | robustrllib.tasks |
make_env(task_card, shifts, seed) |
Builds the environment of a task card, for training and evaluation alike |
| Method implementations | baselines/<method>/ |
baselines/train.py -c <experiment.yaml> --seed N |
The training recipe of each method, behind one launcher |
| Configuration | robustrllib/configs/ |
Cards: algorithm, task, eval, experiment |
What to train, on which task, evaluated on which grid |
| Evaluation | baselines/evaluate.py |
--run <dir> [--eval <grid>] |
Runs the frozen checkpoint on every condition of a grid |
| Metrics | robustrllib.metrics |
normalize, condition_metrics, summary_metrics |
Turns returns into the reported scores |
Where to go next:
| To | Read |
|---|---|
| Run an existing method | Train an algorithm |
| Declare a shift | Shift Sources and Modes |
| Evaluate a checkpoint | Evaluation Protocol |
| Add a method of your own | Add a new algorithm |
| Add a simulator | Add a physics simulation |
To install the library and run a first method, continue with Quick Start.