RARL¶

Robust Adversarial Reinforcement Learning

OnlineEnvironment-centricBase · PPO / TRPOClaims · dynamic shift

A two-player zero-sum game against an adversary that applies external forces.

Features¶

Mechanism¶

RARL casts robustness as a two-player zero-sum game. An adversary applies bounded external forces to the robot and is trained to minimise the protagonist's return, and the two are updated in alternating blocks.

The library provides two variants that differ only in the policy optimiser. TRPO appears in the library only as the base of RARL-TRPO.

Run the method¶

python baselines/train.py \
    -c robustrllib/configs/experiment/rarl_ppo_hopper.yaml --seed 0
python baselines/train.py \
    -c robustrllib/configs/experiment/rarl_trpo_hopper.yaml --seed 0
python baselines/evaluate.py --run runs/rarl_ppo_hopper/seed0

Training rollouts come from the method's own perturbed environment. The configuration files are explained in Train an algorithm, and the evaluation of the frozen checkpoint in Evaluation Protocol.

Robust performance¶

Normalized score of the frozen checkpoint. Q1 to Q4 are the severity quartiles of each shift factor's scales, ordered by displacement from the nominal setting, and All covers every shifted condition. The last row averages the tasks.

RARL-PPO

RARL-TRPO

References¶

• Pinto et al. Robust Adversarial Reinforcement Learning. ICML, 2017.