RAMBO¶

Robust Adversarial Model-Based Offline RL

OfflineData-centricBase · SACClaims · dynamic shift

Model rollouts from an ensemble that is trained adversarially against the policy.

Features¶

Mechanism¶

RAMBO augments the dataset with short rollouts from a learned dynamics ensemble and replaces the uncertainty penalty by adversarial training of the model. The ensemble is updated to lower the critic's value of the policy's transitions while a maximum-likelihood term keeps it close to the data. The policy is behaviour-cloned before the first epoch.

RAMBO is a standalone recipe rather than a mechanism added to a shared base algorithm.

Run the method¶

python baselines/train.py \
    -c robustrllib/configs/experiment/rambo_hopper.yaml --seed 0
python baselines/evaluate.py --run runs/rambo_hopper/seed0

Training is on the nominal task. The configuration files are explained in Train an algorithm, and the evaluation of the frozen checkpoint in Evaluation Protocol.

Robust performance¶

Normalized score of the frozen checkpoint. Q1 to Q4 are the severity quartiles of each shift factor's scales, ordered by displacement from the nominal setting, and All covers every shifted condition. The last row averages the tasks.

References¶

• Rigter et al. RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning. NeurIPS, 2022.