ATLA¶

Alternating Training with Learned Adversaries

OnlineLearner-centricBase · PPOClaims · observation shift

A policy trained against an observation adversary that is itself an RL agent.

Features¶

Mechanism¶

ATLA trains a policy against an optimal observation adversary in the state-adversarial MDP. The adversary is itself an RL agent that outputs a bounded perturbation of the observation, is rewarded by the protagonist's negative return, and is updated in alternation with the protagonist.

Run the method¶

python baselines/train.py \
    -c robustrllib/configs/experiment/atla_hopper.yaml --seed 0
python baselines/evaluate.py --run runs/atla_hopper/seed0

Training is on the nominal task. The configuration files are explained in Train an algorithm, and the evaluation of the frozen checkpoint in Evaluation Protocol.

Robust performance¶

Normalized score of the frozen checkpoint. Q1 to Q4 are the severity quartiles of each shift factor's scales, ordered by displacement from the nominal setting, and All covers every shifted condition. The last row averages the tasks.

References¶

• Zhang et al. Robust Reinforcement Learning on State Observations with Learned Optimal Adversary. ICLR, 2021.
• Zhang et al. Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations. NeurIPS, 2020.