PLR-PVL¶

Prioritised world-model selection by positive value loss

OfflineData-centric · generativeBase · IQLClaims · dynamic shift

An ensemble of world models, prioritised per sample by the value loss they induce.

Features¶

Mechanism¶

PLR-PVL trains a collection of world models consistent with the offline data, treats each as a level in the sense of unsupervised environment design, and prioritises them during policy learning by the value loss they induce.

The library replaces the original world model with the flow-matching model of FWM. The ensemble consists of flow models that differ only in training seed, each proposing one multi-step future for the same conditioning. The critic target is taken per sample from the model whose rollout carries the largest positive value loss, measured by the generalised-advantage residual of the current critic along that rollout.

Implementation notes.

• The selection rule is a learning-potential criterion, not a worst-case-return criterion: it picks the model the critic currently explains worst.
• The flow models are frozen during policy learning. No network is trained adversarially.

Run the method¶

python baselines/train.py \
    -c robustrllib/configs/experiment/plr_pvl_hopper.yaml --seed 0
python baselines/evaluate.py --run runs/plr_pvl_hopper/seed0

Training is on the nominal task. The configuration files are explained in Train an algorithm, and the evaluation of the frozen checkpoint in Evaluation Protocol.

Robust performance¶

Normalized score of the frozen checkpoint. Q1 to Q4 are the severity quartiles of each shift factor's scales, ordered by displacement from the nominal setting, and All covers every shifted condition. The last row averages the tasks.

References¶

• Berdica et al. Robust Offline Learning via Adversarial World Models. NeurIPS Workshop on Open-World Agents, 2024.