Evaluation Protocol¶

Every score of the benchmark is produced by one evaluator under one protocol. A policy is trained on the nominal task, a checkpoint rule fixed before evaluation picks the checkpoint to freeze, and the frozen checkpoint is scored on a grid of shift conditions.

Summary¶

The same evaluator scores every method of baselines/, offline and online: it loads the checkpoint with the class the method's algorithm card names, so it never depends on how the method was trained. The Isaac Lab and VLA studies have evaluators of their own, described in isaac/README.md and vla/README.md of the repository.

Evaluation grids¶

The excerpts leave out most conditions.

episodes: 20
seed: 10000
grid:
- name: nominal
  shifts: []
- name: gravity_0.9
  axis: gravity
  quartile: 1
  severity: 0.9
  shifts:
  - target: dynamics
    mode: scale
    params: {param: gravity, factor: 0.9}

- name: act_delay_8ms
  channel: theta_tau
  shifts:
  - target: latency
    mode: fixed
    params: {steps: 1}
- name: gear_ramp_0.8
  channel: theta_p
  shifts:
  - target: dynamics
    mode: scale
    params: {param: actuator_gear, index: all, factor: 1.0}
    schedule: {type: linear, start: 1.0, end: 0.8, t0: 0, t1: 1000}

- name: friction_0.6
  axis: friction
  quartile: 4
  severity: 0.6
  env_kwargs: {friction_scale: 0.6}
  shifts: []

The three excerpts are a Dynamic shift in the Parametric mode, a Latency shift and a Dynamic shift in the Non-stationary mode, and a condition CarRacing reads when it builds the track. part1_mujoco.yaml holds the nominal condition and three axes, gravity, limb length and actuator gear, each at eight factors from 0.6 to 1.4: 25 conditions.

A grid is checked before a long evaluation by building each condition once.

import yaml
from robustrllib import make_env

task = yaml.safe_load(open("robustrllib/configs/task/hopper.yaml"))
grid = yaml.safe_load(open("robustrllib/configs/eval/part1_mujoco.yaml"))["grid"]
for condition in grid:
    env = make_env(task, shifts=condition["shifts"], seed=0)
    env.reset(seed=0)
    env.close()
print(len(grid))

25

Run an evaluation¶

python baselines/evaluate.py --run runs/rorl_hopper/seed0

Checkpoint selection¶

The checkpoint is chosen by a rule that is written down before any evaluation, so that the grid never selects the policy it scores. The rule can be set per method, in the checkpoint key of the algorithm card, and per run on the command line.

python baselines/evaluate.py --run runs/rorl_hopper/seed0 --checkpoint last:3
python baselines/evaluate.py --run runs/rorl_hopper/seed0 --checkpoint best

checkpoint: last     # last, last:k or best; the default of every run of the method

With last:k the result file lists the k checkpoints, and returns of each condition holds k times episodes values. The rule in force is recorded next to the numbers, so results from different rules are never mixed unnoticed.

The Part 2 and Part 3 grids score the Part 1 Hopper runs a second and a third time.

python baselines/evaluate.py --run runs/rorl_hopper/seed0 --eval robustrllib/configs/eval/part2_hopper.yaml
python baselines/evaluate.py --run runs/rorl_hopper/seed0 --eval robustrllib/configs/eval/part3_hopper.yaml

Two kinds of shift need more than a frozen policy and a condition.

Result file¶

eval/<grid>.json records the protocol next to the numbers. Values are replaced by their types.

{
  "run": "rorl_hopper",
  "checkpoint_rule": "last",
  "checkpoint": "runs/rorl_hopper/seed0/ckpt/update_3000000.pt",
  "seed": 10000,
  "episodes": 20,
  "conditions": {
    "nominal": {"return_mean": "<float>", "return_std": "<float>", "return_min": "<float>",
                "cvar": "<float>", "n_episodes": 20, "score": "<float>",
                "episode_length_mean": "<float>", "returns": ["<float>", "..."]},
    "gravity_0.9": {"...": "...", "axis": "gravity", "quartile": 1, "severity": 0.9}
  }
}

Metrics¶

The normalized score uses one fixed pair of reference returns per environment, shared by every method and condition: the return of a random policy and the return of a reference policy. The score is the return minus the random reference, divided by the distance between the two references, times 100.

R_MIN, R_MAX = -20.27, 3234.3          # Hopper-v5


def normalized(ret):
    return 100.0 * (ret - R_MIN) / (R_MAX - R_MIN)


for ret in (R_MIN, R_MAX, 3600.0):
    print(f"{ret:12.3f} -> {normalized(ret):6.1f}")

     -20.270 ->    0.0
    3234.300 ->  100.0
    3600.000 ->  111.2

The score is not clipped. A return above the reference gives a score above 100, and a return below the random reference gives a negative score.

Protocol defaults¶

Combine training runs¶

A reported number is the mean over the training seeds of the normalized score.

import glob
import json
import numpy as np

paths = sorted(glob.glob("runs/rorl_hopper/seed*/eval/part1_mujoco.json"))
runs = [json.load(open(path)) for path in paths]

for name in runs[0]["conditions"]:
    scores = [run["conditions"][name]["score"] for run in runs]
    print(f"{name:16s}{np.mean(scores):7.1f} +/- {np.std(scores, ddof=1):4.1f}")

ddof=1 gives the sample standard deviation across training seeds.

Reproducible reporting¶

• The replication unit is an independently trained policy; the spread of a score is taken across training seeds.
• Episodes of one checkpoint are not independent samples, and their spread is not reported as the uncertainty of a method.
• The checkpoint rule is fixed before evaluation and recorded with the result; evaluation results are never used to select a checkpoint.
• Every method is scored on the same grid, with the same evaluation seed and the same number of episodes.
• Scores are normalized with the fixed reference pair of the environment and are not clipped.
• A result from an explicit checkpoint is reported as a diagnostic, not as the protocol score.