IQL¶

Implicit Q-Learning

OfflineStandard reference

Expectile-regression value learning that never queries out-of-distribution actions.

Features¶

Feature	Value
Group	Standard Algorithms
Setting	Offline
Family	Standard reference
Base algorithm	—
Claimed robustness	—
Training budget	1M updates, batch 256
Original paper	Kostrikov et al. Offline Reinforcement Learning with Implicit Q-Learning. arXiv:2110.06169, 2021.

Mechanism¶

IQL avoids querying out-of-distribution actions altogether. A value function is fitted to the critic by expectile regression, the twin critics bootstrap from that value function, and the actor is extracted by advantage-weighted regression.

Run the method¶

python baselines/train.py \
    -c robustrllib/configs/experiment/iql_hopper.yaml --seed 0
python baselines/evaluate.py --run runs/iql_hopper/seed0

File	Path
Implementation	`baselines/iql`
Algorithm card	`robustrllib/configs/algorithm/iql.yaml`
Experiment file	`robustrllib/configs/experiment/iql_hopper.yaml`

Training is on the nominal task. The configuration files are explained in Train an algorithm, and the evaluation of the frozen checkpoint in Evaluation Protocol.

Robust performance¶

Normalized score of the frozen checkpoint. Q1 to Q4 are the severity quartiles of each shift factor's scales, ordered by displacement from the nominal setting, and All covers every shifted condition. The last row averages the tasks.

Task	Nominal	Q1	Q2	Q3	Q4	All
Door	61.5	61.8	41.3	19.1	16.6	34.7
HalfCheetah	121.3	75.4	41.8	25.8	19.0	40.5
Hopper	106.8	59.2	51.2	44.0	24.7	44.8
LunarLander	100.0	94.7	86.5	72.8	70.4	81.6
PointMaze	100.0	64.0	51.3	44.0	38.7	49.5
Walker2d	125.1	109.6	69.7	46.8	43.0	67.3
Average	102.5	77.5	57.0	42.1	35.4	53.1

References¶

Kostrikov et al. Offline Reinforcement Learning with Implicit Q-Learning. arXiv:2110.06169, 2021.