Exact-oracle conformance tests for agentic RL

Know whether your credit estimator rewards the right turn.

AgentCreditBench tests GRPO, RLOO, GAE, GiGPO, Monte Carlo, and custom turn-level credit estimators against exact policy advantages in tiny, reproducible finite-horizon environments.

Run the benchmark Read the validation report

Source install git clone https://github.com/hectopascal/agent-credit-bench.git

The failure it catches

A successful trajectory can still reward the mistake.

In RecoveryEnv, a poor first action can be repaired later. On successful BAD → RECOVER trajectories, exact advantage assigns opposite signs. Outcome-broadcast methods do not.

Recovery diagnostic. Exact advantage assigns negative credit to BAD and positive credit to RECOVER. Broadcast, batch-centered, GRPO-style, and GiGPO-style estimators assign both actions positive credit on selected successful trajectories. — Selected successful paths only. GiGPO separates the magnitudes, but still gives BAD positive credit on this slice.

Mean turn credit on selected successful paths
Estimator	BAD	RECOVER
Exact oracle	−0.25	+0.50
Outcome broadcast	+1.00	+1.00
GRPO-style	+0.58	+0.58
GiGPO-style	+1.15	+1.57

This is a conditional identification diagnostic—not, by itself, evidence of a biased expected policy gradient. AgentCreditBench reports both questions separately.

Download the underlying CSV

Two questions, not one score

Credit fidelity and gradient validity are different.

Did the estimator identify the right action?

Compare its literal turn credit with exact advantage.

RMSE and centered RMSE
Spearman rank correlation
Sign accuracy
Zero-credit leakage

Does it induce the right training signal?

Compare the expected tabular-softmax policy gradient.

Mean-gradient cosine
Relative mean-gradient error
Gradient variance
Normalized gradient MSE

When to use AgentCreditBench

A small test bench before a large training run.

Validate a turn-level or step-level credit estimator against exact policy advantages.
Test whether GRPO, RLOO, GAE, GiGPO, or custom code leaks credit onto irrelevant turns.
Separate per-action credit quality from expected policy-gradient validity.
Run conformance checks against verl, TRL, OpenRLHF, and verifiers integrations.

Deliberately out of scope

End-to-end learning quality, function approximation, token-level policies, PPO clipping, learned critics, off-policy correction, and stochastic reward models. Tiny tabular MDPs are the point: they make the answer exact.

Three tiny environments, exact answers

Each isolates one credit-assignment pathology.

DelayedEffectEnv

One consequential action, followed by irrelevant distractor turns with exactly zero advantage.

Tests delayed leakage

RecoveryEnv

An early mistake can be repaired by a later action, separating contribution from outcome.

Tests repair credit

VariableHorizonEnv

STOP or CONTINUE at every turn, with the value of continuing changing over time.

Tests active-set baselines

Test the estimator you actually run

Transparent references plus pinned framework paths.

Start with pure-Python reference estimators, then score adapters that call or fingerprint the released advantage code in production RL stacks.

Core estimators

Exact oracleOutcome broadcastRLOO-style centeringTurnLOOGRPO-styleGiGPO-styleMonte Carlo

Framework paths

verl 0.9.0TRL 1.12.0OpenRLHF 0.11.0verifiers 0.1.14

Evidence, not screenshots

Every chart has committed data and a reproduction path.

Three-panel delayed-effect horizon sweep. Broadcast estimators keep roughly constant per-distractor leakage while total distractor credit and leakage fraction increase with horizon. Oracle leakage remains zero. — **Delayed leakage.** Per-distractor leakage stays roughly constant while total irrelevant credit grows with horizon; the leakage share approaches 0.97 at horizon 32. CSV

Log-log convergence curve. Monte Carlo advantage RMSE decreases from about 0.56 with one continuation to about 0.018 with 1,024 continuations, following an inverse-square-root reference rate. — **Monte Carlo convergence.** RMSE falls from 0.557 at one continuation to 0.0178 at 1,024, consistent with the inverse-square-root sampling rate. CSV

Run it locally

Source install, CPU only.

The distribution is not on PyPI yet, so the canonical installation path is the repository checkout. The core has no runtime dependencies.

Install and test

git clone https://github.com/hectopascal/agent-credit-bench.git
cd agent-credit-bench
python -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev]"
pytest

Score an estimator

from agent_credit_bench import run_benchmark, UniformPolicy
from agent_credit_bench.envs import DelayedEffectEnv
from agent_credit_bench.estimators import BatchCenteredBroadcast

result = run_benchmark(
    mdp=DelayedEffectEnv(horizon=16),
    policy=UniformPolicy(),
    estimator=BatchCenteredBroadcast(),
    batch_size=1_000,
    seeds=range(10),
)
print(result.mean_gradient_cosine)

Questions

What a good score does—and does not—mean.

Is AgentCreditBench an agent benchmark?

No. It scores credit estimators, not trained agents. There is no learning loop in the core benchmark.

Does exact credit guarantee better end-task performance?

No. Passing these diagnostics is evidence that the estimator obeys specific credit and gradient contracts in controlled finite MDPs; it is not an end-to-end training claim.

Why report identification and gradient metrics separately?

Adding a state-dependent baseline can change literal credit values while preserving the expected policy gradient. One scalar cannot faithfully score both properties.

Use and cite

Built to be inspected, extended, and cited.

AgentCreditBench is MIT licensed. Citation metadata is available in both CFF and BibTeX form.

CITATION.cff Copy BibTeX Archived v0.5.0

@software{yiyan2026agentcreditbench,
  author  = {Yi Yan Ng},
  title   = {{AgentCreditBench}: A Conformance-Test Suite
             for Turn-Level Credit Estimators},
  year    = {2026},
  version = {0.5.0},
  doi     = {10.5281/zenodo.22178939},
  url     = {https://github.com/hectopascal/agent-credit-bench},
  license = {MIT}
}