Building a heads-up NLHE agent to beat Slumbot (2018 ACPC champion). The approach: Full CFR blueprint for preflop + real-time Cython-accelerated endgame solving for turn/river.
Background on CFR, Leduc Hold'em, and the LLM Head Coach experiments:
notebooks/cfr_leduc_theory.ipynb
| Config | Hands | mb/h | CI (95%) |
|---|---|---|---|
| 5M MCCFR, preflop only (M6) | 1K | -1092.9 | — |
| Full CFR + Endgame, baseline guards | 2K | -219.5 | ±554 |
| Full CFR + Endgame, tighter guards (V17) | 2K | -80.0 | ±681 |
| Full CFR + Endgame, V17 config (V21, ongoing) | 20K | ~-330 at 7K | ±409 |
mb/h = milli-big-blinds per hand. CI at 2K hands is wide — V21's 20K run is the statistical benchmark.
Preflop: Full CFR blueprint lookup
5M conservative MCCFR, 1667 info-sets, 200BB depth
Trained with ALLIN penalized for weak equity
Turn: Real-time subgame solving (~100ms)
50K Cython-accelerated MCCFR iterations
Equity-percentile opponent range (top 50%)
Conditional equity guard vs estimated range
River: Real-time subgame solving (~60ms)
300K Cython-accelerated MCCFR iterations
Conditional equity guard vs estimated range
Guards: Preflop fold: equity bucket < 65 at pot ≥ 10BB
Solver call: min bucket 62–85 depending on pot size
Conditional equity on turn/river calls in pots > 30BB
Key files:
rl_poker/agents/nlhe_full_cfr_endgame_agent.py— main agentrl_poker/solvers/vectorized_endgame.py+_cfr_core.pyx— Cython MCCFRrl_poker/utils/full_cfr_range_tracker.py— range narrowingmodels/nlhe_conservative_5M_cnodes.json— trained blueprint
The goal was always to replicate (or approximate) the techniques behind top academic poker bots. We worked through them roughly in order of implementation complexity.
Libratus (2017 ACPC champion) uses a precomputed CFR blueprint for early streets and resolves subgames in real time on the turn/river using the actual cards dealt. This is what the current M8 agent does — offline 5M-iteration Full CFR blueprint, then 50K/300K iteration Cython MCCFR per hand at runtime. The gap vs Slumbot (~-300 mb/h) is mostly from range model accuracy: our top-50% equity-percentile range doesn't capture Slumbot's action-conditioned range the way a full blueprint would.
NFSP trains two networks: a best-response network (deep Q-learning) and an average-strategy network (supervised learning on past play). The two play against each other and gradually converge toward Nash. We ran this with PyTorch on the M4 MacBook and got −718 to −1711 mb/h. The failure modes were instability in the Q-learning phase and the sheer size of the NLHE game tree — NFSP converges well on small games but struggled here.
SD-CFR replaces the lookup table with a neural network that approximates counterfactual values across all board textures. Theoretically this eliminates the need for hand abstraction entirely. We implemented it from scratch with a Cython-accelerated traversal loop and a 3-layer MLP. See the full journey below — the short version is: three non-obvious bugs, a self-play aggression inversion that persisted to 40M traversals, a fixed-opponent workaround that introduced its own bias, and ultimately worse Slumbot performance than the M6 CFR lookup table.
DeepStack (2017) uses a neural network trained offline to estimate the value of any (hand, board, range) tuple, then uses those estimates inside a real-time depth-limited subgame solver. We didn't implement this directly but the conditional equity guard (_conditional_equity_bucket) is a lightweight version of the same idea — using the opponent's estimated range at runtime to compute action-conditioned equity rather than equity vs a random hand.
| Milestone | Status | Notes |
|---|---|---|
| M1: TAG baseline | ✅ | +514 mb/h vs random |
| M2: HUD telemetry | ✅ | VPIP/PFR/AF/3-bet tracking |
| M3: LLM Head Coach | ✅ | Underperformed TAG |
| M4: NFSP | ❌ Abandoned | Unstable (-718 to -1711 mb/h) |
| M5: GTO-CFR hybrid | ✅ | +474 mb/h avg over TAG |
| M6: 5M MCCFR preflop | ✅ | -1092.9 mb/h vs Slumbot |
| M7: Deep CFR (SD-CFR) | ❌ Concluded | See section below |
| M8: Full CFR + Endgame | 🔄 In progress | ~-80 to -330 mb/h vs Slumbot |
Deep CFR is theoretically the most powerful approach — replacing hand-abstraction lookup tables with a neural network that generalises across all board textures without bucketing. In practice, getting it to work correctly uncovered non-obvious failure modes at every layer.
SD-CFR (Single Deep CFR) — reinitialises the advantage network each round and accumulates a strategy network from snapshots. Implemented from scratch with:
- A Cython-accelerated game traversal hot loop (
_deep_cfr_core.pyx) with a pure-C Monte Carlo equity evaluator running nogil - A 3-layer MLP (256 hidden, 129-dim input, 9-action unified output)
- Multi-process traversal (14 workers on GCP
c2-standard-16, 4 locally), with a reservoir advantage buffer
1. CHECK logic bug — The most damaging bug. CHECK was treated identically to CALL, immediately advancing the street. The fix: only advance on a check-check sequence. Without this, every training sample computed wrong utilities. The agent at 1M traversals was entirely based on corrupted data.
2. Column aliasing (the "unified action space" rewrite) — The original 5-action output had two overlapping action sets: NO_BET_ACTIONS and FACING_BET_ACTIONS both mapped to columns 0–4. Column 0 was trained as both CHECK (25% of positions) and FOLD (75% of positions). regret_matching averaged these to near-zero → 0% CHECK probability at all board textures, even at 40M traversals. Fix: unified 9-action output space where every action always maps to the same neuron.
3. Advantage normalisation — Raw targets of ±100 BB caused gradient explosion (adv_loss ~3,000,000). Fix: divide by STARTING_STACK. Loss dropped 60,000× to ~633. Strategy is unaffected since regret matching only uses relative magnitudes.
After fixing all the above, a structural problem emerged: both players in self-play mutually reinforce overbetting. With weak hands, the learner overbets because its opponent (another copy of itself) has learned to fold too much to big bets. Both sides converge to a wrong equilibrium — not a slow convergence to the right one. Scaling to 40M traversals produced identical results to 1M.
The diagnostic probe — flop-weak-ip aggr (aggression with a weak hand in position) — was stuck at 77%+ across all scales.
Breaking the mutual-reinforcement loop by training against a fixed calling-station heuristic (fold < 20% equity, call 20–70%, raise > 70%):
- Weak-hand aggression collapsed from 77% → 37–40% by round ~1600, then plateaued
- Implemented in C inside the Cython hot loop — early-exit before MLP inference for opponent nodes — 15× local speedup
- Trained 18.5M traversals on GCP (
c2-standard-16, 16 vCPU)
Internal eval: r250 (2.5M trav) vs r1500 (15M trav)
| Opponent | r250 | r1500 |
|---|---|---|
| Random | +239.8 mb/h | -18.5 mb/h |
| Calling Station | +4727.2 mb/h | +5455.7 mb/h |
| Nit | +460.2 mb/h | +540.0 mb/h |
| TAG | +580.3 mb/h | +1026.3 mb/h |
Clear improvement internally. But Slumbot told a different story.
| Model | vs Slumbot |
|---|---|
| M6 (CFR + TAG postflop) | -1092 mb/h |
| Deep CFR r250 (2.5M trav) | -1932 mb/h |
| Deep CFR r1500 (15M trav) | -2433 mb/h |
More traversals → better vs internal bots → worse vs Slumbot. The model learned to exploit the calling-station's tendencies, which Slumbot (a near-Nash solver) punishes hard with reraises.
- Self-play in SD-CFR does not converge to Nash in practice — it converges to a mutual-exploitation equilibrium between two copies of the same learner. This is a known structural problem, not a scale problem.
- Fixed-opponent fixes aggression inversion but introduces bias — the quality of convergence is bounded by the quality of the fixed opponent. A calling-station teacher produces a calling-station-exploiter.
- Internal evals are misleading — "beats TAG in 3/4 matchups" does not predict Slumbot performance. Bots too similar to the training opponent produce optimistic internal numbers.
- The correct fixed opponent is GTO — but if you had a GTO opponent already, you wouldn't need to train. The real solution is population-based training or a self-play algorithm with better guarantees (NFSP, DREAM, full CFR with neural function approximation).
- Cython early-exit is the right abstraction — skipping the MLP entirely for fixed-opponent nodes (pure C equity function instead) gave 15× speedup and is the correct pattern for any future mixed-opponent training.
rl_poker/
├── agents/
│ ├── nlhe_full_cfr_endgame_agent.py # Current best: Full CFR + endgame solver
│ ├── nlhe_full_cfr_agent.py # Blueprint-only agent
│ ├── nlhe_baseline.py # TAG heuristic (M1)
│ ├── nlhe_gto_agent.py # CFR preflop + TAG postflop (M5)
│ ├── nlhe_deep_cfr_agent.py # Deep CFR agent (M7, concluded)
│ ├── leduc_headcoach.py # Leduc LLM Head Coach
│ └── agents.py # Core Leduc agents
├── solvers/
│ ├── full_cfr.py # Full CFR (5-action preflop)
│ ├── vectorized_endgame.py # Cython MCCFR endgame solver
│ ├── _cfr_core.pyx # Cython C hot loop
│ ├── deep_cfr.py # SD-CFR (M7)
│ ├── _deep_cfr_core.pyx # Cython traversal + C equity for Deep CFR
│ └── nlhe_cfr.py # Abstracted MCCFR (169 preflop + equity buckets)
├── utils/
│ ├── full_cfr_range_tracker.py # Range narrowing (Full CFR)
│ ├── slumbot_client.py # Slumbot API client
│ ├── slumbot_bridge.py # RLCard ↔ Slumbot bridge
│ └── hud_tracker.py # HUD telemetry
scripts/
├── benchmark_full_cfr_endgame_slumbot.py # Main Slumbot benchmark
├── train_nlhe_cfr.py # Train Full CFR blueprint
├── train_deep_cfr.py # Train Deep CFR (M7)
└── evaluate_endgame.py # Local endgame eval vs baselines
models/
├── nlhe_conservative_5M_cnodes.json # Full CFR blueprint (5M iter)
└── nlhe_strategy_m6.json # Abstracted MCCFR (5M iter, 747 info-sets)
notebooks/
├── cfr_leduc_theory.ipynb # CFR from scratch + Leduc + LLM Head Coach
└── train_deep_cfr_kaggle.ipynb # Deep CFR training (Kaggle/GCP)
git clone https://github.com/Saheb/rl-poker.git
cd rl-poker
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
# Build Cython solver (required once)
PYTHONPATH=. python rl_poker/solvers/setup_cython.py build_ext --inplace
# Benchmark vs Slumbot
PYTHONPATH=. .venv/bin/python scripts/benchmark_full_cfr_endgame_slumbot.py \
--hands 2000 \
--strategy models/nlhe_conservative_5M_cnodes.json \
--resolve-street 2 \
--turn-iters 50000 \
--river-iters 300000 \
--range-mode equity_percentile
# Train blueprint (5M iterations, ~3h on M4)
PYTHONPATH=. .venv/bin/python scripts/train_nlhe_cfr.py --iterations 5000000Developed by Saheb.