Building an RL environment where safety is non-negotiable

Most RL environments let the agent fail. Bump into a wall, lose points, learn. Railway signalling doesn't have that property — a collision isn't a negative reward, it's a catastrophic outcome that must be structurally impossible.

I built a railway traffic controller as an OpenEnv-protocol environment to explore how you encode that.

Block signalling as an invariant

Real railways divide track into fixed blocks. The rule is absolute: one train per block. Signals control entry, and a signal cannot clear into an occupied block.

The naive RL approach is to let the agent set signals freely and penalise collisions heavily. This works badly — the agent spends enormous sample budget rediscovering a rule you already know, and any residual probability of violating it is unacceptable.

Instead, the invariant belongs in the environment: the agent chooses among safe actions only. It cannot express an unsafe one. Now the learning problem is the interesting one — throughput, priority, delay minimisation — instead of "please don't crash."

Difficulty as curriculum

Four levels, each adding one concept:

1. Basic Control — single line, signal timing only
2. Junction Management — routing conflicts at converging tracks
3. Express Priority — trains with differing precedence, so optimal play means deliberately delaying something
4. Rush Hour — density high enough that greedy dispatch deadlocks

Level 4 is the one that matters. Greedy local decisions produce gridlock: every train advanced into the next available block, no block free to resolve the conflict. Getting through it requires holding a train that could move.

Shipping it as a tool surface

Wrapping the environment in FastAPI with WebSocket support, plus an MCP tool suite for signal control, routing and network status, meant an LLM agent could drive it directly — no RL training loop needed to evaluate reasoning about dispatch.

Docker packaging made the whole thing one command to run, which matters more than it sounds: an environment nobody can start is an environment nobody evaluates against.

What I'd change

The reward function is still too hand-tuned. Throughput, delay and priority-adherence are combined with weights I picked by intuition, and different weights produce meaningfully different policies. A proper treatment would learn from dispatcher demonstrations rather than asserting the trade-off up front.