Signal · reinforcement learning for traffic control

One small brain that runs every intersection.

A traffic signal decides who gets the green, usually by a rule written by hand. Here it is learned. I built a small traffic simulator and trained a signal controller inside it with reinforcement learning, then ran that one policy at every intersection: it beats a hand-tuned timing plan on road networks from one signal to twenty-five, the same 62-thousand-parameter model unchanged. The learned controller wins, which isn't surprising. What's worth knowing is when one signal talking to its neighbors actually helps. Mostly it doesn't; this page shows the cases where it does, and why. The figures run live in your browser.

scroll
The setup

Every light is a learning agent.

The usual way to run a signal is a rule someone wrote down, a fixed timing plan or a heuristic like MaxPressure that reads the queues and switches accordingly. Those are good: a tuned fixed-time plan is fine on steady demand, and MaxPressure is strong under load. Instead of writing the rule, I let each signal learn one. That is reinforcement learning: the controller (the agent) picks an action, gets a number back rating how well traffic then flowed (its reward), and shifts toward the choices that score higher, over and over. What it converges on is a policy, a mapping from what it sees to what it does. Run that for millions of steps inside a traffic simulator and it settles on a controller nobody hand-tuned.

Here, each intersection is one agent. It watches the queues on its incoming roads (its approaches) and a phase timer; that short list of numbers is its observation. Every few seconds it picks a phase: one of the handful of legal green patterns, like north–south through-traffic, or the protected left turns.

Its reward is negative pressure, roughly how much traffic is stacked up on its own approaches minus what is stacked downstream. It is cheap to compute from what one intersection can see, and it punishes flushing your queue into a street that is already full. Pressure is the exact quantity MaxPressure acts on; here it is what the policy is trained to reduce.

Anatomy of one intersectionapproaches, lanes, one signal
↑ N signal stop line through lanes approach (one road in) left-turn bay
One four-way, seen from above. Every road arriving at the signal is an approach. Near the signal an approach fans into lanes (here, two through lanes and a short left-turn bay), so cars turning left wait in their own pocket instead of blocking the ones going straight. The signal holds each movement at the stop line until it gets a green (here the two through lanes are green, the left bay is not). That's the physical picture; the next figure is about which movements are allowed a green at the same time.
Why an intersection needs more than one phasemovement conflicts
Phase 0 runs north–south through-traffic and right turns together with no crossing paths; if the protected left turns were added to that same phase they would cross the oncoming through-traffic, so they get their own phase instead.
Not every pair of movements can share a green. North–south through-traffic and its right turns never cross, so they run together as one phase. Add the left turns to that same green and they cut straight across the oncoming through-traffic and collide. So the lefts get a phase of their own. An intersection ends up with a handful of these non-conflicting green patterns, and choosing which one to run, and for how long, is the agent's whole job.
What one agent seesthe observation
this signal — the agent neighbor queue + wait, per approach the observation — one fixed-length list of numbers own approaches phase timer a summary of each neighbor same length on a 1-signal map or a 25-signal one
What one agent sees, and nothing more: the queue and wait on each of its own approaches, its current phase and how long it has held, and a one-line summary of each immediate neighbor: their queues, their phase, and the road between you. That is the whole input, and it is the same length whether the map has one signal or twenty-five, which is why a single learned policy can run any of them.
Three designs

How much each one sees.

The three designs differ in how much each one gets to see. Independent: one network per intersection, each blind to its neighbors. Shared: one network, every intersection runs the same weights, and each still sees a short summary of its immediate neighbors. The third adds a critic that, during training only, sees the whole network at once. The shared policy is the one that ships. It's the cheapest to train (every intersection feeds the same network, so it gets N times the experience per step) and the only one that transfers to a road network of a different size without retraining.

All three beat a fixed-time cycle. So does seeing the neighbors help at all? To answer that you have to know what coordination even buys. The clearest case is a green wave.

Green wave · a one-way arterial, five signalscoordinated
medium
tuned
stops per car—
cars cleared / min—
signals synced toprogression
Drag the offset. Zero = every light switches together; tuned = each turns green just as the platoon arrives.
A green wave is pure coordination: the signals' phase offsets are set so a platoon (a cluster of cars moving together) released by the first light hits each downstream light exactly as it turns green. Switch to every light for itself and the same traffic stops and starts at each signal. This is the coordination a neighbor-aware policy can learn and a blind one can't, but only if the geometry lets a platoon stay together long enough to matter.
See it run

Watch the learned policy drive.

Before putting a number on it, watch the policies drive. Here is the shared policy running a 3×3 grid of nine signals under a steady rush of traffic, next to the two controllers it has to beat: a fixed-time cycle that switches on a timer, and a random legal switcher. Same streets, same cars arriving in the same order; the only thing that changes is who decides the greens. Amber cars are moving; red cars are stopped; at each intersection a green stub marks the approach the signal is waving through right now. Flip between the three and watch which approaches each policy greens, and the running wait-time tally.

Live rollout · 3×3 grid, 9 signals · steady loadt = 0 s
moving stopped approach has green ↑ north
avg wait0s
cars cleared0
queued now0
in system0
average wait (seconds per car, lower is better)
Random0
Fixed-time0
Trained0
2×
Every controller sees the identical stream of cars, so any gap is the policy's doing. The clip opens on an already-loaded network, a few minutes into a steady rush, so you see the network already running rather than filling from empty. The trained shared policy holds the grid at a steady flow: queues form at the lights and drain. Fixed-time and random can't keep pace, so their queues keep growing (watch the red build and the in-system count climb) and their average wait runs about 60% higher. The bars keep a running tally across all three. These are recorded from the real simulator; the page replays the rollout rather than re-simulating it.
The measurement

Putting a number on the neighbor view.

Shared (sees neighbors) against independent (blind), each trained from scratch on the same network at a demand high enough that queues outgrow their own block and spill back into the intersection upstream. Both are compared to a fixed-time cycle. The number that matters is the coordination premium: how much more the shared policy beats fixed-time than the blind one does. If the neighbor view is worthless, the premium is zero and the two policies tie.

Pick a network below. Some are ordinary grids; some are the kind of thing the neighbor view was supposed to help with: a couplet (a pair of one-way streets running opposite directions), or an arterial (a through-road carrying most of the traffic).

Coordination premium · shared minus blind, under heavy loadselect a network
Couplet grid network map
What the gap tracks

Coordination pays where the streets are coupled.

The couplet grid (a whole grid of those one-way pairs) is where the neighbor view pays the most: a car leaving one signal has nowhere to go but straight into the next, so a green wave is both available and necessary. The single arterial is the same story on a smaller network.

Then there are the flat cases. In the one-way pair embedded in a two-way grid, the arterial where side streets can only turn right on and off it (right-in, right-out, or RIRO), and the grid hit by platoon surges, the blind policy matches the neighbor-aware one exactly. These aren't easy networks: RIRO and the platoon grid are congested and hard. They tie because traffic has somewhere else to go. When there are alternative routes and slack, a light that sees only its own queues does just as well. The premium measures how tightly one intersection's decisions land on the next; a busy, complicated-looking network can still score zero.

What these numbers don't claim

Where the numbers are thin.

Every figure is measured, but the budget was a laptop. The couplet and corridor coordination gaps come from three training runs each; the flat scenarios from two, and a few stress tests from one. The gaps hold across runs, but this is a workstation study, not a benchmark suite. I ran one layout per network type, with no sweeps over training settings.

The score is a within-simulator pressure return, not a field measurement, and it only compares within a network. A −112 on one map and a −4 on another are different maps, not a ranking. And each agent only ever sees one hop out (its immediate neighbors), so the observation itself limits how well any policy can do; some of the gap that stays open is baked into the inputs, not the model.

Where this leaves us

What constrains the result, and what's next.

Three things set the ceiling here: what each agent can observe, how the reward is shaped, and how tightly the road geometry couples one intersection to the next. Model size isn't one of them. The biggest network on this page is a couple of megabytes. Coordination isn't free performance you can add anywhere; it pays only where the streets are tight, and you can now predict which streets those are.

The one-hop view (each signal sees only its immediate neighbors) is still a keyhole. The next rung is a policy that can look further than one hop, so a signal reacts to a jam building two or three blocks away before it arrives. After that, I want to drop all of this into a game where you build the streets yourself.

Signal is a from-scratch, dependency-free traffic microsimulator and a multi-agent PPO (proximal policy optimization) trainer written around it. Every number on this page is measured, greedy-evaluated, and reproducible; the figures replay the results rather than render them. Some comparisons are single-run and marked as such in the source. Built by Matthew Burke.