Elevator dispatch is a solved problem. The rules that run most buildings are seventy years old and genuinely good. Solved, anyway, until you’re the one in the lobby watching every car idle on the wrong floor, wondering what the holdup could possibly be. That wait is what made me try to beat the status quo: a learned policy against the incumbent, on identical traffic, measured honestly to find where it wins and where it doesn’t. Scroll to see what the agent sees, how it’s scored, how its network is wired, and the one change that mattered most.
The interactive visualizations need a wider screen; on mobile you'll get the prose and the tables.
Elevator dispatch is not an open problem. Buildings have run on hand-engineered rules for seventy years, and the rules are genuinely good: collective control (LOOK), cost-function dispatch, destination dispatch.
Take LOOK, the most common one. Each car sweeps in one direction, answering every call along the way, and turns around only when nothing is left ahead of it, like a bus that runs its route to the end and comes back. Unanswered hall calls get handed to whichever idle car is nearest. It needs no tuning, and in a quiet building it is hard to beat. What it can't do is plan. The assignment is greedy and car-by-car, with no picture of what the other cars, or the next minute of arrivals, will do. (The mechanics, and its stronger cost-function cousin, are laid out in full further down.)
So the honest framing here is narrow. A tuned rule set is a fixed strategy, and as buildings get bigger (more floors, more cars, more coordination between them) it should start leaving service on the table. That's the bet this project measures. In a small building I expect to lose. The interesting question is where the crossover is.
The agent's entire input is one flat vector of numbers: 98 of them in the small building, 254 in the mid-size, 648 in the large. But that vector is a live status panel, and it's easiest to read one block at a time.
Where is each car? A one-hot row per car marks the floor it's on, its height in the shaft.
Is each car in service? One bit per car flags the ones switched off; they gray out and drop from the agent's control.
Which way is it moving, and where between floors? Travel direction (down, idle, up) plus a smooth fractional position.
How full is it? Riders over capacity, drawn as the fill gauge, the difference between a car that can still stop for you and one that can't.
Where do the people aboard want off? The in-car destination buttons light the floors its riders are headed to.
Which floors are calling? A lamp beside a floor, up or down, means someone is waiting there.
And how long have they waited? The call lamps warm from steel to coral as the oldest person there closes on the 45-second abandon limit.
Together, these seven blocks are the observation, everything the agent knows and nothing else, flattened into that one vector.
The agent has one blind spot. It sees that people are waiting, and for how long, but never where they want to go. A real hall call carries no destination, so it must infer demand from the calls alone. This limit comes back later in a way I didn't expect.
Every half-second decision earns one number, summed from seven terms. Delivering a passenger is the big prize: +10.
Underneath, two costs accrue every second the system is congested: people riding, and, weighted three times heavier, people still waiting in the hall.
Let traffic overwhelm the cars and the catastrophic penalties apply: −8 when someone gives up, −5 when a full queue turns an arrival away.
The original design also adds a small shaping nudge: ±0.4 per floor a loaded car moves toward (or away from) a rider's destination. It looks harmless, but it turns out to be the most important coefficient in the whole project.
The simplest network flattens the whole status panel into one block: a plain MLP, one action head per car. No structure, and the network widens with every floor and car you add.
The structured alternatives all encode a prior. A convolution slides one filter up the floors, the same weights reused at every height, the way tall buildings repeat themselves.
One shared sub-network per car reads a common summary, so the weight set no longer grows with the fleet. Letting those per-car networks communicate through attention targets the hardest part of dispatch directly: coordination, so two cars don't chase the same call.
Each step adds more structure, and more built-in theory about the problem. The expected result is that structure wins as the building grows. I test that below rather than assume it.
The reward function is seven numbers. Drag them in the panel and watch which behavior wins. The coefficients are the goal, and small changes reorder the leaderboard.
Then open the live elevator simulator to run a full building yourself: arrivals, queues, and all six actions per car.
Every number here is a greedy policy scored on the same protocol as the classical baseline: hour-long episodes, five fixed seeds, identical passenger traffic for every method, wait times measured over delivered passengers. No cherry-picking against a weakened baseline; LOOK runs at its best.
The thesis held: in the small building RL ties the heuristic, because there was nothing to win. In the mid-size building it cuts mean wait by a quarter with zero abandonment; in the large one it delivers more people while cutting abandonment nearly five-fold.
None of that came from a fancier network. Flip the reward selector between shaped and unshaped and watch the same flat MLP jump. The full story is below the fold.
The simulation is turn-based underneath: passengers arrive by a Poisson process whose rate follows a 48-bin daily profile, queue per floor and direction (capacity 12, abandon after 45 s), and board capacity-8 cars. The policy acts every 0.5 s of simulated time, choosing among six discrete actions per car (idle, up, down, board-up, board-down, unload), while the physics ticks at 0.1 s underneath: a semi-MDP with reward accumulated between decisions. The experiments run at three building sizes:
| rung | floors | cars | population | observation dim | joint actions |
|---|---|---|---|---|---|
| S | 8 | 3 | 360 | 98 | 6³ = 216 |
| M | 16 | 5 | 600 | 254 | 6⁵ = 7,776 |
| L | 30 | 8 | 960 | 648 | 6⁸ ≈ 1.7M |
A dispatcher is only as interesting as the demand it faces, so the arrivals are a stochastic process with structure. Every tick, each floor draws a Poisson count of new passengers from a rate that follows a twelve-hour daily profile (48 fifteen-minute bins), and each new passenger draws a destination from an origin-dependent distribution. The profile is the sum of three overlapping demand components, and it's their balance over the day that produces the classic traffic patterns:
Origins are lobby-weighted (all incoming traffic starts at floor 0), and destinations follow a matrix: from the lobby, roughly uniform across the upper floors; from an upper floor, split between “back to the lobby” and “across to another upper floor.” A single intensity parameter scales the whole thing to a target load (population × a per-slot factor). The tensor experiments run the interfloor / midday slice at nominal load, deliberately the least forgiving regime for the agent's blind spot, since demand is spread across floors rather than funneled through the lobby, so the cars can't camp at the bottom. And because the RNG is counter-based, the same seed replays the identical passenger tape for every method, so LOOK and the learned policy are graded on the same people arriving at the same instants.
The elevator simulator began as a Unity project driven by ML-Agents, a full 3D build that steps one building at a time. For this work I rebuilt the entire simulation from scratch as a pure-Python tensor environment: identical rules and traffic, but the whole state lives in plain arrays, so hundreds of buildings step at once with no game engine in the loop. Every result on this page comes from that Python rebuild.
The tensor environment plus the PPO learner runs at ~46,000 agent-steps/s on a laptop CPU (about 37× the original Unity/ML-Agents pipeline, measured end-to-end on the same machine), so a five-million-step run drops from hours to roughly two minutes. That single change is why an experiment matrix this wide was possible; the Unity build is now the thing I check against, not the thing I train in.
And to trust a rewrite you have to prove it matches, so I built the Python version twice: a slow, readable single-instance reference and the fast batched one, both from a single spec, sharing a counter-based RNG so every random draw is a pure function of (seed, episode, step, slot). A differential-test battery steps the two side by side and compares every value at zero tolerance. The state is all-integer by design, so there is nothing to be approximately right about. When a number moves in the results below, it's the experiment that changed, not a drifting simulator.
The trainer is PPO with one categorical head per car and action masking: a car mid-travel can only no-op; boarding is masked when the car is full. Observations are normalized online; entropy is annealed to zero over training and the best checkpoint by episode return is kept. Two details took work to get right: evaluation is greedy (argmax, matching how deterministic dispatchers run), and a slightly under-converged stochastic policy can look competent while its argmax parks a full car on a legal no-op forever. Masking board-when-full and annealing entropy removed that failure on every seed. I also tried a value-based D3QN, natively greedy, stable, and ~1.7× LOOK's mean wait; PPO stayed.
Networks: 256×2 at S, 768×4 at M and L (the structured variants below use 512×3). Budgets: 5M steps at S (parity with the original Unity run), 10–15M at M/L. Evaluation protocol, identical for every method including LOOK: 3,600 s episodes with a 300 s warmup, seeds 1–5, identical traffic per seed, wait statistics over delivered passengers with p95 from a 64-bin histogram. Every RL number on this page is stated relative to LOOK on the same traffic tape.
Final policy (flat MLP, unshaped reward) versus LOOK, per building size, means over the five seeds:
| rung | method | delivered | wait mean (s) | wait p95 (s) | abandoned |
|---|---|---|---|---|---|
| S | LOOK | 245 | 6.0 | 16.5 | 0.2 |
| PPO | 245 | 5.85 | 16.2 | 0 | |
| M | LOOK | 418 | 9.1 | 26.1 | 3.4 |
| PPO | 422 | 6.86 | 18.0 | 0 | |
| L | LOOK | 678 | 10.4 | 30.3 | 14.6 |
| PPO | 686 | 9.83 | 28.3 | 3.0 |
Exactly the shape the thesis predicted: a tie where the heuristic is already near-optimal, and a clear win once there's coordination to exploit. At M that's −25% mean wait and zero abandonment; at L, more delivered with a fraction of the abandonment.
The reward has seven terms (scene 02). Shaped is the original design: it includes a small ±0.4-per-floor bonus/penalty for a loaded car moving toward (or away from) a rider's destination, a hint that pays the agent for looking like progress before anyone is delivered. Unshaped deletes those two movement terms, so the agent is paid only for real outcomes: deliveries, time spent waiting and riding, and the abandon / reject penalties. Every other coefficient is identical. That one difference is the whole experiment below.
I got the next part wrong. I built four architectures expecting structure to win at scale. Measured at M under the original (shaped) reward:
| architecture | training return | wait mean (s) | wait p95 (s) | abandoned |
|---|---|---|---|---|
| Flat MLP | 1,258 | 7.5 | 21.1 | 0.8 |
| Attention | 1,449 | 8.0 | 23.9 | 3.4 |
| Per-car | 1,366 | 8.9 | 27.0 | 5.0 |
| Conv | 1,326 | 8.9 | 31.6 | 26.0 |
Read those two columns against each other. Every structured network out-optimized the flat one on the training objective, and not one of them out-served it. They were learning the wrong thing better. That pattern, higher reward but worse service, points at the reward itself.
The suspect was the ±0.4 movement nudge from scene 02. It pays a loaded car for drifting toward destinations whether or not anyone gets delivered, a subsidy the more expressive networks learned to farm. Deleting those two coefficients and retraining the plain flat MLP:
| rung | reward | wait mean (s) | wait p95 (s) | abandoned |
|---|---|---|---|---|
| M | shaped | 7.5 | 21.1 | 0.8 |
| unshaped | 6.86 | 18.0 | 0 | |
| L | shaped | 9.45 | 28.4 | 19.8 |
| unshaped | 9.83 | 28.3 | 3.0 |
Strictly better where it counts. At L the abandonment drops from 20 to 3 for the same architecture. The single most important change in this project was deleting two numbers from the reward function. Months of architecture work resolved to an objective-alignment bug, which is either deflating or the most useful lesson here, depending on how you look at it. My take: measure the objective before you upgrade the model.