A player who is far better than their rating suggests, in a game where matches are played for money, has an obvious move: lose on purpose often enough that the rating never catches up, and keep farming beginners. Every rating system in wide use scores you on wins and losses, and one rate of losing on purpose - a tank rate - makes that record exactly what an honest player at that rating would produce. No test looking at results alone can ever separate the two. But they are there for the money, and every match they have to decide how much to stake before the result is known - and unlike everyone else at the table, they have already decided whether they are trying. This page builds a detector out of that, breaks it twice on purpose, and then stops guessing what a cheater looks like and learns it from the population. No statistics beyond what the page explains as it goes; the simulations run live in your browser, and the two figures that needed an offline model fit say so.
A smurf is an experienced player on a fresh account, stomping beginners for fun. It's annoying, matchmaking eventually sorts it out, and the cost is a bad hour.
Change one thing - make the matches wagered, for real money - and the same behavior becomes a business. Now the incentive is to stay in easy games, indefinitely, because the low bracket is where the profitable opponents are. That means managing your rating downward - losing on purpose, tanking, at whatever rate keeps the number where the money is. The industry word for this player is a shark, and unlike a smurf they have every reason to be patient and unremarkable.
To see why they get away with it, you need ninety seconds on how the scoreboard works.
Nearly every competitive rating in use descends from Elo - invented for chess by Arpad Elo in 1960, and quietly running ranked ladders, esports and even AI leaderboards ever since. It does exactly two things.
Before a match, it converts the gap between two ratings into a predicted win probability: E = 1 / (1 + 10^(-gap/400)). Equal ratings predict 50%. A 400-point favorite is predicted to win 91% of the time. The curve flattens toward the extremes but never quite reaches 0% or 100% - no rating gap makes an upset impossible.
After the match, it pays out surprise. Each rating moves by K · (result − E): a heavy favorite who wins collects almost nothing, because the result was already priced in, while an underdog who wins collects a lot. K is the volatility parameter - how hard one result moves the number.
The property everyone trusts is that this is self-correcting. Rated too low, you keep beating the prediction and climb; rated too high, you keep disappointing it and fall. But the correction audits only how often you win, against how often you should. So it self-corrects only against players who are trying to win. A player who controls how often they win controls their rating, and the machinery meant to correct them instead holds it exactly where they want it. (I've written a longer interactive guide to Elo if you want to play with the curve itself.)
Drag the tank rate - the fraction of matches deliberately thrown. Watch the rating go flat.
At a tank rate of zero the rating runs away and the shark loses their farm within fifty matches. Push the slider and the climb flattens. Press Solve for invisible and it stops dead.
That setting isn't found by search. Write e for the win probability the shown rating predicts and e_delta for the one the player's real strength predicts. Throwing a fraction tau of matches gives an observed win rate of (1 - tau) · e_delta + tau · 0.02. Set that equal to e - the record an honest player would produce - and solve for tau. The answer, which we'll call tau*, the pinning rate, is one line:
# sharkhunt/elo.py # expected: the win probability Elo assigns from the two shown ratings # e_delta: that probability re-priced as if the player were delta stronger # tank_win_prob: even a thrown match gets won 2% of the time - # a loss can be made very likely, never guaranteed def pinning_tank_rate(expected, delta, tank_win_prob=0.02): e_delta = shift_expectation(expected, delta) return (e_delta - expected) / (e_delta - tank_win_prob) # +400 Elo of hidden skill, even matchup: tau* = 0.460 # At tau* the observed win rate is 0.500 - the honest number, exactly.
The claim is stronger than "the test is slow". Any test built on results watches them arrive one at a time and accumulates evidence match by match. At tau* the evidence per match is zero, so a million matches accumulate a million zeros. There is no sample size at which the outcome record separates these two players, because there is nothing in it to separate.
The textbook tool is a sequential test: rather than gathering a fixed pile of data and analyzing it once, you score each match as it arrives and stop the moment the evidence is decisive, whichever way it points. The classic version is Wald's sequential probability ratio test, from 1945 - the same machinery computer-chess engine testing uses to decide whether a new engine is stronger, rather than just ahead so far.
It needs the accusation and the innocence stated precisely enough to price any result. H0, innocent: their skill is exactly what their rating says. H1, guilty: they are 400 Elo stronger and throwing 40% of their matches to hide it. Because both are fully specified, each can say precisely how likely any given result was.
That makes the scoring mechanical. After every match, ask both hypotheses how probable that result was, and add log( P(result | H1) / P(result | H0) ) - the log-likelihood ratio - to a running total. Concretely, take an even matchup: H0 prices a win at 50%, while H1 - stronger, but throwing 40% of the time - prices it at 55%. A win contributes log(0.55 / 0.50) = +0.10, a nudge toward guilt; a loss contributes log(0.45 / 0.50) = −0.11, a nudge toward innocence. Every match nudges the total one way or the other.
The total wanders like a random walk, and two horizontal lines bound it: cross the top and you accuse; cross the bottom and you clear the player and stop watching. Wald's result is that you can place those lines directly from the two mistakes you're willing to tolerate - α, the chance of ever accusing an honest player, and β, the chance of clearing a real shark. Accepting 1% of each puts the lines at ±4.60 (the general recipe is log((1-β)/α) and log(β/(1-α))).
Press Play. Twenty honest players, one shark who doesn't bother hiding, one who does.
The unconcealed shark gets caught every time, in a median of 56 matches. That is the test working. The honest players wander and mostly settle out the bottom, correctly cleared.
Now look at the amber line. It drifts down, the same direction as the honest players, and eventually exits through the "this player is fine" boundary. Over 2,000 simulated players, an outcome-only test caught 0.0% of sharks tanking at tau*, and exonerated most of them.
That's the worst possible failure. The test clears the shark.
Everything above only looks at who won. But a wagered game attaches a second number to every match, and the shark has to choose it before the result.
What they don't control is the result. Being better is not the same as winning. Hiding 400 Elo means winning about 91% of the matches you try to win - not all of them. And a match you are throwing still gets won 2% of the time, because your opponent can blunder badly enough that you win despite yourself. Over a hundred matches, our shark takes about five losses they never intended and backs into one win they didn't want.
What they do control is intent - try, or throw - fixed before the bet is placed. The stake therefore reveals the intent, and intent only correlates with the outcome rather than fixing it.
An honest player's stake cannot correlate with their result at all, because nothing they know beforehand predicts it. A shark's stake correlates with theirs imperfectly but relentlessly. That gap is the entire signal, and its imperfection is why no single match ever proves anything - a big bet that loses is something a shark does, roughly one time in eleven. It is why this has to be a test that accumulates, and why it takes a couple of dozen matches rather than one.
Before using it, the ground rules for the money on this page - every profit number later depends on them. A bet is one of three sizes, tiers min, mid and max, at 10%, 40% and 100% of the table cap. Both players post the same stake, the winner takes the pot, and the house keeps a 5% cut of it. The cut means a fair match is slightly worse than break-even: an honest player, matched fairly, loses money slowly. That slow loss is the baseline any cheat has to beat before the cheating pays for itself.
The obvious way to use the stakes is to keep the same test and scale each match's evidence by the size of the bet, so a big bet counts heavily and a tiny one barely at all. Call it the wager-weighted score.
# sharkhunt/detectors.py - the intuitive first attempt class WeightedLLR(OutcomeSPRT): def increment(self, obs): return super().increment(obs) * STAKES[obs.tier] / max(STAKES)
obs.tier is which of the three bet sizes the player posted; the multiplier rescales that match's outcome evidence by the stake.It works, in the sense that it fires. The shark who was invisible a moment ago now gets caught about 55% of the time. That feels like progress, and it is where a lot of real systems stop.
It shouldn't be. Two things are wrong with it, and one of them is not a matter of taste.
Multiply the terms of a log-likelihood ratio by an arbitrary weight and the result is no longer the log of any probability ratio. Wald's thresholds were derived for a quantity this isn't, so ±4.60 no longer buys a 1% error rate - it buys an unknown rate that depends on how the player bets.
That sounds like pedantry until you measure it. Below, "resolved" means the test reached a verdict either way within 400 matches, on players who are all honest. One of them is a whale - the casino word for a player who always bets the table maximum, honestly, because money isn't their constraint.
A minimum-bet loss is exactly what a shark throwing a match cheaply looks like. The wager-weighted score multiplies that match's evidence by 0.1, then by an outcome term that is itself near zero at tau*. It scores the single most suspicious event in the dataset as nothing at all.
The weighting was a patch bolted onto the old test. The real fix is to change what one observation is. Until now the detector saw one thing per match - won or lost - and the bet came in from the side as a multiplier. Instead, make the observation the pair: which of the three tiers they bet, and whether they won. Three tiers times two results gives six possible observations, and each hypothesis now has to put a probability on all six. That is what a joint distribution is - "joint" because it prices the bet and the outcome together, so it can express a relationship between them, which their separate frequencies cannot.
And the relationship is exactly where the two hypotheses disagree. Under H0 the stake can't know something the player doesn't, so bet and outcome are independent: the probability of any pair is just how often they post that tier, times how often they win. Under H1 the pair shares a hidden cause - the shark's private decision, made before betting, about whether this is a match they intend to win. That one decision pushes the bet and the result in the same direction, which is the correlation an honest player has no way to produce.
Writing it down takes three ingredients, each a short list of probabilities: pi, how often this player posts each tier when nothing is going on; phi_play, how a shark bets on matches they mean to win; phi_tank, how they bet on matches they mean to throw. For now every one of these is assumed - plausible values I wrote down by hand, which a later section replaces with fitted ones.
# Bernoulli(y ; p) just means: y is a win with probability p, a loss otherwise # H0: skill equals rating, so the bet says nothing about the result P(t, y | H0) = pi[t] * Bernoulli(y ; e) # H1: a private choice drives both the bet and the result P(t, y | H1) = tau * phi_tank[t] * Bernoulli(y ; 0.02) + (1-tau) * phi_play[t] * Bernoulli(y ; e_delta) # Both sum to 1 over the six cells, so the ratio is a real # likelihood ratio and the thresholds mean what they say again.
Below is the evidence each of the six observable events contributes, under this model and under the weighted score. The values are in the same units the running total accumulates - and the accuse line sits at 4.60, so an event worth +0.67 carries about a seventh of a conviction on its own.
In an even matchup against a shark hiding 400 Elo, a minimum-bet loss is worth +0.67 - the second most incriminating thing a player can do. The weighted score values that same event at 0.00.
And a minimum-bet win is worth −1.72, strongly exonerating, because sharks don't win the matches they staked nothing on. No multiplier can express that. A multiplier can shrink evidence toward zero; it can't flip its sign.
The model above needs pi, the player's distribution over bet tiers. The easy thing is to use the field's average. Do that and the detector catches 100% of sharks in a median of 13 matches, which looks like a triumph until you check who else it caught.
A whale - honest, just rich, always bets the table maximum - never makes the small bets the field constantly makes. Match after match they look slightly more like H1 than H0. A slight bias, accumulated sequentially, convicts 100% of them.
The fix is to stop assuming everyone bets alike and learn each player's pi from their own bets. Done naively, that breaks at both ends: a brand-new player has no history to learn from, and a patient shark can manufacture a convenient one. So the estimate is stabilized the standard Bayesian way - start every player with a handful of phantom matches whose bets follow the population average, then let their real matches gradually outvote the phantoms. The formal name for this is a Dirichlet prior; the arithmetic is one line:
# the bet profile the detector holds for one player, n matches into their career P(next bet = tier t) = (count[t] + strength · pop[t]) / (n + strength) # count[t]: how often THIS player has posted tier t so far # pop[t]: how often the whole field posts tier t # strength: how many phantom matches the population average is worth
Switch between players. Every detector runs on the same match stream, side by side. The tilter is the interesting honest case - a player who chases losses, betting bigger after a defeat. Their stakes do correlate with results, just with the previous one rather than the coming one - a real correlation in an innocent direction.
The population-profile detector and the per-player one behave identically on the shark. On the whale they diverge: one convicts, the other clears them decisively.
The number of phantom matches - the prior's strength - is now the thing to get right, and both extremes fail in opposite directions.
Trust each player's own history too much and their few early matches become their whole standard, so ordinary honest players get accused on noise - 12.5% of them at the far left. Trust the population too much and you are back to judging the whale by other people's habits, up to 53% at the far right. Everything from about 4 to 16 holds sharks at 100% with neither failure mode.
That is a comfortable window, and it is also a number somebody chose by looking at a chart.
Everything so far fixes H1 at one specific shark: 400 Elo hidden, throwing 40% of matches. Point that detector at somebody who hides 400 Elo and throws none - the crude version from the first widget - and it catches only 86% of them, because their bets carry no trace of the tanking the hypothesis is looking for.
The outcome-only test catches that player every time. So the two detectors have complementary blind spots, which is an uncomfortable place to leave a system whose whole job is not having blind spots.
Stop assuming one shark. Make H1 a mixture over a grid of them, and average:
# "Some kind of shark, weights w over the kinds" - average the fifteen # six-cell tables. The weights sum to 1 and each table sums to 1, so the # average still sums to 1: the ratio remains a genuine likelihood ratio. P(t, y | H1) = sum_k w[k] * P(t, y | H1 ; delta[k], tau[k])
That closes the hole. Both shark styles are now caught at 100%, false positives are unchanged, and on the blatant shark the mixture is four times faster than the outcome test - a median of 13 matches against 56 - because it gets to use the bets as well. It costs a little sharpness on the awkward middle cases, which the table below reports.
The tank rates, the hidden-skill grid, what a shark's bets look like when they mean to win, the prior strength I picked off a chart two sections ago - all chosen by hand, all defensible, none of them measured.
They don't have to be, because all of them can be learned from the population's match records - with no labels, since nobody knows who the cheaters are. The trick is to write a model of how the whole season's data came to be, with every number I was choosing by hand promoted to an unknown:
Each player is secretly one of two kinds, decided by a weighted coin flip. The weight - the fraction of the population that is cheating, prevalence - is unknown. An honest player gets personal betting habits pi drawn from around a population average (unknown), held together by a phantom-match strength (unknown - the parameter I was setting by eye two sections ago), and their matches follow the honest half of the six-cell model. A shark instead draws a strategy - how much hidden skill, what tank rate - from the fifteen-cell grid with unknown weights, plus the betting habits phi_play and phi_tank (both unknown), and their matches follow the shark half.
Nothing per-player survives into the fitting. The secret coin and the personal habits would be thousands of parameters, but both can be summed away exactly. The coin has only two outcomes, so each player contributes one quantity: the probability I am honest and produced this record, plus the probability I am a shark and produced it. And averaging over every possible betting habit consistent with a player's bet counts has a closed form. What remains is roughly twenty numbers describing the whole population.
Those twenty are fitted by MCMC - a guided random walk through parameter settings that lingers wherever the data is well explained, so time spent at a setting is proportional to how plausible it is (the specific walker is NUTS, PyMC's default). What comes back is therefore a posterior rather than one best answer: a cloud of parameter settings, each weighted by how consistent it is with everything observed. The width of the cloud measures how much the data pins down.
# sharkhunt/hierarchical.py - the story above, as PyMC sees it z[p] ~ Bernoulli(prevalence) # the secret coin: shark or not pi[p] ~ Dirichlet(strength * pop_profile) # honest habits, near the average # neither z nor pi is ever sampled - each is summed out exactly, # leaving one term per player that covers both possible identities: pm.Potential("population", logsumexp( [log1p(-prevalence) + ll_honest, # "I'm honest and produced this record" log(prevalence) + ll_shark], # "I'm a shark and produced this record" axis=1).sum())
Nobody tells this model what a shark looks like, and it gets no labels, confirmed cases, or examples - its only foothold is structural: one subpopulation's bets are independent of their results, and another's are not. From that alone it has to discover, in an undifferentiated pile of match records, that the second group stakes heavily on the games it means to win and minimally on the ones it means to throw.
The bars below are posterior means - the center of the fitted cloud - drawn over the true values the simulation was run with. Being able to check against the answer key is the one luxury of synthetic data.
The prior strength is the one to check against yourself. I picked 4 to 16 off a chart a few sections ago; the posterior puts it at 3.1, just outside the window I'd talked myself into. Not a dramatic miss, but it is the kind of thing that stays wrong forever when the number is a constant in a config file rather than a parameter with data behind it.
A new title has fifty players and three weeks of history. Fit the model there from scratch and something worse than imprecision happens.
An unlabeled mixture doesn't know which of its two components is supposed to be the cheaters. Nothing in the likelihood says "the shark group is the guilty one," so the fit can just as well decide that the cautious players are the anomalous subpopulation and score everyone accordingly. With a large, rich dataset the honest explanation wins on sheer weight of evidence. With a small one, it often doesn't.
Below, the new game has a deliberately different crowd - far more cautious, almost no whales - subtler sharks who hide less skill and let their bets track their intent only loosely, and just 40 matches per player. The score is AUC: draw one shark and one honest player at random, and ask how often the model ranks the shark as the more suspicious of the two. 1.0 is a perfect ranking, 0.5 is a coin flip - and below 0.5 the ranking is backwards. The prior carried over from the established title is doing something more fundamental than adding data.
The cold-start line thrashes. At two of the four sizes it lands below a coin flip - it found the mirror-image answer and ranked the cheaters as the safest players on the platform. More players don't fix it, because two different explanations fit the same data about equally well and nothing tells the sampler which one we meant.
The transferred prior settles that by carrying over the statement this is what a cheat looks like from a game where we had enough evidence to learn it. The local data then adjusts the details.
Transfer isn't the only way to break that symmetry - an ordering constraint on the components would do it, and so would a handful of confirmed cases to anchor the labels. It is the one that doesn't require you to already have caught somebody.
A shark reading this far knows the counter. The detector keys on the correlation between stake and outcome, so break the correlation: bet the same way regardless of intent. The tell disappears and detection falls to the false-positive floor.
It also means posting full stake on the matches they are about to throw on purpose. And tanking is not optional - it is the whole mechanism keeping them in the soft bracket. At the pinning rate they must throw 46% of their matches. Do that at full stake and the arithmetic stops working.
Fully correlated, the shark clears +0.223 per match and is caught every time. Fully decorrelated they are invisible - and make −0.017, against the −0.022 an honest player pays to the house cut. They have spent a great deal of effort to earn half a percent of a stake more than someone who isn't cheating at all.
Bet discipline isn't their only option. They could instead hide less skill - sit 100 Elo above their rating rather than 400 - and let chance do the concealing for them. It works: at 100 Elo of hidden skill they win only 64% of the matches they try to win, so a big bet that loses stops being surprising, and the evidence per match gets much weaker.
Hiding less buys time but not immunity. It stretches survival by about two and a half times - and costs roughly two thirds of the edge to do it. Below about 200 Elo of concealed skill the arithmetic has already stopped being worth the trouble, and that is before counting the effort of maintaining a second identity and throwing every other match on purpose.
So both evasions are priced. No setting on either curve is both profitable and safe. The deliverable is a tax that scales with how hard you press.
The wager evidence resolves a case in a median of fourteen matches. That is far quicker than a rating converges, and it is not zero - which leaves the shark one obvious move. Burn the account and open a new one. New accounts start in the middle of the distribution because nothing in the system has any reason to think otherwise, and the fourteen-match clock restarts in a fresh set of beginner lobbies.
I don't think that's unfixable, and it isn't fixable with anything on this page. It needs a different kind of evidence: how the player moved, not what happened in the match - visible immediately, and thrown away by a rating system.
That turns out to be its own project, with its own uncomfortable questions, so it gets its own writeup rather than a hurried ending here. The short version: you can train reinforcement-learning agents to a measured ladder of skill before your game has any players at all, and use them as the labeled data a behavioral classifier needs. Whether an agent's blunders look anything like a beginner's is exactly the part worth being skeptical about, and that's the experiment.
You need wager data joined to match data, which most skill-gaming platforms already have and most traditional games don't. Without stakes attached to matches, none of this applies - the whole method is built on a number that exists only when money is on the table.
You need somewhere to put the accusations. A detector that produces a ranked list and no review process is worse than no detector, because it launders a statistical judgment into an operational one.
The population fit is offline: minutes of MCMC over the whole player base, re-run occasionally. The serving path is a running sum per player per match, which costs nothing. The expensive part is not compute, it's the review queue and the honesty to treat scoring your own players as the surveillance question it is.
A shark can control what the scoreboard says about them, but not what they came for. Aim the test at the stake and their discipline stops helping.
Shark hunt · Orbitope