Abuse, spam and fraud signals
Finding the bad actors in the traffic without blocking the good ones.
01The intuition
The problem it solves
Gmail receives billions of messages a day, and most of them are unwanted. A payments app sees a few thousand transactions a minute, and a handful are stolen cards. A social site gets a million sign-ups a week, and some are bots. In each case the bad traffic looks almost like the good traffic, arrives mixed in with it, and adapts when you block it.
The two mistakes have different costs. Letting spam through annoys a user. Blocking a real email from a hospital, or declining a real customer's card, is far worse, and a system tuned to catch everything will do that often. So the problem is not "find the bad traffic" but "find most of it while almost never blocking a real person".
Build it one piece at a time
Step 1: rules on single facts. Start with lists: a sender domain known for spam, an IP range that only ever sends junk, a card number reported stolen. A lookup in a set, checked at the edge. Cheap and exact, but it only catches what you have already seen.
Step 2: signals, not verdicts. Instead of one rule that blocks, compute many small facts about each event and give each a weight: the account is 2 minutes old (+3), the message has 40 links (+2), the sender has been replied to by this user before (−5), the card's country matches the device's country (−1). Add them up into a score. No single signal decides, which is what makes the system hard to game.
| Event | Signals | Score | Action |
|---|---|---|---|
| email A | new sender +3, 40 links +2, no reply history +1 | 6 | spam folder |
| email B | replied before −5, 1 link 0 | −5 | inbox |
| payment C | new device +2, amount 10× usual +3, foreign country +2 | 7 | ask for a second check |
Step 3: rate and velocity signals. Many abusers are only visible over time: 500 sign-ups from one IP in an hour, one card tried at 30 shops in a minute. Keep counters per IP, per card, per account over sliding windows, using the same machinery as rate limiting, and feed the counts in as signals.
Step 4: three outcomes, not two. Scores in the middle do not get a verdict; they get friction. Ask the sender to solve a challenge, send the payment to a second factor, hold the post for review. Friction stops bots, costs a real user a few seconds, and gives you a labelled example either way.
Step 5: learn the weights. Users mark spam and not-spam, banks report chargebacks weeks later, reviewers label held posts. Those labels train a model that sets the weights, retrained daily, because the attackers change what they send once they learn the current rules. Keep the hand-written rules too, as a floor the model cannot fall below.
Why it works, with the numbers
A classifier is judged by two rates: the fraction of bad events it catches (recall) and the fraction of its blocks that were right (precision). Because good traffic vastly outnumbers bad, a small false-positive rate still blocks many real people:
1,000,000 emails/day, 1% spam; a 0.1% false-positive rate blocks 0.001 × 990,000 ≈ 990 real emails/day, nearly as many as the 10,000 spam it catches
which is why the middle band and friction exist: a wrong "challenge" costs seconds, a wrong "block" costs trust. The guarantee is statistical and stated as a target, for example "block under 0.05 percent of legitimate messages". The honest failure modes: adversaries adapt, so yesterday's precision is not today's; signals leak (a block that fires instantly tells the attacker which rule they hit, so some systems delay or randomise responses); and the labels arrive late, chargebacks by weeks, so the model is always trained on an old picture.
The trade-off you must say out loud
Every tightening catches more abuse and hurts more real users; pick the point on that curve for the product and say why a bank and a comment box choose differently. The interviewer will ask how you avoid training on your own mistakes, which is sampling some events that were allowed and some that were blocked for human review, and how the pipeline keeps up when a botnet sends a million events in a minute, which is scoring at the edge with cached signals.
Where it shows up in the 100 Google problems
- Design Gmail: scoring at receipt with sender reputation, content signals and user feedback, with the spam folder as the middle band.
- Design Google Pay: velocity signals per card and device, a step-up check for mid scores, and chargebacks as delayed labels.
- Fraud and DDoS protection for a social service: counters at the edge, challenges for suspect clients, and a denylist as the exact-match floor.