What a weak learner is
Why this site is named after something that's barely better than a coin flip.
Picture a hundred people guessing tomorrow's weather. None of them is any good. Each one is right a little more than half the time. Ask any single one, and you might as well flip a coin.
Now take a vote. If their mistakes are different mistakes, the wrong guesses scatter. The right ones pile up. The crowd is right far more often than anyone in it.
That's the whole trick. In machine learning, each of those people is called a weak learner: a rule that does just a bit better than chance. The classic one is a decision stump. It asks one question, like "is this point left of this line?", and answers yes or no.
A vote only works if the mistakes differ. Boosting makes them differ on purpose.
How boosting works
Start with one stump. It gets some examples wrong. Before you pick the next stump, make those missed examples count for more. Now the next stump cares most about exactly what the first one missed.
Repeat that a few hundred times. Every new stump is hired to fix the team's current worst mistakes.
Each stump also gets a vote, sized by how good it was. A stump that's right 51% of the time gets almost no say. One that's right 90% of the time gets a lot. In AdaBoost, the best-known version, the size of the vote is:
vote = ½ · ln( (1 − error) / error )
error 49% → vote 0.02
error 10% → vote 1.10
That's 55 times the say for the better stump. Nobody is ignored. Everyone is weighed.
Watch it happen
One of the three drawings on the front page is this, running live. Every faint straight line is one stump. No single line can draw a letter. All any of them knows is whether a point sits inside my name. Stacked up a few hundred deep, they draw it.
The catch
Boosting chases whatever it gets wrong. That's its strength, and its danger. If some examples are just noise, boosting will work hard to explain the noise. It will find patterns that aren't there.
I know this from my own data. A pattern I was sure of turned out to be noise, and ten games were enough to show it. That's the next thing I wrote about.
Why I named the site after it
Most of my ideas start weak. A hunch about which map I win on. A guess about which hour I play best. On its own, each one is barely better than a coin flip. Some get stronger as the evidence piles up. Most don't.
The name is a reminder to keep score, and to let the score decide.