Why I built it
My younger brother and I have played Othello for years and I wanted a program that could beat us both.
An Othello engine that learned from 700,000 games against itself. By the end of the summer it beat my brother and me.
My younger brother and I have played Othello for years and I wanted a program that could beat us both.
A model that taught itself Othello by playing against itself and adjusting based on which moves led to wins. It has never seen a human game and has no opening book.
It runs in the browser, so there is nothing to install.
A neural network guides a tree search, and the games it plays against itself become the training data for the next round. One round has three steps:
The model plays full games against itself.
A tree search picks each move, guided by the network's guess of which moves are good.
The network learns from those games.
Which moves were played and who won become the training data.
The new model is tested.
It plays earlier versions of itself, and I play it in the browser.
Then it starts again. I ran that loop for about 49 hours on one rented T4.
The network is a program with about 3 million numbers that get adjusted during training. Given a board, it makes two guesses: which moves look good, and which side is likely to win. Before each move, a search plays out possible continuations and spends more of its time on the moves the network likes.
After each game, the network is adjusted so its guesses move closer to the moves the search chose and to the real winner. A little randomness is added to its choices so it keeps trying new moves. A board that is rotated or mirrored is the same position, so every position is saved in all eight orientations. I watched training live in Weights and Biases.
At two and a half seconds a game, a full run would have taken weeks, and my laptop could not stay on that long, so I rented a T4. Profiling put the time in move generation and tree search. I vectorized the tree search into a batched engine, batched GPU inference across games, added multiprocessing, and moved move generation and search into C++ behind pybind11. Self-play went from 0.4 to 4.0 games per second.
A rewrite in another language can introduce bugs that never show up as crashes, so I ran the C++ engine against the Python one and required bit-identical output.
Every round, the new model played the previous model and won more often, so every chart I had said things were improving. Then I sat down and played it with an ordinary strategy and beat it.
Each generation was getting good at punishing the specific mistakes the last generation made, and none of that transfers to an opponent who makes different mistakes. So the self-play win rate went up without the model getting stronger.
I changed two things. I started checkpointing partway through training and judging each checkpoint on how it actually played, and I built the web app so I could play it myself. Once the failure showed up in hour two, the fix was a settings change: push it harder toward moves it had not tried.
I handed it to my brother and found out that training speed and inference speed are separate problems. On a normal PC it took about 30 seconds to pick a move. Quantizing and distilling it into a smaller model brought that under 5 seconds, and that version is the one deployed. It beats both of us.