What is the backtest and how should I use it?

I had a great question from one of our users:

‍
Is the backtest how the model would have performed over past seasons? Also, how much do backtest stats tend to differ from the real-world performance?

Here's my answer:

‍
When a model is built, about 2000 games are held out of the training set in order to test its performance. After the model is trained, those 2000 games' closing lines are used to make predictions to simulate what the model would have said. From those predictions, all games with a positive expected value (edge) are graded against the games' actual results. The backtest rate is the percentage of those predictions with a positive edge win.

We have found that the backtest rate is correlated with use in practice, but it's not exact; a model with a lower backtest rate sometimes outperforms a higher one in actual use. This is primarily because you don't bet on *all* games with a positive edge. By being selective, you can bet on the games where your models are more reliable. Some users have found that by being selective the lower backtesting models outperform the others.

The best way to understand your models is to track predictions before placing money bets. You'll quickly get a sense of where they work best and where you should be cautious.

Have fun! Let us know how it goes!

‍