To test a financial model that makes any kind of predictions, it is clearly impractical to enter the currently available data and then wait to see how well forecasts are met - particularly as it will be necessary to do this many times in order to obtain a statistically meaningful measure of the accuracy of forecasts.
The solution is to test the model by using it on only the data available at some past date, and then comparing the predictions to what happened subsequently. This is back testing.
For example, suppose we estimate the volatility of securities based only on their past price movements (this type of model is often used for risk management). The way we would test this would be, in outline, something like this:
- Decide on a time period, a pool of securities, the length of time (call it t) over which to take data over.
- Pick a time at random (call it t1), within the time period.
- Pick a security at random from the pool.
- Feed the price data for the security for the period t1 to t1 + t into the model.
- Compare the resulting estimate with the actual volatility over a period after t1 + t.
- Go back to step two.
The errors in the estimates can then be used, using standard statistical techniques, to estimate the reliability of the estimates.
Back testing will usually be computerised rather than carried out entirely by hand. The greatest difficulties are avoding model risk, especially the inadvertant inclusion of data that was not actually available at the time.