More on 4th downs

I've been thinking more about 4th downs and how I could test my hypothesis that Belichick goes for it on 4th down because his job is secure, and other coaches do not go for it because their jobs are not as secure. The most enlightening paper I have found on 4th down decisions is David Romer's paper (pdf) which analyzes whether coaches make win maximizing decisions in the 1st quarter of games. The paper uses standard Macroeconomic methods, I never thought they could be useful, but apparently they are. The conclusions are worth a read even if you don't read through the model.

The paper hypothesizes several reasons for the departure from win-maximization, but does not test any of them. As a refresher on my hypothesis, I supposed that coaches who are concerned about job security do not go for it on 4th down as much as they should because it is an unconventional move. In terms of job security, an anti-herd decision is much higher risk than the typical herd decision. If they make a conventional decision, even if it fails, their job security does not change much. If they make an unconventional decision, and it fails, then their job security is likely to decrease significantly. The expected benefits from the anti-herd decision are minimal (Romer estimates win-maximizing decisions on 4th down would result in one additional win every three years). The expectation of one additional victory every three years is nothing compared to the expected several instances a year in which the anti-herd decision fails and results in a loss. When this happens, the fans, media, and owner can look back and say what if we had a conventional coach without considering the fact that the decision actually was win-maximizing.

To test my hypothesis empirically, I considered analyzing anti-herd win-maximizing choices. It might be that coaches who make win-maximizing anti-herd decisions end up being better coaches because they are better at win-maximizing. Alternatively, it could be that good coaches have good job security, and then feel more comfortable making anti-herd decisions. To test this, one could compare the correlation between the propensity to make anti-herd decisions and coaching performance strictly after the anti-herd decision with the correlation between the propensity for anti-herd decisions and coaching performance strictly before the anti-herd decision. If coaches are good because they make anti-herd win-maximizing decisions, we would expect these correlations to be the same. If coaches make anti-herd win-maximizing decisions because they have good past performance and hence good job security, then we would expect the second correlation to be higher.

I would be very eager to start this research if not for several issues which would likely cloud any results. The first is that there is a relatively small sample of anti-herd win-maximizing decisions. In Romer's data covering three years of play, of the 1068 times going for it on 4th down would have been win-maximizing in the first quarter, only 109 times was the win-maximizing anti-herd decision made. Another confounding problem would be if coaches have consistent performance over time then strictly past performance and strictly future performance would be correlated and weaken any difference seen in the correlations we want to compare. Additionally, there are other reasons a coach might make an anti-herd decision, for example maybe coaches who have very little job security will become risk loving in job security in an effort to raise their job security above some threshold and begin to make unconventional moves purely because they are unconventional. This last problem might be caught in my proposed comparison of correlations though because it is based on past job performance and independent of future job performance.

Romer suggests two alternatives for how coaching decisions will evolve. The first is where they stay the same, because of some preferences which are not based solely on win-maximization. However, if coaches don't win-maximize just because they are imperfect optimizers, then 4th down decisions should evolve towards win-maximizing decisions through trial and error, use of statistical modeling, etc. I think I have seen this happening over the last 10 years anyway (and Bill Belicheck is often the one being emulated). A couple years ago he intentionally took a safety when it was a win-maximizing decision. Since then a couple coaches have emulated it, when they probably never would have considered it before. If these anti-herd decisions become more popular, then perhaps the data will become robust enough to test my hypothesis.

One last interesting part of Romer's paper was when he asks how could it be that NFL coaches who are paid millions of dollars in a very competitive market are imperfect optimizers. He concludes that there are many other things that are much more important for an NFL coach. After all, under proper optimization if a coach can only expect one more win every three years, then that effect is not an important one. As an example, I think I would be far better than many NFL coaches (including Andy Reid) at making most gameday decisions. However, I'm pretty sure I would not be nearly as good at motivating the players or running a practice, and those things are more important than one win every three years.

I'd like to embark on leisure research of the NFL at some point, so attempting this would probably be a good start. Even if I don't get results, at least I'll have a good dataset to do future research. None of this is likely to happen before I pass prelims, though.