Wednesday, August 26, 2009

The Bayesian information criterion (BIC) doesn't make sense to me

For fitting a model, several different criteria can be used. The first is the Aikake information criterion (AIC), which is basically -2*loglikelihood + 2*# parameters. So if you add a parameter to the model, it penalizes the AIC by 2, so you would need a commensurate decrease in -2*loglikelihood (so basically the loglikelihood would have to increase by at least 1) to make it worth adding.

The BIC penalizes the -2*loglikelihood by # parameters*log(sample size). So for large sample sizes, to add a parameter to the model you would have to improve the loglikelihood by a lot more. But, doesn't a richer sample allow you to explore more parameters for the model? So in the case where you are able to explore more parameters, the BIC forces you to use fewer. Doesn't make a lot of sense to me, although the loglikelihood does take on a larger range of values with a larger sample as it sums over the sample.