marginalisation of variables and rao-blackwell

Adapted from pymc example.

Marginalising out variables from a model results in lower variance estimates of parameters in the model through Rao-Blackwell’s theorem.

Samplers approximate the expectation for some function with respect to a distribution . By law of total expectation we know that

Letting , we know by law of total variance that

Because the expectation is over a variance it must always be positive, and thus we know

Marginalising variables () in your model lets you use instead of . This lower variance manifests most directly in lower Monte-Carlo standard error, and indirectly in a generally higher effective sample size.