Break it to grok it: The best way to understand how a method works is go construct scenarios where it fails
Scheduled to appear in May on blog; posting here now, just for fun
Someone who is working on a statistical problem in political science writes:
I came across an algorithm . . . [follows with description of some classical and Bayesian approaches that use this algorithm] . . . Now the results I have gotten from these models seem very accurate. Is there a theoretical basis for why this model should or shouldn't work?
I'm skipping the details so as to emphasize the general nature of my advice on this sort of problem.
Here's how I replied: I am sure you can construct an example where the method under discussion gives bad estimates. That does not imply that it is a bad method, just that there will be conditions under which it does not work. If you can understand the conditions where the method fails, this should help you better understand the method. So try to figure out where the method works and where it doesn’t, get some sense of the boundary between these zones in problem-space, and that just give you some insight into the method as well as a sense of where to apply it.
Also remember that a method that fails isn’t so bad. What you really want to avoid is a method that fails without you realizing that it’s failing.