Common Pitfalls When Explaining AI and Why Mechanistic Explanation Is a Hard Problem.

Recently researchers have started using explainability techniques for several different applications—to help foresee how a model might operate in the field, to persuade others to trust a model, and to assist with debugging errors. A large number of explainability techniques have been published with very little empirical testing to see how useful they actually are for each of these use cases. We discuss several pitfalls one can encounter when trying to utilize explainability techniques. We then discuss how recent work on the double descent phenomena and non-robust features indicate that mechanistic explanation of deep neural networks will be very challenging for most real-world applications. In some cases, one may be able to use an easily interpretable model, but for many applications deep neural networks will be more accurate. In light of this, we suggest more focus should be given to implementing out-of-distribution detection methods to detect when a model is extrapolating and thus is likely to fail. These methods can be used in lieu of explainability techniques for increasing trust and debugging errors.