[D] Are black-box explainers (e.g. SHAP) just another black-box?

Tools such as SHAP are now very common to explain "black box" models such as GBMs or Neural Networks. However, people who use them often don't really understand how they work. And these tools are no less complex than the models they would like to explain. Aren't we just explaining "black box" models with black box explainers?

machinelearning shap