The Implicit Bias of Heterogeneity towards Invariance and Causality

arXiv:2403.01420v1 Announce Type: new
Abstract: It is observed empirically that the large language models (LLM), trained with a variant of regression loss using numerous corpus from the Internet, can unveil causal associations to some extent. This is contrary to the traditional wisdom that ``association is not causation'' and the paradigm of traditional causal inference in which prior causal knowledge should be carefully incorporated into the design of methods. It is a mystery why causality, in a higher layer of understanding, …

abstract arxiv association bias causality causation cs.lg internet language language models large language large language models llm loss math.oc paradigm regression type