Towards Causal Foundation Model: on Duality between Causal Inference and Attention

arXiv:2310.00809v2 Announce Type: replace
Abstract: Foundation models have brought changes to the landscape of machine learning, demonstrating sparks of human-level intelligence across a diverse array of tasks. However, a gap persists in complex tasks such as causal inference, primarily due to challenges associated with intricate reasoning steps and high numerical precision requirements. In this work, we take a first step towards building causally-aware foundation models for complex tasks. We propose a novel, theoretically sound method called Causal Inference with Attention …

abstract array arxiv attention causal causal inference challenges cs.ai cs.lg diverse foundation foundation model gap however human inference intelligence landscape machine machine learning reasoning stat.me stat.ml tasks type