PAPERCLIP: Associating Astronomical Observations and Natural Language with Multi-Modal Models

arXiv:2403.08851v1 Announce Type: cross
Abstract: We present PAPERCLIP (Proposal Abstracts Provide an Effective Representation for Contrastive Language-Image Pre-training), a method which associates astronomical observations imaged by telescopes with natural language using a neural network model. The model is fine-tuned from a pre-trained Contrastive Language-Image Pre-training (CLIP) model using successful observing proposal abstracts and corresponding downstream observations, with the abstracts optionally summarized via guided generation using large language models (LLMs). Using observations from the Hubble Space Telescope (HST) as an example, …

abstract arxiv astro-ph.im clip cs.cl cs.cv cs.ir cs.lg image language modal multi-modal natural natural language network neural network pre-training representation telescopes training type