all AI news
KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation. (arXiv:2205.09921v1 [cs.CL])
May 23, 2022, 1:12 a.m. | Ta-Chung Chi, Ting-Han Fan, Peter J. Ramadge, Alexander I. Rudnicky
cs.CL updates on arXiv.org arxiv.org
Relative positional embeddings (RPE) have received considerable attention
since RPEs effectively model the relative distance among tokens and enable
length extrapolation. We propose KERPLE, a framework that generalizes relative
position embedding for extrapolation by kernelizing positional differences. We
achieve this goal using conditionally positive definite (CPD) kernels, a class
of functions known for generalizing distance metrics. To maintain the inner
product interpretation of self-attention, we show that a CPD kernel can be
transformed into a PD kernel by adding a …
More from arxiv.org / cs.CL updates on arXiv.org
Jobs in AI, ML, Big Data
Senior ML Researcher - 3D Geometry Processing | 3D Shape Generation | 3D Mesh Data
@ Promaton | Europe
Cleared Senior Software Engineer, Computer Vision, Federal
@ CCRi | Chantilly, Virginia, United States
Data Analyst - B2C
@ DAZN | Hyderabad, India
Product Marketing Manager - AI Chatbot
@ SendBird | San Mateo, California, United States
Alternance Alternant Ingénieur Développement logiciel temps réel embarqué / computer vision (F/H)
@ Alstom | Villeurbanne, FR
AOT Data Analyst II - Highway Project Delivery
@ State of Vermont | Barre, VT, US