Questions about GPT-1 in Huggingface.

Hi all, currently I'm learning about LLMs and I have a couple of noob questions.



First, let's start with the GPT-1 paper: https://cdn.openai.com/research-covers/language-unsupervised/language\_understanding\_paper.pdf



**Question 1: What is exactly the shape of the input to the embedding step?**



Let's look at expression block (2) in the paper.



According to the paper the input to the model is named **U**. From what I have gathered so far, these should be token\_id's after a first tokenization step. However I am a bit confused …

block embedding gpt gpt-1 huggingface languagetechnology llms look paper question questions