all AI news
GPT-2 from scratch (PyTorch) | Video Tutorial
Web: https://www.reddit.com/r/LanguageTechnology/comments/sh5vlm/gpt2_from_scratch_pytorch_video_tutorial/
Jan. 31, 2022, 4:53 p.m. | /u/mildlyoverfitted
Natural Language Processing reddit.com
Hey everybody,
I created a video tutorial on GPT-2 and how to implement it "from scratch".
It only focuses on the inference (training is not discussed). It covers concepts like masked self attention, decoder blocks and generating new tokens. It is heavily based on minGPT (see link below).
Hope some of you could find it useful!
Relevant links:
- Paper: https://openai.com/blog/better-language-models/
- Code minGPT: https://github.com/karpathy/minGPT
- Code transformers: https://github.com/huggingface/transformers/blob/0f69b924fbda6a442d721b10ece38ccfc6b67275/src/transformers/models/gpt2/modeling_gpt2.py#L946
[link] [comments]
More from reddit.com / Natural Language Processing
Top2Vec topic modelling and semantic search
1 day, 5 hours ago |
reddit.com
Parallel Indic language corpus between Hindi and Bengali.
2 days, 17 hours ago |
reddit.com
Latest AI/ML/Big Data Jobs
Engineering Manager, Machine Learning (Credit Engineering)
@ Affirm | Remote Poland
Sr Data Engineer
@ Rappi | [CO] Bogotá
Senior Analytics Engineer
@ GetGround | Porto
Senior Staff Software Engineer, Data Engineering
@ Galileo, Inc. | New York City or Remote
Data Engineer
@ Atlassian | Bengaluru, India
Data Engineer | Hybrid (Pune)
@ Velotio | Pune, Maharashtra, India