FineWeb 45TB Dataset: $500k GPU costs and Adult Content Improving LLM Quality

This week Hugging Face has released what seems to be the largest (15 trillion tokens) open dataset specifically created for LLM training: FineWeb.

It is based on internet crawls between the Summer of 2013 and the Winter 2024. The 15T size of the dataset resonates with the Llama 3 release that happened just a week before - it was trained with a 15T dataset as well (versus 2T used with Llama 2 series). The leap in the amount of …

ai aiops costs dataset face genai gpu hugging face improving internet llama llm machinelearning quality tokens training