How to identify OpenAI's crawler bot to stop it slurping websites for training data

Aww, c'mon, let us scrape your pages, we've got billions at stake

OpenAI, the maker of machine learning models trained on public web data, has published the specifications for its web crawler so that publishers and site owners can opt out of having their content scraped.…

bot crawler data identify machine machine learning machine learning models openai public publishers training training data web web crawler websites