all AI news
Disallow GPT Bot from Scraping our Blog Posts
Aug. 8, 2023, 7:49 p.m. | Ervin Szilagyi
DEV Community dev.to
Lately, we can bloc GPT bots from scraping our pages for a site that we control, by setting the following lines in the robots.txt file:
User-agent: GPTBot
Disallow: /
I, myself, found out this from a tweet from Gergely Orosz:
My stance on this is similar to what Gergely is saying. GPT offers no citation to the information it provides. While I did update the robots.txt file on my personal website, I am also cross-posting to DEV. If we …
blog bot bots control discuss found gpt gpt3 gptbot openai robots scraping tweet watercooler
More from dev.to / DEV Community
Jobs in AI, ML, Big Data
Lead Developer (AI)
@ Cere Network | San Francisco, US
Research Engineer
@ Allora Labs | Remote
Ecosystem Manager
@ Allora Labs | Remote
Founding AI Engineer, Agents
@ Occam AI | New York
AI Engineer Intern, Agents
@ Occam AI | US
AI Research Scientist
@ Vara | Berlin, Germany and Remote