all AI news
A look at robots.txt, a good will-based social contract governing the behavior of web crawlers, as experts call for more rigid tools for managing AI crawlers (David Pierce/The Verge)
Feb. 15, 2024, 9:25 a.m. |
Techmeme www.techmeme.com
David Pierce / The Verge:
A look at robots.txt, a good will-based social contract governing the behavior of web crawlers, as experts call for more rigid tools for managing AI crawlers — For decades, robots.txt governed the behavior of web crawlers. But as unscrupulous AI companies seek out more and more data …
behavior call david experts good look robots social tools web will
More from www.techmeme.com / Techmeme
Jobs in AI, ML, Big Data
Founding AI Engineer, Agents
@ Occam AI | New York
AI Engineer Intern, Agents
@ Occam AI | US
AI Research Scientist
@ Vara | Berlin, Germany and Remote
Data Architect
@ University of Texas at Austin | Austin, TX
Data ETL Engineer
@ University of Texas at Austin | Austin, TX
Machine Learning Engineer - Sr. Consultant level
@ Visa | Bellevue, WA, United States