Feb. 14, 2024, 2 p.m. | David Pierce

The Verge - All Posts www.theverge.com


Illustration by Erik Carter

For decades, robots.txt governed the behavior of web crawlers. But as unscrupulous AI companies seek out more and more data, the basic social contract of the web is falling apart. 



Continue reading…

ai companies basic behavior carter companies data file illustration internet reading robots social text web

Lead Developer (AI)

@ Cere Network | San Francisco, US

Research Engineer

@ Allora Labs | Remote

Ecosystem Manager

@ Allora Labs | Remote

Founding AI Engineer, Agents

@ Occam AI | New York

AI Engineer Intern, Agents

@ Occam AI | US

AI Research Scientist

@ Vara | Berlin, Germany and Remote