Observability for LLMs // Phillip Carter // LLMs in Production Conference Lightning Talk 4

// Abstract
Making LLMs reliable is hard. You can't debug or unit test them, not in the traditional sense at least. Instead, you'll need to turn to the practice of Observability, by instrumenting your feature to produce rich telemetry and analyzing behavior from that data. Observability can also act as a key source of data for evaluations.

// Bio
Phillip is on the product team at Honeycomb where he leads their AI initiatives and works on a bunch of different …

abstract behavior carter conference data debug feature least lightning llms making observability practice production sense talk telemetry test them