[D] Product evaluations is one of the most under-discussed topics

We are an AI consultancy and this happens with us over and over again...

We start a new LLM project with a client.

Their engineers get to 80% super quickly.

They have a lot of edge cases and want us to finish up the remaining 20%.

We ask them about evals.

And of course they don't have them.

We create the eval framework, improve the pipeline iteratively, and voila.

Job done, everybody is happy.

I seriously think that based on …

ai consultancy cases client consultancy edge engineers llm machinelearning product project them topics