all AI news
Lessons after a half-billion GPT tokens
Simon Willison's Weblog simonwillison.net
Lessons after a half-billion GPT tokens
Ken Kantzer presents some hard-won experience from shipping real features on top of OpenAI's models.
They ended up settling on a very basic abstraction over the chat API - mainly to handle automatic retries on a 500 error. No complex wrappers, not even JSON mode or function calling or system prompts.
Rather than counting tokens they estimate tokens as 3 times the length in characters, which works well enough.
One challenge they highlight for …
abstraction ai api basic billion chat error experience features function generativeai gpt json llms openai promptengineering shipping tokens