Stream LLM Responses from Cache

LLMs can become more expensive as your app consumes more tokens. Portkey's AI gateway allows you to cache LLM responses and serve users from the cache to save costs. Here's the best part: now, with streams enabled.

Streams are an efficient way to work with large responses because:

• They reduce the perceived latency when users are using your app.


• Your app doesn't have to buffer it in the memory.

Let's check out how to get cached responses to your app …

app become cache costs latency llm llms part reduce responses save serve tokens work