Speed and Sensibility: Balancing Latency and UX in Generative AI // Julia Kroll // LLMs III LT

// Abstract
Conversational AI demands low latency for a seamless dialogue between humans and AI. However, engineers face the dilemma that some latency is inherently required in order to process human speech and craft a response. Some incremental wins to shave off milliseconds involve trade-offs against how the AI response could be enriched during the additional processing time. Others simply refactor out inefficiency to obtain more performant results from AI devtools. This talk presents best practices of designing streaming speech-to-text …

abstract conversational conversational ai dialogue engineers face generative human humans iii incremental julia latency llms low process speech speed