[P] We've unified LLMs w/ vector memory + reranking & pruning models in a single process for better performance

There is a lot of latency involved shuffling data for modern/complex ML systems in production. In our experience these costs dominate end-to-end user experienced latency, rather than actual model or ANN algorithms, which unfortunately limits what is achievable for interactive applications.

We've extended Postgres w/ open source models from Huggingface, as well as vector search, and classical ML algos, so that everything can happen in the same process. It's significantly faster and cheaper, which leaves a large latency budget available …

algorithms ann applications costs data end user experience interactive latency llms machinelearning memory performance process production pruning systems vector