[D] Best Practices for Semantic Search on 200k vectors (30GB) Worth of Embeddings?

Hi, I have converted some domain-specific name vectors into embeddings, with a dataset size of 200k words. All the embeddings were generated using OpenAI's embedding model 3 (3072 dim per embedding) . Now I am planning to implement semantic search similarity. Given a domain keyword, I want to find the top 5 most similar matches. After embedding all 280k words, the size of the JSON file containing the embeddings is around 30GB. (Edit, as suggestion saved in msgpack format, 6.5GB …

best practices dataset domain embedding embeddings generated machinelearning model 3 openai per planning practices search semantic vectors words