Vector DB directory structuring - ideal?

Vector DB offerings today are structured in such a way that the user is expected to have all files/file embeddings in the same place, and every time a search is effected, the entirety of that pool is queried through.

If so prepared, a user can do some filtering through metadata tags. However, this feels like a limited and clunky way to reduce the scope of what's queried.

Am I missing something here? Do most use cases call for all files/vectors …

deeplearning directory embeddings every files filtering metadata pool search through vector