[P] Reducing 2048 dimensions to 2000 dimensions for PGVector

Hi people of ml,

I am working on a project that uses PGVector for efficient similarity search, and I use feature vectors I obtain from EfficientNet B5 which outputs 2048d. The issue is that I need to index my tables based on the vectors, otherwise, your typical DB hardware problems occur (Not enough RAM). However, the methods PGVector offers have a limit, the vectors can be at most 2000d. One solution I have found is PCA, but I have quite …

feature hardware index issue machinelearning people pgvector project search tables vectors