Delta Tensor: Efficient Vector and Tensor Storage in Delta Lake

arXiv:2405.03708v1 Announce Type: cross
Abstract: The exponential growth of artificial intelligence (AI) and machine learning (ML) applications has necessitated the development of efficient storage solutions for vector and tensor data. This paper presents a novel approach for tensor storage in a Lakehouse architecture using Delta Lake. By adopting the multidimensional array storage strategy from array databases and sparse encoding methods to Delta Lake tables, experiments show that this approach has demonstrated notable improvements in both space and time efficiencies when …

abstract applications architecture artificial artificial intelligence arxiv cs.db cs.dc cs.lg data delta development growth intelligence lake lakehouse lakehouse architecture machine machine learning multidimensional novel paper solutions storage tensor type vector