all AI news
Transfer SQL-> analytics 30x faster with ConnectorX + arrow + dlt
Oct. 27, 2023, 10:58 a.m. | adrian
DEV Community dev.to
dlt is a recently released python library for data extraction and loading, the EL in ETL. At dltHub we are big fans of optimising things and integrating those optimisations into our toolkit to enable others to re-use them.
Speed boosts and schema from arrow, dlt for loading with schema evolution
In this example, we combine ConnectorX + Arrow + dlt to extract data and load it to a strongly typed environment 30x faster than classic data transfer via sqlalchemy.
Result: …
analytics arrow big data dataengineering data extraction dlt etl extraction fans faster library loading python schema speed sql them toolkit transfer
More from dev.to / DEV Community
Jobs in AI, ML, Big Data
AI Engineer Intern, Agents
@ Occam AI | US
AI Research Scientist
@ Vara | Berlin, Germany and Remote
Data Architect
@ University of Texas at Austin | Austin, TX
Data ETL Engineer
@ University of Texas at Austin | Austin, TX
Lead GNSS Data Scientist
@ Lurra Systems | Melbourne
Lead Data Modeler
@ Sherwin-Williams | Cleveland, OH, United States