For our community DuckDB is the default data warehouse for local development environment. Last month +90,000 users used dlt (and their AI code editor) to load data into DuckDB.

Because of our proximity to the DuckDB community we are seeing enterprise DuckDB usage first hand. People imo sleep on the data volumes DuckDB can handle. We see Fortune 100 companies use dlt and DuckDB in production on their Lakehouses in hybrid cloud deployments. I can eg mention Stellantis (Chrysler, Jeep, Peugeot etc) because they talk about it publicly.

We launched dlt on HN 7 months ago with a mission to make getting datasets fast and easy. Now dlt helps its users to code around a thousand new data sources each month and to maintain many thousands of live datasets in production.

Today we are releasing *dlt-init-openapi,* a Python CLI tool that generates a dlt data pipeline from any OpenAPI spec. It brings the time to create a dataset down to a few minutes.

Here’s a Colab demo: https://colab.research.google.com/drive/1MRZvguOTZj1MlkEGzji...

Here’s a video walkthrough: https://youtu.be/b99qv9je12Q

---

In the past you had to analyze REST API endpoints, response data types, pagination styles and write lots of custom Python code to create a dataset. OpenAPI specifications standardized API definitions, making it easier to interact with APIs. Now, dlt-init-openapi leverages these OpenAPI specs to automate the manual work around dataset creation you had to do before. OpenAPI is a growing standard with almost every API using it, and used by default in frameworks like FastAPI.

What can dlt-init-openapi do for you?

- It generates Python scripts with dlt pipelines that you can run to pull data from your API into a structured destination of your choice (Parquet files, SQL DBs, Databricks, Snowflake, etc.)

- Infers and evolves schema for the endpoints from the actual data!

- Discovers pagination style for each endpoint

- Finds and unwraps data entities for each endpoint, also for deep, nested JSON responses

- Discovers the primary key for each entity

- Discovers authentication schema and generates code and config files to pass required credentials

- You always have the last say. The generated code is declarative and ready to hack in case we pick the wrong paginator or response entity.

The tool and dlt are open source, find the code here: https://github.com/dlt-hub/dlt-init-openapi and here: https://github.com/dlt-hub/dlt

Comments URL: https://news.ycombinator.com/item?id=40512113

Points: 7

# Comments: 1

We have a PR (https://github.com/dlt-hub/dlt/pull/594) that is about to merge that makes the above highly configurable, between evolution and hard stopping: - you will be able to totally freeze schema and reject bad rows - or accept the data for existing columns but not new columns - or accept some fields based on rules'

We're Anna, Adrian, Marcin and Matt, developers of dlt. dlt is an open source library to automatically create datasets out of messy, unstructured data sources. You can use the library to move data from about anywhere into most of well known SQL and vector stores, data lakes, storage buckets, or local engines like DuckDB. It automates many cumbersome data engineering tasks and can by handled by anyone who knows Python.

Here’s our Github: https://github.com/dlt-hub/dlt

Here’s our Colab demo: https://colab.research.google.com/drive/1DhaKW0tiSTHDCVmPjM-...

— — —

In the past we wrote hundreds of Python scripts to fit messy data sources into something that you can work with in Python - a database, Pandas frame or just a Python list. We were solving the same problems and making the similar mistakes again and again.

This is why we built an easy to use Python library called dlt that will automate most data engineering tasks. It hides the complexities of data loading and automatically generates a structured and clean datasets for immediate querying and sharing.

— — —

At its core, dlt removes the need to create the dataset schemas, react to changing data, generate append or merge statements, and to move the data in transactional and idempotent manner. Those things are automated and can be declared right in the Python code, just by decorating functions.

Add @dlt.resource decorator, give it a few hints, and convert any data into a simple pipeline that creates and updates datasets.

dlt gets the details out of your way:

1. You do not need to worry about the structure of a database or parquet files

dlt will create a nice, typed schema out of your data and will migrate it when the data changes. You can put some data contracts and Pydantic models on top to keep your data clean.

2. You do not need to write any INSERT/UPDATE or data copy statements

dlt will push the data to DuckDB, Weaviate, storage buckets and many popular SQL stores. It will align the data types, file formats, and identifier names automatically

3. You do not need to worry when you need to add new data or update the changes.

dlt lets you declare how to load the data, how to increment it and will keep the loading state together so they are always in sync.

4. You keep how you develop and test your code

Iterate and test quickly on your laptop or in a dev container. Run locally on DuckDB and just swap destination name to go to the cloud - your code, schema and data will stay the same.

5. You can work with data on your laptop.

Combine dlt with other tools and libraries to process data locally. duckdb, Pandas, Arrow tables and Rust based loading libraries like ConnectorX work nicely with dlt and process data blazingly fast, compared to the cloud.

6. You do not need to worry if your pipeline will work when you deploy it.

dlt is a minimalistic Python library, requires no backend and works whenever Python works. You can finetune it to work on constrained environments like AWS Lambda or run with Airflow, GitHub Actions or Dagster.

dlt has an Apache 2.0 license. We plan to make money by offering organizations a paid control plane, where dlt users can track and policy what every pipeline does, manage schemas and contracts across organization, create data catalogues, and share them with the team members and customers.

Comments URL: https://news.ycombinator.com/item?id=37999527

Points: 114

# Comments: 54