Data Engineering
nydas
•
8mo ago
•
87%
Building a Data Pipeline from Scratch
https://medium.com/@nydas/building-a-modern-data-pipeline-a-journey-from-api-to-insight-636819c972e6?sk=2649aff67be571fd3c6b5f94f2211f62I’ve written a series of Medium articles on creating a Data Pipeline from scratch, using Polars and DeltaTables. The first (linked) is an overview with link to the GitHub repository and each of the deeper dive articles. I then go into the next level of detail, walking through each component.
The articles are paywalled (it took time to build and document), but the link provided is the ‘family & friends’ link which bypasses the paywall for the Lemmy community.
I hope some of you may find this helpful.
Comments 0