Ingestion Mini-Pipeline
Pull a paginated REST API into Parquet and upsert it into warehouse staging with watermark-based incremental loads. Retries, idempotent re-runs, and a findings note with row counts — the Week-4 ingestion lab hardened into a portfolio repo.
What you'll learn
- —Paginate a REST API with retries + exponential backoff
- —Land raw extracts as partitioned Parquet
- —Upsert Parquet into staging inside one transaction
- —Track watermarks so re-runs are idempotent
The 4 milestones
Each milestone is reviewed before you advance.
- 01
Repo setup + fixture landing
Repo layout, fixture payloads, and staging DDL running.
- 02
Paginated fetch with retries
fetch.py pulls every page with backoff on 429/5xx.
- 03
Incremental load with watermark
load.py upserts Parquet to staging; watermark advances on success.
- 04
README + failure drill
Portfolio README plus a demonstrated retry/recovery run.
Reading & references
Build your portfolio & get certified
Want more projects like this? Join our Data Analyst Work Experience Program to complete 5+ guided industry projects, gain real experience, and earn an internship certificate.
Free Ingestion Mini-Pipeline portfolio project — with AI review
Build a real ingestion mini-pipeline project for your data analyst portfolio in your own public GitHub repo — free brief, starter template, and milestone guides. When your code is ready, the Free pass gets every milestone reviewed against a fixed rubric (de ingestion mini pipeline walkthrough included) and issues a verifiable certificate on completion.
Is the Ingestion Mini-Pipeline project free?
The full project brief, milestone guides, and starter template are free to audit. The AI-powered review pass and verified certificate unlock at Free.
Can I add the Ingestion Mini-Pipeline project to my resume and GitHub?
Yes — that is the point. You build in your own public GitHub repo, every milestone is reviewed against a fixed rubric, and the certificate links to the repo employers can inspect. Only list what is visible in your repo.
How long does the Ingestion Mini-Pipeline project take?
Most students finish the 4 milestones in 2–4 weekends. Milestones unlock in order, and you can re-submit any milestone that needs work.
Rubric-reviewed · Plagiarism-checked · Verifiable certificate