Build 1 — CLI Tool (Typer + tests + package + publish)

Scenario

You are tasked with building a production-grade Command-Line Interface (CLI) tool for an internal data engineering team. The tool ingests raw event log files (.csv or .jsonl), validates records against a strict Pydantic v2 schema, applies transformations, and outputs clean normalized records.

Requirements

1. CLI Ergonomics: Build the CLI using Click or Typer. Support commands:
   • ingest --input-path <path> --output-path <path> --config <config.yaml>
   • Flags: --dry-run (processes 10 records and prints sample), --log-level [DEBUG|INFO|WARNING]
2. Strict Validation: Use Pydantic v2 BaseModel with @field_validator to validate email formats, ISO timestamps, and positive integers. Bad records must be isolated to an errors.jsonl dead-letter file.
3. Memory Streaming: Use generator functions to process inputs line-by-line so files larger than RAM can be processed seamlessly.
4. Packaging: Configure pyproject.toml with console script entry-point: data-pipeline = "data_pipeline.cli:main".

Starter Code Layout

build-1-cli/
├── pyproject.toml
├── config.example.yaml
├── src/
│   └── data_pipeline/
│       ├── __init__.py
│       ├── cli.py        # Typer entrypoint
│       ├── config.py     # Pydantic schema validation
│       └── processor.py  # Streaming generator pipeline
└── tests/
    ├── test_cli.py
    └── test_validation.py

Verification Command

pytest tests/ -v --tb=short
data-pipeline ingest --input-path sample.csv --dry-run

100-Point Evaluation Rubric

• CLI Design & Error Handling (25 pts): Clear help messages, argument parsing, graceful error exits on invalid file paths.
• Pydantic v2 Validation (25 pts): Strict schema enforcement, dead-letter recording with failure reason.
• Streaming Pipeline (25 pts): Generator-based streaming with verified O(1) memory overhead.
• Packaging & Typing (25 pts): Clean src/ layout, passing mypy --strict, console script execution.

Coming from the fundamentals track? This build streams input with generator functions — covered in Module M3 (Generators, Iterators & itertools). Skim that lesson first if yield is new to you.

Grading rubric

Your submission passes when all four hold:

1. Works: CLI accepts the documented flags; Pydantic rejects bad rows into errors.jsonl; --dry-run prints 10 sample records; pytest suite passes.
2. Tested: a pytest suite covers the happy path plus one failure mode, and all tests pass.
3. Readable: a README explains setup, the two most important commands, and one design decision you made.
4. Production habits: structured logging on failures and no hardcoded secrets/paths (env or config file).

Submission contract

Submit a public repo URL containing source, tests, README, and requirements.txt (or pyproject.toml). A reviewer runs your tests and two commands from your README; if either fails on a clean machine, the submission is returned with notes.