Build 3 — High-Performance Data Processor

Scenario

Build an in-memory streaming transformation pipeline that processes continuous telemetry events using Python generators and itertools.

Requirements

1. Pipeline Architecture: Compose stages using chained generator functions: event_source -> parse_json -> filter_anomalies -> rolling_aggregate -> sink
2. Memory Bounding: The pipeline must process a stream of 1,000,000 events with peak RAM usage under 50MB.
3. Sliding Window: Implement a sliding-window moving average using collections.deque(maxlen=K) without recomputing window sums from scratch.
4. Generator Teardown: Use try...finally inside generator stages to ensure resource handles are closed when generator is terminated via .close().

Starter Code Layout

build-3-stream-processor/
├── src/
│   └── stream/
│       ├── __init__.py
│       ├── pipeline.py   # Composable generator stages
│       └── window.py     # Deque-based moving window
└── tests/
    └── test_pipeline.py  # Memory & correctness assertions

Verification Command

pytest tests/test_pipeline.py -v
python -m stream.benchmark  # Asserts peak memory < 50MB via tracemalloc

100-Point Evaluation Rubric

• Generator Composition (30 pts): Pure generator stages yielding one record at a time.
• Memory Ceiling Enforcement (25 pts): Verified flat memory profile across 1M records via tracemalloc.
• Sliding Window Performance (25 pts): O(1) amortized window update using deque.
• Resource Cleanup (20 pts): Proper .close() and generator exit handling.

Grading rubric

Your submission passes when all four hold:

1. Works: 1M events under 50MB peak; sliding average is O(1) per event; generator teardown closes resources on .close().
2. Tested: a pytest suite covers the happy path plus one failure mode, and all tests pass.
3. Readable: a README explains setup, the two most important commands, and one design decision you made.
4. Production habits: structured logging on failures and no hardcoded secrets/paths (env or config file).

Submission contract

Submit a public repo URL containing source, tests, README, and requirements.txt (or pyproject.toml). A reviewer runs your tests and two commands from your README; if either fails on a clean machine, the submission is returned with notes.