Build 2 — Async API Client Library

Scenario

Build a resilient, asynchronous HTTP scraper capable of collecting data from hundreds of remote endpoints concurrently while respecting server rate limits and handling transient network failures.

Requirements

1. Concurrency Control: Use aiohttp.ClientSession and asyncio.Semaphore to enforce a hard concurrency ceiling (e.g., max 20 simultaneous sockets).
2. Resilience & Retries: Implement an async retry decorator with exponential backoff and jitter for 5xx responses and network timeouts. Respect 429 Too Many Requests headers.
3. Structured Metrics: Emit JSON-formatted logs for every request including URL, HTTP status, latency milliseconds, and bytes received.
4. Graceful Shutdown: Intercept SIGINT/SIGTERM to allow in-flight requests to finish cleanly and close connections.

Starter Code Layout

build-2-async-scraper/
├── src/
│   └── scraper/
│       ├── __init__.py
│       ├── client.py     # aiohttp session with Semaphore
│       ├── retry.py      # Exponential backoff decorator
│       └── runner.py     # Main event loop runner
└── tests/
    └── test_scraper.py   # aioresponses mock tests

Verification Command

pytest tests/test_scraper.py -v
python -m scraper.runner --urls sample_urls.txt --concurrency 10

100-Point Evaluation Rubric

• AsyncIO & Semaphore Mechanics (30 pts): Concurrency bounded strictly by semaphore; no unbounded socket creation.
• Retry & Error Isolation (25 pts): Backoff calculation correct; transient errors retried; fatal errors logged without crashing.
• Metrics & Logging (25 pts): Structured JSON logs with p50/p95 throughput summary.
• Clean Teardown (20 pts): Session and event loop cleanly closed on exit.

Grading rubric

Your submission passes when all four hold:

1. Works: Concurrency never exceeds the cap; 5xx/429s retried with backoff; JSON logs carry URL/status/latency/bytes; SIGINT shuts down cleanly.
2. Tested: a pytest suite covers the happy path plus one failure mode, and all tests pass.
3. Readable: a README explains setup, the two most important commands, and one design decision you made.
4. Production habits: structured logging on failures and no hardcoded secrets/paths (env or config file).

Submission contract

Submit a public repo URL containing source, tests, README, and requirements.txt (or pyproject.toml). A reviewer runs your tests and two commands from your README; if either fails on a clean machine, the submission is returned with notes.