Recap — BackgroundTasks vs Celery vs ARQ

# BackgroundTasks vs Celery vs ARQ When an endpoint needs to do work that shouldn't make the user wait (send email, generate report, process upload), offload it. Three options scale with complexity. ## Option 1: FastAPI BackgroundTasks (simplest) ```python from fastapi import BackgroundTasks def send_email(to: str, subject: str, body: str): # actual email-sending code smtp.send(to, subject, body) @app.post("/signup") def signup(payload: UserCreate, background_tasks: BackgroundTasks): user = create_user(payload) background_tasks.add_task(send_email, user.email, "Welcome", "Hi!") return {"id": user.id} ``` How it works: - Response is returned immediately. - The task runs in the same process AFTER the response is sent. - If the process crashes before the task runs, the task is lost. When to use: - Simple, non-critical work (welcome emails, audit logs). - Tasks that complete quickly (~1 sec). - You don't need retries. When NOT: - Heavy CPU work (blocks the worker process). - Tasks that take 10+ seconds (lose if process restarts). - Important work that must succeed (no retries built-in). ## Option 2: Celery (the heavyweight standard) A separate process pool consumes tasks from a queue (Redis or RabbitMQ). ```python # tasks.py from celery import Celery celery_app = Celery("worker", broker="redis://localhost:6379") @celery_app.task(bind=True, max_retries=3) def send_email(self, to: str, subject: str, body: str): try: smtp.send(to, subject, body) except SMTPError as e: raise self.retry(exc=e, countdown=60) ``` In FastAPI: ```python from tasks import send_email @app.post("/signup") def signup(payload: UserCreate): user = create_user(payload) send_email.delay(user.email, "Welcome", "Hi!") # enqueue return {"id": user.id} ``` Run the worker: ```bash celery -A tasks worker --loglevel=info ``` Features: - Persistent queue (Redis); tasks survive restarts. - Built-in retries. - Scheduled tasks (Celery Beat). - Result backend for getting task return values. - Massive ecosystem. Use when: - Heavy or long-running tasks. - Need retries / scheduling. - Production-grade task management. Cons: - Complex setup (worker process, broker, result backend). - Sync workers; not async-native. - Operational overhead. ## Option 3: ARQ (async-native, lighter than Celery) ```python # tasks.py from arq.connections import RedisSettings async def send_email(ctx, to: str, subject: str, body: str): # async email sending async with smtp_client() as smtp: await smtp.send(to, subject, body) class WorkerSettings: functions = [send_email] redis_settings = RedisSettings() ``` In FastAPI: ```python from arq import create_pool async def get_arq_pool(): return await create_pool(RedisSettings()) @app.post("/signup") async def signup(payload: UserCreate, redis = Depends(get_arq_pool)): user = create_user(payload) await redis.enqueue_job("send_email", user.email, "Welcome", "Hi!") return {"id": user.id} ``` Run worker: ```bash arq tasks.WorkerSettings ``` Features: - Async-native (uses asyncio). - Lighter than Celery. - Redis-only (no RabbitMQ). - Retries, scheduling, simpler API. Use when: - Already async stack. - Want simpler ops than Celery. - Don't need Celery's massive feature set. ## Comparison | Aspect | BackgroundTasks | Celery | ARQ | |---|---|---|---| | Persistence | No | Yes | Yes | | Retries | No (manual) | Yes (built-in) | Yes (built-in) | | Scheduling | No | Yes (Celery Beat) | Yes | | Async-native | Not really | No | Yes | | Setup complexity | Trivial | Heavy | Medium | | Best for | Simple post-response work | Production task queues | Async-first stacks | ## Decision flowchart ``` Need task to survive process crash? ├── No → BackgroundTasks └── Yes ├── Async stack? → ARQ └── Sync stack / want max features? → Celery ``` ## Common patterns ### Pattern 1: Fire-and-forget email ```python @app.post("/orders") def create_order(payload: OrderCreate, background_tasks: BackgroundTasks): order = process_order(payload) background_tasks.add_task(send_order_confirmation, order.id) return order ``` BackgroundTasks: simple, fast, fits the use case. ### Pattern 2: Report generation (slow) ```python @app.post("/reports") async def request_report(payload: ReportRequest, redis = Depends(get_arq_pool)): job = await redis.enqueue_job("generate_report", payload.dict()) return {"job_id": job.job_id, "status": "processing"} @app.get("/reports/{job_id}") async def get_report(job_id: str, redis = Depends(get_arq_pool)): job = await redis.job(job_id) return {"status": job.status, "result_url": job.result if job.status == "complete" else None} ``` User polls for completion. ARQ handles persistence. ### Pattern 3: Scheduled task (cron) With Celery Beat: ```python celery_app.conf.beat_schedule = { "cleanup-old-sessions": { "task": "tasks.cleanup_sessions", "schedule": crontab(ho