Distributed systems
Event-Driven Notification System
Event-driven notification platform built with Python, FastAPI, Celery, Redis, PostgreSQL, and Docker. Designed for high reliability with idempotency, retry logic, dead-letter queues, and horizontal scalability.
Overview
Project overview
Event-driven notification platform built with Python, FastAPI, Celery, Redis, PostgreSQL, and Docker. Designed for high reliability with idempotency, retry logic, dead-letter queues, and horizontal scalability.
Engineering highlights
- 01
Built an event-driven notification platform that sends email, SMS, and webhook notifications to up to 1,000 recipients per event, using separate Celery queues so each channel can scale independently.
- 02
Made fan-out recoverable by saving every notification to PostgreSQL before dispatching worker tasks, preventing work from being lost when the dispatcher or a worker crashes.
- 03
Prevented duplicate sends with Redis idempotency checks, PostgreSQL unique constraints, and row-level locking, keeping delivery correct even when Redis is unavailable or two workers receive the same task.
- 04
Added channel-specific retries with exponential backoff and jitter, along with a PostgreSQL-backed dead-letter queue for failed notifications and manual or bulk replay.
- 05
Built a scheduled reconciliation process that recovers overdue retries and tasks left in progress by crashed workers, using FOR UPDATE SKIP LOCKED to allow multiple sweep workers to run safely in parallel.
- 06
Added atomic Redis rate limiting, HMAC-SHA256 webhook signing, and SSRF protection for outbound webhook requests.
- 07
Built a React and TypeScript dashboard for queue monitoring, delivery analytics, failure inspection, and dead-letter replay.
- 08
Containerised the platform with Docker Compose and covered the API, workers, retry behaviour, and service logic with more than 200 tests.