TL;DR: Built a CQRS-style event pipeline using Kafka for ingestion, Rust/node consumers for processing, and a split write/query model to achieve 1.8M records processed in under 10 minutes.

Problem & goals

Ingest massive volumes without slowing writes.
Maintain near-real-time projections for query consumers.
Keep the system testable with E2E scenarios.

Architecture overview

Ingestion: Producers publish to Kafka topics (partitioning by entity id)
Processing: Consumer groups (Rust & Node.js) commit offsets carefully with batching
Command side: Postgres for canonical writes (ACID)
Query side: MongoDB projections for read-optimized queries
Event store: Kafka acts as the append-only log

Performance tuning

Carefully size partitions to match consumer parallelism.
Use batched commits and idempotent consumers.
Optimize serialization format (Avro / Protobuf recommended).
Use consumer prefetch & flow-control to avoid GC spikes.

Results & metrics

Records processed: ~1.8M
Time: < 10 minutes in optimized runs
Lessons: Partition key design and consumer concurrency make or break high-throughput pipelines.

Stack

Kafka · Rust · Node.js · PostgreSQL · MongoDB · Docker

Repo / demo

Public repo and demo links available on my GitHub. See: https://github.com/Adityaadpandey. :contentReference[oaicite:2]{index=2}