The custom reconciliation dashboard built for the finance team.
Context
Series B fintech, $2M+ monthly in cross-border transactions. Reconciliation ran on cron jobs and spreadsheets, failing multiple times weekly. Finance burned 15+ hours on manual fixes. Engineering flew blind on stuck transactions.
Problem
- -No single source of truth for transaction state
- -Race conditions between payment processor webhooks and internal state updates
- -Zero observability. Failures discovered when customers complained
- -No retry mechanism; failed reconciliations required manual database queries
Constraints
- -Could not pause transaction processing during migration
- -Existing data inconsistencies needed to be resolved without data loss
- -Team of 3 engineers, two of whom were junior
- -6-week deadline before audit season
Role & Ownership
Sole architect, primary implementer. Mentored two junior engineers on event-driven patterns. Reported to VP Engineering on progress and tradeoffs.
Technical Approach
Before changing any processing logic, I instrumented the existing system. Added structured logging, created a transaction state dashboard, and built alerts for stuck transactions. This alone cut incident response time from hours to minutes.
Introduced an append-only transaction event table as the source of truth. Every state change (initiated, confirmed, failed, reconciled) became an immutable event. Reconciliation became a projection over events rather than a brittle comparison of mutable states.
Rebuilt the payment processor integration with idempotency keys and proper retry handling. Webhooks that arrived out of order no longer corrupted state. Used the transactional outbox pattern to guarantee event publication.
Replaced cron jobs with an event-driven pipeline using NATS JetStream. Dead letter queue for failed reconciliations with automated retry and alerting. Finance dashboard for real-time reconciliation status.
Results
- ✓Daily reconciliation failures: 47 → 0
- ✓Manual intervention hours: 15/week → 3/week
- ✓Mean time to detect stuck transactions: 4 hours → 8 minutes
- ✓Passed financial audit with zero findings on transaction integrity
- ✓Both junior engineers confidently deploying to the reconciliation system within 8 weeks
What I'd Do Next
Next: distributed tracing to correlate events across payment processor, core banking, and notifications. Snapshot-based reconciliation for month-end to reduce event replay overhead.