Building a Self-Healing, Triple-Verified Ledger System for Tixin.in
When building Tixin.in — an event-ticketing platform — we needed a ledger system capable of handling thousands of financial transactions daily while maintaining absolute accuracy.
The challenge wasn’t just storing transactions.
It was designing a self-healing, triple-verified financial system that detects and corrects discrepancies automatically.
The Core Principle: Trust Nothing
One of our earliest lessons:
Never trust a single source of truth.
In production systems, things go wrong:
- Network failures during database writes
- Race conditions between concurrent transactions
- Admins manually correcting data
- Payment gateway success while your DB update fails
So we built a system with three independent sources of truth that continuously verify each other.
The Triple-Verification Architecture
We maintain three independent records that must always agree.
1. Event Analytics (Revenue Truth)
EventAnalytics.revenue // Total ticket sales per event
2. Payout Records (Debit Truth)
Payout.approvedAmount // Only payouts with status: COMPLETED
3. Ledger Entries (Immutable Transaction History)
LedgerEntry {
entryType: CREDIT | DEBIT
amount: Decimal
balanceAfter: Decimal
referenceType: EVENT_REVENUE | PAYOUT
referenceId: string
}
Continuous Reconciliation
Every minute, a worker verifies:
Account.balance === (TotalRevenue - TotalPayouts)
Account.balance === LatestLedgerEntry.balanceAfter
LedgerCredits === EventRevenue
LedgerDebits === CompletedPayouts
If anything doesn’t match → the system auto-corrects.
Implementation Deep Dive
1. The Ledger Worker (Runs Every 60 Seconds)
async function syncAllLedgers() {
const listers = await prisma.lister.findMany({
include: {
Account: true,
Event: {
include: { EventAnalytics: { select: { revenue: true } } }
}
}
});
for (const lister of listers) {
await syncListerLedger(lister);
}
}
Responsibilities:
- Ensures each lister has an account
- Detects missing ledger entries
- Creates adjustment entries
- Updates balances atomically
2. Immutable Ledger Entries
We never modify existing ledger entries — only append new ones.
await prisma.$transaction([
prisma.ledgerEntry.create({
data: {
accountId: account.accountId,
entryType: difference.greaterThan(0) ? "CREDIT" : "DEBIT",
amount: difference.abs(),
balanceAfter: newBalance,
referenceType: "EVENT_REVENUE",
referenceId: event.eventId,
},
}),
prisma.account.update({
where: { accountId: account.accountId },
data: { balance: newBalance },
}),
]);
Atomicity ensures:
Either both operations succeed, or both fail — never partial updates.
3. The Self-Healing Reconciliation Function
async function reconcileBalance(listerId: string) {
const [totalRevenue, completedPayouts, ledgerEntries] = await Promise.all([
prisma.eventAnalytics.aggregate({
where: { event: { listerId } },
_sum: { revenue: true }
}),
prisma.payout.aggregate({
where: { listerId, status: "COMPLETED" },
_sum: { approvedAmount: true }
}),
prisma.ledgerEntry.aggregate({
where: { accountId: account.accountId },
_sum: { amount: true }
})
]);
const computedFromRevenue = totalRevenue - completedPayouts;
const ledgerBalance = latestLedgerEntry.balanceAfter;
const correctBalance = ledgerEntries.count > 0
? ledgerBalance
: computedFromRevenue;
if (!currentBalance.equals(correctBalance)) {
await prisma.account.update({
where: { accountId },
data: { balance: correctBalance }
});
}
}
4. Transparent Account Response
{
"accountBalance": "31918.05",
"totalRevenue": "51081.05",
"totalPayouts": "19163",
"computedBalance": "31918.05",
"ledgerBalance": "31918.05",
"ledgerCredits": "51081.05",
"ledgerDebits": "19163",
"balancesMatch": true
}
All values are cross-verified.
Key Design Patterns
1. Immutable Event Sourcing
Append-only ledger → complete audit trail.
2. Running Balance Pattern
Each entry stores the post-transaction balance.
This makes balance lookup O(1):
prisma.ledgerEntry.findFirst({
where: { accountId },
orderBy: { createdAt: 'desc' }
});
3. Idempotent Workers
Running workers multiple times does not create duplicates.
4. Parallel Queries for Speed
await Promise.all([...])
Huge speed improvement.
5. Atomic Database Transactions
Guarantee no partial updates during financial operations.
Integrity Checks
Daily Ledger Integrity
if (account.balance !== latestEntry.balanceAfter) {
fixAutomatically()
}
Orphaned Ledger Entry Detection
SQL left join detect entries referencing deleted events.
Performance Optimizations
Before:
- 250–300ms per financial action
- Blocking requests
- No recovery from failures
After:
- <50ms worker processing
- 20ms balance lookups
- 10k+ transactions/min
- Automatic recovery from failures
Real-World Flow Example
t = 0 – Ticket Sold
Event revenue increases.
t = 60s – Worker Runs
Creates ledger entry + updates balance.
Query Anytime
Instant O(1) latest-entry lookup.
Verification
All sources cross-check automatically.
Edge Cases We Solved
✔ Network failure mid-transaction ✔ Concurrent workers ✔ Manual DB edits ✔ Payment gateway → success, DB → failure ✔ Duplicate worker runs
All handled automatically.
Monitoring & Alerts
We log and alert:
- Balance mismatches
- Orphaned ledger entries
- Worker failures
- Auto-corrections
logger.warn("Balance discrepancy detected", { ...})
Lessons Learned
- Multiple sources of truth > single DB trust
- Immutability makes debugging trivial
- Self-healing > manual ops
- Running balances > summing history
- Atomic transactions are mandatory
- Parallel queries massively improve performance
Simple Ledger vs Triple Verification
| Simple Ledger | Our System | | ----------------------- | ----------------------- | | Single balance column | 3 independent sources | | Trust DB writes blindly | Continuous verification | | Manual reconciliation | Auto-correcting | | No audit trail | Immutable history | | Update balance directly | Append-only | | Hope for the best | Paranoid by design |
Core Utility Functions
Get Current Balance
async function getCurrentBalance(accountId: string) { ... }
Verify Ledger Integrity
async function verifyLedgerIntegrity() { ... }
Conclusion
Financial systems can’t rely on trust — they need redundancy, immutability, and continuous correction.
By combining:
- Triple verification
- Immutable event-sourced ledger
- Atomic transactions
- Running balances
- Continuous reconciliation
…we built a system that is fast, reliable, self-healing, and production-grade.
Tech Stack: TypeScript, Prisma, PostgreSQL, Node.js Performance: <50ms balance queries, 10k+ tx/min, zero data loss Live: Powering real financial transactions at Tixin.in