PayFlow Payment Processing Platform

Overview

PayFlow is a from-scratch payment backend that models the full payment lifecycle (create, authorize, capture, settle, refund, reconcile) as a finite state machine, backed by an immutable double-entry ledger and a stack of concurrency and resilience mechanisms. Three guarantees hold under real load: a balance can never go negative, a payment can never be charged twice, and a scheduled job can never run twice across replicas.

It runs on Spring Boot 4.0.7, Java 21, PostgreSQL 17, and Redis 8, deliberately the newest generation of each, which meant working through undocumented migration issues. It is verified by 84 tests across 11 classes, including integration suites against real PostgreSQL and Redis via Testcontainers. The repo is public at github.com/atharva009/payflow.

It is a portfolio-grade reference implementation, not a live money-mover: the external processor is a configurable mock simulating realistic outcomes and latency, but everything around that call is built and tested to the standard a real payment system requires.

---

The Problem

Moving money correctly is deceptively hard. "Debit A, credit B, mark it done" reads like three lines of code but hides correctness problems that only surface under load, failure, or retry:

• A client's request times out and it retries. Did the original go through? Does the retry double-charge?
• Two requests race the same account balance. Can both pass a sufficient-funds check and jointly overdraw it?
• The processor takes three seconds or fails intermittently. Does the service hold a lock and connection open the whole time?
• The service crashes between deciding to charge and recording it. Is the ledger left half-written, or the payment stuck forever?
• Multiple replicas each fire the same scheduled job. Does the work get duplicated?

There was also a personal version. Payment batch processing and token generation are real professional experience for me at LTM, but that work lives in a private, employer-owned codebase: describable in an interview, not runnable. PayFlow closes that gap with a fully public system that demonstrates the same discipline.

---

Goals

• Model the payment lifecycle as a compile-time-fixed, exhaustively validated finite state machine: no ad hoc status strings, no invalid transitions reachable at runtime.
• Guarantee, under real concurrent load, that a balance can never go negative, proven by a concurrency test rather than asserted by design.
• Guarantee a payment is never charged twice, including across a crash mid-authorization.
• Enforce exactly-once semantics at the API edge through defense-in-depth idempotency that fails closed rather than silently degrading under a cache outage.
• Build an append-only double-entry ledger enabling O(1) balance reconstruction and clean reconciliation.
• Run scheduled work exactly once across however many replicas are deployed, without relying on node clocks.
• Build resilience around a flaky dependency: circuit breaking and retry keyed on failure type, not a blanket retry-everything policy.
• Ship cleanly on the newest Spring Boot generation, documenting the migration issues rather than avoiding them.

---

Technical Architecture

The Defining Decision: A Synchronous Edge, an Asynchronous Core

Every other decision flows from one choice: POST /api/v1/payments returns 202 Accepted with the payment PENDING, not 201 Created with a finished transaction. No money moves in the request thread; the request is validated and recorded as an intent, and authorization happens moments later off the request path in a background poller. Because the write path never blocks on the processor, it stays fast and retry-safe however flaky the processor is, and the service scales horizontally with no request-time coordination between replicas.

The request path is a filter chain: MdcTraceFilter assigns a trace ID so every log line is correlated; IdempotencyFilter handles deduplication (scoped to POST /api/v1/payments) before Spring Security; security validates the JWT and enforces role-based access (ROLE_USER for payments, ROLE_ADMIN for admin). On the other side of the boundary sit background workers, each on its own schedule, each coordinated across replicas by a distributed lock:

The Payment State Machine

Payment status is an 8-value enum, and the legal transitions are fixed at compile time as an EnumMap<PaymentStatus, EnumSet<PaymentStatus>>: exactly 11 legal edges, everything else structurally unreachable.

SETTLED, CANCELLED, FAILED, and REFUNDED are terminal, but terminality and having an outgoing edge are independent: SETTLED to REFUNDED is legal because a settled payment can still be reversed weeks later. transitionTo(target) is a pure function with no dependency on Spring or the database, so the whole graph is exhaustively unit tested against an independently hand-written oracle, asserting all 79 from-to-legality combinations in milliseconds, touching no infrastructure.

Idempotency: Four Layers Deep, Fail-Closed

Clients retry, networks time out, load balancers occasionally deliver the same request twice, so a payments API treats duplicate delivery as the normal case. Four independent layers each close a gap the previous one leaves open:

1. SHA-256 of the request payload. The body hash is stored with the cached response; replaying the same key with a different body is rejected as 409 IDEMPOTENCY_CONFLICT.
2. A Redlock distributed mutex on lock:idempotency:<key> (5-second wait, 10-second lease), serializing concurrent same-key requests across every replica, not just within one process the way a plain lock or SETNX wouldn't under failover.
3. A two-layer cache. A Redis L1 fronts a PostgreSQL L2 holding the authoritative record; a miss repopulates Redis with the correct remaining TTL, and expired keys are treated as fresh requests.
4. A database UNIQUE constraint on the key column, the backstop no application-level bug can race around.

The failure mode matters most. If Redis is unavailable the system fails closed with 503 CACHE_UNAVAILABLE rather than falling back to database-only checking. Redis is most strained during a retry storm, exactly when duplicate suppression matters most, so degrading gracefully there means losing the one guarantee the system exists to provide.

The Two-Transaction Processor Pattern: Crash Safety Around an External Call

PaymentPoller fetches a batch of PENDING payments with SELECT ... FOR UPDATE SKIP LOCKED in a short transaction, so the row locks release the instant the batch is fetched rather than being held through processing. Authorizing one payment then spans two deliberately separated transactions, with the external call sandwiched between them:

@Scheduled(fixedDelay = 5000)
@SchedulerLock(name = "paymentPoller", lockAtLeastFor = "1s", lockAtMostFor = "30s")
public void poll() {
    List<Payment> batch = fetchBatch(); // FOR UPDATE SKIP LOCKED, short tx, released fast

    for (Payment payment : batch) {
        self.assignProcessorRef(payment);                 // Tx A: REQUIRES_NEW, commits
        ProcessorResponse response = processor.charge(payment); // outside ANY transaction
        self.completeAuthorization(payment, response);    // Tx B: REQUIRED, all-or-nothing
    }
}

Transaction A commits a processor_ref before the charge happens. The charge runs outside any transaction, so the processor's multi-second latency never holds a database connection or row lock hostage. Transaction B then does the money movement, ledger entries, balance updates, and status transition as one atomic unit.

Committing A before the call is the entire point: processor_ref doubles as the processor's own idempotency key. If the service crashes after A but before B, the next poller cycle sees a processor_ref already assigned and retries with that same reference, which the processor deduplicates. No double charge, even across a hard crash. Generating a fresh reference per retry would silently turn a safe retry into a second, real charge.

One detail: self is a @Lazy-injected self-proxy, not a plain this. Calling this.assignProcessorRef(...) from within the same class bypasses Spring's AOP proxy, so @Transactional is silently ignored with no compile- or runtime error: the code looks correct, passes a read-through, and quietly runs non-transactionally, undermining the crash-safety guarantee.

The Double-Entry Ledger

Every money movement is recorded, never mutated. LedgerService writes exactly one DEBIT and one CREDIT entry per authorization, each stamped with a balance_after snapshot; reversals (an AUTHORIZED to CANCELLED, say) append compensating entries rather than editing originals. This buys O(1) balance reconstruction (the current balance is the most recent entry's balance_after, not a running sum over history) and clean reconciliation (an immutable, tamper-evident audit trail the reconciliation job compares against the processor's records).

Concurrency Control: Matching the Primitive to the Contention Profile

PayFlow uses a different primitive for each access pattern, chosen for the contention it actually experiences:

• Pessimistic locking on accounts (SELECT ... FOR UPDATE). An account balance is a shared mutable counter and check-then-debit has to be atomic, so locking serializes two concurrent debits: the second sees the first's committed balance and rejects if funds are now insufficient.
• Optimistic locking on payments (@Version). A payment row is owned by exactly one processing thread at a time, so there's no real contention, just the rare accidental double-pickup; the cheaper primitive is the correct one.
• SELECT ... FOR UPDATE SKIP LOCKED for the poller's batch fetch. SKIP LOCKED grabs the next available row rather than blocking. Combined with a partial index on WHERE status = 'PENDING', this turns a plain table into a horizontally scalable work queue: any number of replicas poll concurrently, each grabbing a disjoint slice, with no coordination beyond the database.
• ShedLock with database-time leases for scheduled jobs. @SchedulerLock ensures only one replica executes a given method at a time, with lease expiry measured against the database's clock via usingDbTime(), not each node's local clock, because clock skew could otherwise let two nodes both start the same job.

Each contention point gets the cheapest primitive that's still correct: pessimistic everywhere would tank throughput on the payment table; optimistic everywhere would let concurrent account debits overdraw a balance.

Resilience: The Exception Type Is the Retry Policy

MockProcessorAdapter.charge() is wrapped with both @CircuitBreaker and @Retry (Resilience4j), and the exception type itself is the control signal. TransientProcessorException is retryable: it triggers exponential backoff (3 attempts, 500ms base, doubling) and counts toward the circuit breaker's failure rate. PermanentProcessorException is ignored: it fails fast and does not trip the breaker, because a legitimate card decline isn't a signal that the processor is unhealthy. So there's zero imperative "should I retry this?" branching anywhere: the decision is encoded once in the exception type, and Resilience4j handles the rest (a 50% failure rate over a 10-call sliding window opens the circuit for 30 seconds before half-open probe traffic).

@TimeLimiter was deliberately removed: the Resilience4j aspect hard-throws IllegalReturnTypeException on a synchronous, non-CompletionStage method, which the mock call is. Rather than force an artificial async wrapper, the timeout config was retained (documented, inert) for a future async processor and the annotation removed from the synchronous path.

Settlement and Reconciliation

The nightly SettlementJob (a Spring Batch tasklet under ShedLock) collects the day's CAPTURED payments, transitions them to SETTLEMENT_QUEUED, and computes bilateral netting, collapsing back-and-forth obligations between the same two accounts into a single net instruction, entirely in SQL via a self-join using LEAST/GREATEST to canonicalize each counterparty pair. Netting in the JVM heap risks an OutOfMemoryError at volume; the self-join scales with the database instead.

The ReconciliationJob classifies every ledger-vs-processor discrepancy into one of six categories: matched, missing in the processor, missing internally, amount mismatch, status mismatch, or timing discrepancy. That last category avoids false positives: a payment still PENDING or AUTHORIZED internally naturally differs from the processor's view for a little while, and treating that lag as a hard STATUS_MISMATCH would flood the report with noise.

---

Key Technical Challenges

Challenge 1: Keeping a Multi-Second External Call Out of the Transaction Boundary Without Losing Crash Safety

Holding a transaction open for a timeout-prone HTTP call is a throughput and connection-pool disaster; moving it outside any transaction risks a crash losing track of whether the charge happened, or retrying and charging twice. The two-transaction pattern above resolves this: the processor_ref committed before the call is the idempotency key the mock deduplicates on, so a crash in the transaction-free gap neither loses nor duplicates the charge. Generating the reference after the call, or per retry, would quietly reintroduce the double-charge risk.

Challenge 2: Making Idempotency Fail Safely Instead of Failing Silently

The instinctive fix when the cache goes down, falling back to the database directly, is dangerous in exactly the scenario where it triggers, since Redis outages coincide with the retry storms where duplicate protection matters most; hence the fail-closed 503. The honest limitation, documented rather than hidden: response caching runs after the payment transaction commits, leaving a narrow post-commit window where a cache write failure could return a 503 even though the PENDING payment was created. The 202/PENDING model and the unique constraint narrow this, but a client that retries with a different idempotency key after that specific 503 could still create a second payment record. It's a known, scoped follow-up.

Challenge 3: Choosing a Different Locking Strategy for Every Entity, Deliberately

One strategy everywhere is the easy path and the wrong one: pessimistic across the board needlessly serializes work where there's no contention; optimistic across the board would let concurrent account debits race into an overdraft. Validating the account-side choice required a dedicated concurrency test: 10 concurrent threads against a single account with a fixed starting balance, asserting the balance never went negative at any point, not just that the final number happened to look right.

Challenge 4: Shipping Cleanly on a Framework Generation Still Settling Its Own Conventions

Building on Spring Boot 4.0.7 rather than the well-trodden 3.x line meant migration issues with thin or nonexistent documentation, several failing in ways that are easy to miss. The most dangerous: the AOP starter was renamed, and importing the old spring-boot-starter-aop instead of the new spring-boot-starter-aspectj produces no build error. The service starts up fine and silently ignores every @CircuitBreaker and @Retry annotation: no exception, no warning, a worse failure mode than a loud one because the system appears protected when it isn't.

Others: spring-boot-starter-web became spring-boot-starter-webmvc; Flyway's PostgreSQL support split into a separate flyway-database-postgresql artifact that must be added alongside the starter or migrations fail with an "unsupported database" error; Jackson 3 relocated its package namespace to tools.jackson.*; and a Spring Security constant (DEFAULT_FILTER_ORDER) moved to a different class. None are individually hard once identified; the work was recognizing a symptom as a rename or split rather than a genuine bug, then documenting each.

---

Outcome and What It Demonstrates

PayFlow passes 84 tests across 11 classes: pure unit tests (including the state-machine verification against an independently written oracle across all 79 from-to-legality combinations), Mockito service tests that verify not just what a method returns but what it persists, and integration tests against real PostgreSQL and Redis via Testcontainers. The concurrency guarantee isn't just claimed: a dedicated test fires 10 concurrent threads at a shared account balance and asserts it never goes negative.

From an engineering standpoint, the project demonstrates:

Correctness Engineering for Financial Systems. The two-transaction processor pattern, the fail-closed idempotency stack, and the append-only ledger are each a purpose-built answer to a named failure mode (crash mid-charge, retry storm, silent balance mutation), not a generic best practice applied by default.

Concurrency Primitive Selection, Not Cargo-Culted Locking. Choosing pessimistic locking for accounts and optimistic for payments, and articulating why each access pattern calls for its primitive, is the difference between understanding concurrency control and knowing its vocabulary.

Distributed Systems Reasoning. SKIP LOCKED as a work queue and ShedLock with database-time leases both prevent duplicate work across horizontally scaled instances, with explicit reasoning about node clock skew.

Test Rigor Beyond the Happy Path. Oracle-based state-machine testing, ArgumentCaptor-verified persistence assertions, negative-space assertions (never().save()), and real-infrastructure integration tests reflect a philosophy of proving correctness properties, not just exercising code paths.

Honest Engineering Communication. The documentation names its residual limitations directly: the idempotency dual-write window, the HS256-over-RS256 tradeoff for a single-service dev context, the single-node Redis that wouldn't satisfy Redlock's real safety property, and hardening left out of scope.

Closing a Portfolio Gap With Real Substance. The professional payment work behind this project lives in a private codebase that can be discussed but not shown. PayFlow makes the underlying engineering (correctness under concurrency, crash safety around external dependencies, idempotent request handling) visible, testable, and citable.

---

Tech Stack Summary