Cloud-Native Premium Calculation Engine

Overview

Premium calculation is one of LTM's most critical operations. Every quote, renewal, and mid-term coverage change triggers a recalculation, and across 1M+ users that means thousands concurrently at peak.

The engine I inherited had two problems, both rooted in a design not built for concurrency: under load it produced intermittently incorrect results, and latency at high concurrency was user-visible.

I diagnosed both through profiling, fixed correctness by structurally eliminating shared mutable state (not by patching with locks), fixed performance through parallelized fetching and Redis caching with programmatic invalidation, and validated under 3,000-session JMeter load.

---

The Problem

What the Engine Does

The engine computes a policy's premium from several inputs: the policyholder's risk profile, coverage type and level, actuarial rate tables, jurisdiction adjustments, account discounts/surcharges, and effective dates. Each comes from a different data source (policy management, actuarial store, account service, jurisdiction rules).

Correctness is non-negotiable: the number the engine produces is the number on the policyholder's contract. A race condition that yields a wrong premium under load isn't a UX problem; it's a billing error on an insurance contract.

The Correctness Problem: Shared Mutable State

The original engine stored intermediate values (rate lookups, discount calculations, jurisdiction factors) as instance variables on Spring-managed singleton beans rather than request-scoped locals.

At low concurrency this worked. As load climbed, interleaving became frequent: Request A writes a jurisdiction factor to shared state, Request B overwrites it mid-calculation, and A finishes using B's factor, producing the wrong premium.

The failures were intermittent and non-deterministic, so they didn't reproduce in low-concurrency test environments. They surfaced through anomaly detection on production billing data, and tracing them to a concurrency bug took significant investigation.

The Performance Problem: Sequential Data Fetching

Fetching was fully sequential: risk profile, then rate tables, then account discounts, then jurisdiction rules, each blocking on the previous. Every network call parked its thread for the full duration.

At 3,000 concurrent sessions the thread pool saturated with blocked threads, requests queued, and latency climbed. The engine wasn't CPU-bound (the arithmetic was trivial); it was I/O-bound on fetching that could have been parallelized.

---

The Investigation

Before any fix, I profiled to confirm the bottleneck, and the result changed the plan.

The initial hypothesis was that the actuarial formulas (multiplications across large rate tables) were slow. A week of profiling under simulated load disproved it: computation finished in single-digit milliseconds, while the I/O calls each took 50–150ms and ran sequentially. Total wall time under load was 400–600ms, almost entirely I/O wait. Optimizing the computation, the original plan, would have changed nothing.

The correctness investigation took a different path: comparing flagged production outputs against manually computed expected values, then adding thread-local logging. The shared mutable state became visible once the logging showed concurrent requests touching the same instance variables within one calculation window.

---

Technical Architecture

Fix 1: Eliminating Race Conditions Through Stateless Request Design

The instinct on a concurrency bug is to add synchronization (a lock, an AtomicReference, a thread-safe collection). That works, but lock contention becomes a new bottleneck under high concurrency, reintroducing the throughput problem.

The better fix was to eliminate shared state entirely. I introduced a PremiumCalculationContext value object: an immutable container holding all inputs, intermediates, and results for one request. Every field that had been a shared instance variable now lives on this per-request context.

Calculation methods now accept a context and return an updated one (functional style, no instance-variable reads or writes in the pipeline). There is nothing to synchronize because there is nothing shared.

// Before: shared mutable state on singleton service
@Service
public class PremiumCalculationService {
    private BigDecimal baseRate;          // shared instance variable
    private BigDecimal jurisdictionFactor; // shared instance variable

    public BigDecimal calculate(PolicyRequest request) {
        this.baseRate = rateTableService.lookup(request.getCoverageType());
        this.jurisdictionFactor = jurisdictionService.getFactor(request.getJurisdiction());
        // ... concurrent requests clobber each other here
        return this.baseRate.multiply(this.jurisdictionFactor);
    }
}

// After: isolated value object per request
@Service
public class PremiumCalculationService {
    public PremiumCalculationContext calculate(PolicyRequest request) {
        PremiumCalculationContext ctx = PremiumCalculationContext.forRequest(request);
        ctx = applyBaseRate(ctx);
        ctx = applyJurisdictionFactor(ctx);
        // ... each request operates on its own isolated context
        return ctx;
    }
}

Race conditions were now structurally impossible. The same inputs from concurrent requests always produce the same output, guaranteed by the absence of shared state rather than by synchronization logic that could be misapplied.

Fix 2: Parallelizing Independent Data Fetches with CompletableFuture

With calculation now operating on a per-request context, the fetching that populates it became the performance target. The four data sources have no dependency on each other, so sequential fetching imposed wait time for no reason.

I parallelized all four fetches with Java's CompletableFuture:

public PremiumCalculationContext fetchInputsParallel(PolicyRequest request) {
    CompletableFuture<RiskProfile> riskFuture =
        CompletableFuture.supplyAsync(() -> riskProfileService.fetch(request.getPolicyId()), executor);

    CompletableFuture<RateTable> rateFuture =
        CompletableFuture.supplyAsync(() -> rateTableService.lookup(request.getCoverageType()), executor);

    CompletableFuture<DiscountSummary> discountFuture =
        CompletableFuture.supplyAsync(() -> accountService.getDiscounts(request.getAccountId()), executor);

    CompletableFuture<JurisdictionFactor> jurisdictionFuture =
        CompletableFuture.supplyAsync(() -> jurisdictionService.getFactor(request.getJurisdiction()), executor);

    CompletableFuture.allOf(riskFuture, rateFuture, discountFuture, jurisdictionFuture).join();

    return PremiumCalculationContext.builder()
        .riskProfile(riskFuture.join())
        .rateTable(rateFuture.join())
        .discounts(discountFuture.join())
        .jurisdictionFactor(jurisdictionFuture.join())
        .build();
}

All four initiate simultaneously and allOf(...).join() waits before calculation. Fetch-phase wall time drops from the sum of the four sequential fetches to roughly the slowest one.

The futures run on a dedicated bounded executor, not the common fork-join pool, so they don't compete with other application work; pool size was tuned from the profiling data for parallelism at peak without excessive contention.

Fix 3: Redis Caching with Programmatic Invalidation

Some sources change infrequently relative to request volume: rate tables on actuarial review cycles (monthly or quarterly), jurisdiction rules with regulations. Fetching these from origin every request was redundant latency.

I added Redis caching for the stable sources via Spring's @Cacheable. Cache keys scope values correctly:

rate-table:{coverageType}:{effectiveDate}
jurisdiction-factor:{jurisdictionCode}:{effectiveDate}

The effectiveDate component is critical: an update for a new date automatically misses the cache rather than serving stale data, preventing staleness structurally without relying on application code to invalidate the right entries.

TTL alone wasn't sufficient: with a 24-hour TTL, a mid-month rate update could serve stale rates (and wrong premiums) for up to 24 hours. So I added programmatic invalidation: the rate table service publishes an event on every committed update, and a listener clears the affected entries immediately. The TTL remains a backstop, but correct data is available the moment an update commits.

Risk profile and account discount data change more frequently and are not cached at this layer; caching them would add more staleness risk than it removes latency.

Infrastructure: AWS ECS Fargate with Stateless Horizontal Scaling

The stateless design that fixed the race conditions also made the engine cleanly horizontally scalable. With no calculation state in the process (each request carries its own context, Redis holds reference data externally), any number of ECS Fargate tasks can handle any request without inter-instance coordination.

Task count scales on CloudWatch CPU metrics; at peak the auto-scaling policy adds tasks. Because every instance is stateless and reads the same Redis cache, scaling is transparent, with no sticky sessions or session affinity.

OAuth2 / JWT is enforced at the API gateway before any request reaches the service. The service verifies the JWT signature and extracts identity and scope claims, but authentication is handled at the boundary.

---

Key Technical Challenges

Challenge 1: Eliminating Shared State Without Regressing Correctness

Refactoring a production engine requires proving the new version matches the original for every valid input, not just the ones you tested. Faster-but-occasionally-wrong is worse than slow-but-correct.

I ran both engines against the same historical production requests, across thousands of cases spanning every coverage type, jurisdiction code, policy configuration, and discount combination. The refactored engine matched the original on every input it got right, and produced the correct result on every input it had gotten wrong under load. No regressions.

Challenge 2: Cache Invalidation Timing in a Financial Context

Cache invalidation is hard everywhere, sharper in a financial context: a stale entry computing on last month's rate tables is a contractual pricing error, not a minor inaccuracy.

Event-driven invalidation depends on the service publishing events and the listener processing them, and either half can fail. The mitigation was defense in depth: the TTL backstop bounds staleness if an event is missed, effective-date keys keep future-dated updates from polluting current entries, and the listener emits a CloudWatch metric on failures so missed invalidations are detectable.

The remaining staleness window (commit to event processing) measured in milliseconds under normal conditions.

Challenge 3: Validating Concurrency Correctness, Not Just Performance

The concurrency failure mode (wrong results from thread interleaving) doesn't show up in latency, error rate, or throughput. A load test measuring only those passes even if the engine is still producing intermittently wrong premiums.

So the JMeter tests validated correctness under load. For a subset of sessions, the test injected configurations with precomputed expected premiums and, under 3,000-session load, asserted every response matched. Across all runs, zero deviations: the correctness guarantee held under full concurrent load.

---

Outcome and What It Demonstrates

The refactored engine eliminated the production race conditions, cut data-fetch latency through parallelization, and offloaded peak traffic from upstream services via Redis caching. JMeter validation confirmed correctness under 3,000-concurrent-session load: the pre-refactor failure mode was absent.

The project demonstrates:

Concurrent Systems Reasoning. Tracing a race condition to shared state on singleton beans and fixing it structurally through stateless design. Adding locks is the instinct; eliminating shared state is the engineering.

Profiling Before Optimizing. Finding the bottleneck was sequential I/O, not computation, after a week on the wrong hypothesis, then pivoting to the correct fix.

Cache Design for a Financial Domain. Effective-date keys, event-driven invalidation with a TTL backstop, and not caching frequently-changing data, where stale data has contractual consequences.

Correctness Validation Under Load. Load tests that assert output correctness via injected known-answer requests, not just performance. Latency alone would not have proven the race conditions were gone.

---

Tech Stack Summary