Case study 02
~85% reduction in worst-case runtime: 6 hours → ~45 minutes
Overview
The platform processed clickstream, cart and purchase events for approximately 500 mid-market e-commerce brands.
It provided analytics around product performance, funnel conversion, cohort retention, and customer behavior.
The data platform processed approximately 3–4 billion events/day and 15–20 TB/day raw data.
Business challenge
Two important daily reports had a 2-hour SLA: Product Performance and Funnel Conversion. They were required before customers’ morning business meetings.
Impact
Morning business meetings depended on Product Performance and Funnel Conversion landing inside a 2-hour SLA.
Late reports and weekly OOM failures eroded trust in the analytics product itself.
Throwing more cluster capacity at skew would raise cost without fixing the root cause.
Engineering
Technical constraints that shaped the architecture and delivery approach.
01
The obvious solution would have been to increase cluster capacity. Instead, I investigated the Spark execution behavior using the Spark UI and History Server — task duration distributions, partition sizes, key frequency, executor utilization, and shuffle behavior.
02
Approximately 30% of traffic was unauthenticated and concentrated around a small number of placeholder user IDs. A normal events → users/sessions join produced extremely uneven partitions. Some partitions grew to tens of gigabytes while most remained hundreds of megabytes or smaller. A handful of Spark tasks therefore ran for hours while the rest of the cluster waited.
03
The optimization couldn’t simply make the job faster. The pipeline had strict downstream requirements around user/session deduplication, exact event counts, and aggregation correctness. The salting implementation therefore had to preserve the original semantics.
Ownership
I owned the investigation and the production optimization path:
Solution
Rather than buying more capacity, the work fixed the skew that made a few tasks dominate wall-clock time — while proving output correctness in shadow mode.
I separated heavily skewed keys from normal keys. The skewed path distributed heavily concentrated keys across deterministic salt buckets. The normal path continued using the standard join.
I enabled and tuned Spark’s Adaptive Query Execution to allow the engine to react to the actual runtime workload.
A small lookup table was being processed through a sort-merge join. I changed this to a broadcast join, eliminating unnecessary shuffle work for that part of the pipeline.
Cluster sizing and dynamic allocation were adjusted to better match the workload rather than compensating for the underlying skew with additional compute.
I used deterministic salt buckets, controlled reassembly before final aggregation, and shadow-mode execution. The new pipeline ran alongside the old implementation for approximately one week, allowing output comparisons before production cutover.
Architecture
Hot keys take a salted path; normal keys keep a standard join — then reassemble for final aggregation.
Comparison
| Area | Before | After |
|---|---|---|
| Runtime | 4–6 hours | ~45 minutes |
| SLA compliance | ~60% | >99% |
| OOM failures | ~2/week | ~0–1/month |
| Compute cost/run | Baseline | ~40% lower |
| Cluster utilization | Highly uneven | More evenly distributed |
Results
Worst-case runtime
6h → ~45m
~85% reduction
SLA compliance
~60% → >99%
OOM failures
~2/week → ~0–1/month
Compute cost/run
~40% lower
The salting approach was subsequently turned into a reusable internal Spark utility, and a skew-detection check was added to help identify problematic key distributions before new jobs reached production.
Technology
Listed last on purpose — technology supported the business outcome; it was not the story.
More work
Case 01
Publishing → distribution → commerce → payment → reading in one production system
Digital Publishing / E-commerce
Case 03
Chat startup ~30 seconds → ~5 seconds (~1 second when cached)
Healthcare / Care Coordination
Case 05
New HR capability inside OfficeAnyWhere — not a disconnected HR product
Enterprise / HR
Next step
Talk to Solastia Systems about the business outcome you need — architecture, delivery, and measurable results follow from that.