나의 발자취

Food Delivery Service (WooWa Bros) system architecture 본문

시스템 디자인

Food Delivery Service (WooWa Bros) system architecture

달모드 2026. 2. 21. 01:50

 

 

These sources detail the sophisticated technical strategies employed by Woowa Brothers to maintain the stability and performance of the Baedal Minjok ecosystem (ref: https://techblog.woowahan.com/2667/). The documentation highlights the "Shop Exposure System," which utilizes Spring WebFlux, Redis, and DynamoDB to manage massive traffic through reactive programming and asynchronous non-blocking I/O. Another key focus is the "Ad Listing System," where Kotlin DSLs and Coroutines are used to simplify complex query logic and improve developer productivity. To ensure system resilience, the engineering teams implement circuit breakers, vendor redundancy, and fault isolation to prevent external service outages from degrading the user experience. By conducting regular dependency audits and adopting a CQRS pattern, the company manages the inherent risks of a large-scale microservices architecture. Ultimately, these texts illustrate a continuous commitment to infrastructure optimization and proactive monitoring to handle the volatile demands of real-time delivery services.

Preface: This is a reflection from a study group I participate in biweekly, in person.

In Week 1 (Session 1), we examined the live shopping event on Toss — a Korean fintech app that leverages open banking (My Data) and also offers stock trading, an expense tracker, and various credit card promotions. I had done this session on paper and hadn't transferred my notes to a computer, so afterward I redrew the architecture digitally to get AI feedback and continue the discussion.

 

This session was centered around designing a high-traffic store listing system — loosely modeled after real-world platforms like Baedal Minjok. As the moderator, I guided the group through  1) Functional Requirements, 2) Non-Functional Requirements, and 3) System Architecture.

 

1) Functional Requirements were split into frontend and backend:

 

Frontend

  • Store list
  • DSL store info (thumbnails, etc.)
  • Delivery zone detection and delivery fee filtering logic
  • Ad banners (curation)

Backend

  • Policy service

 

2) Non-Functional Requirements were broken into three areas: Performance, Availability, and Scalability.

Performance — Scale Estimation

  • DAU: 10 million
  • QPU: 579
  • Peak QPS: 100,000
  • Registered stores: 1 million
  • Response time: ~200ms

Availability: 99.999% Scalability target: 30 million DAU

The most interesting part was designing the 3) System Architecture. The key discussion questions we wrestled with were:

  • Which DB type should we use as the primary DB — synchronous or asynchronous?
  • What replication architecture should we apply to the RDB?
  • If we go with Load Balancer → Cache → Server → RDB, and then separate reads via CQRS, should Elasticsearch be treated as an independent search layer or appended to the read DB? (This question surfaced the fact that most of us had only used ES for search functionality, not as a primary data store.)
  • Should we split the RDB into Master/Slave, or is there a better alternative?
  • How does the cost of filtering in RDB compare to Elasticsearch, and why?
  • When choosing between Elasticsearch and DynamoDB as the primary DB, what are the tradeoffs?
  • What are the tradeoffs of using Elasticsearch as the primary DB rather than an RDB?
  • Instead of using store ID as the PK, what if we used geospatial data (like a QuadTree, similar to Google Maps) as the cache key — wouldn't that improve accuracy? And rather than putting everything in a single cache, what if we split it into four layers: top-ranked locations, detailed filtering, personalization, and real-time state changes?
  • If ES is used for caching and synced periodically with the RDB, what is the data retention span in ES? TTL? But it doesn't make sense to delete data after a fixed interval — searches would break.
  • If we split caching into popular areas (LFU), detailed filtering, personalization, and real-time state changes — how many cache instances do we need, and what does that structure look like?
  • If we set the PK to lat/lng coordinates by region — users within the same zone should see the same results, and this approach seems more semantically correct.
  • How exactly does distributed processing work using Elasticsearch nodes and shards?
  • Where should the cache sit, and what spec should it use?

 

This architecture diagram is the final output of our discussion.

 

 

 

After the session, I asked a generative AI to identify and correct any missing or incorrect parts of the architecture.

 

 

Self-Reflection: CQRS + CDC Architecture Study Session

What I Think Went Well

The decision to adopt CQRS with a CDC pipeline was the right call. At a peak QPS of 100,000, a single RDB handling both reads and writes would inevitably become the bottleneck. Separating the write path (server → Relational DB) from the read path (Worker → Redis / Elasticsearch → Local Cache) is a pattern that production systems at this scale actually use, and I'm glad the group landed on it organically rather than being told the answer.

The idea of splitting the cache into four distinct buckets — Popular Areas (LFU-based), Detailed Filtering, Personalization, and Real-time State Changes — also felt like a meaningful insight. Rather than dumping everything into one cache layer and hoping for the best, this approach respects the fact that different data has different access patterns, different TTL requirements, and different tolerance for staleness. Separating Real-time State Changes into its own Pub/Sub-driven bucket in particular shows an understanding that open/close status updates cannot be treated the same way as relatively stable store metadata.

Choosing Debezium → Kafka → Worker as the synchronization backbone was also well-reasoned. Polling the database for changes is fragile and expensive. CDC via binlog capture is how this problem is actually solved at scale, and the group arrived at this without it being handed to them.

Where I Was Wrong or Shallow

1. The Numbers Lack a Derivation

We stated DAU of 10 million, peak QPS of 100,000, and QPU of 579 — but I cannot clearly explain where QPU 579 came from. In any serious design review or technical interview, the derivation matters more than the number itself. The expected reasoning looks something like:

10,000,000 DAU
× avg. requests per session (e.g., 5)
/ 86,400 seconds
≈ ~578 avg QPS

Peak QPS = avg QPS × spike multiplier (e.g., ×150 during lunch rush)
≈ 86,700 → rounded to 100,000

Without walking through this explicitly, the numbers are just decoration. I need to be able to justify every figure I put on a design document.

2. I Was Confused About What Elasticsearch Actually Is

The question "what if we use ES as the primary DB?" came up in our session, and I didn't shut it down cleanly enough. The answer is: you can't, and you shouldn't. Elasticsearch does not support ACID transactions. It doesn't guarantee durability the way a relational database does. A partial update internally triggers a delete and reindex. It is a search index built on top of a document store — it is a derived view of your data, not the source of truth.

The correct mental model is: the Relational DB owns the truth, and ES reflects it. Elasticsearch answers questions like "which stores match this filter set sorted by distance?" It does not replace the system that answers "what is the canonical state of store #4821?"

Going forward, I need to be precise about this distinction every time ES comes up in a discussion.

3. The QuadTree / Geospatial PK Idea Was Undercooked

The suggestion to use geospatial data — specifically QuadTree-based region classification — as a cache key had the right instinct behind it: users in the same geographic zone should see the same results, so keying the cache by region rather than by individual user makes sense. However, the way I phrased it ("use lat/lng as the PK") is technically incorrect and would be a red flag in a design review.

Raw latitude/longitude coordinates are floating-point numbers. They're noisy. Two users standing ten meters apart will generate different coordinates, and floating-point precision differences mean you cannot reliably use them as cache keys. The correct approach is to discretize the space first — hash the coordinates into a fixed-size cell using a scheme like Geohash, Google S2, or Uber's H3. Then that cell ID becomes the cache key.

The distinction is: QuadTree is a data structure for spatial indexing and range queries. Geohash / H3 is a way to assign a stable string identifier to a geographic region. They solve adjacent but different problems. I was conflating the two, and I need to be clearer about which tool does what.

4. The Worker's Sequential Update Logic Has a Gap

We defined the update order as: ① Redis 1st → ② Redis 2nd → ③ Elasticsearch. But I never addressed the failure case: what happens if Redis update succeeds but ES indexing fails?In a distributed system, partial failures are not edge cases — they are normal operating conditions. At minimum, the design needs a retry mechanism: failed ES updates should be sent to a dead-letter queue or retry topic so they can be replayed without data loss. Without this, the system silently drifts into inconsistency and there's no way to detect it.

5. 99.999% Availability Was Written Without Infrastructure to Back It Up

Five nines means 5 minutes and 15 seconds of downtime per year. I wrote this number on the diagram without specifying what it actually requires: multi-AZ deployment, active-active configuration, automated failover under 30 seconds, and careful handling of network partitions. None of that was reflected in the architecture. If I'm going to claim 99.999%, the diagram needs to show how the system survives the loss of an availability zone. Otherwise it's a wish, not a design target.

What I'm Taking Away

The most valuable part of this session wasn't the final diagram — it was the questions the group asked that I couldn't fully answer in the moment. Those gaps are the actual learning agenda:

  • I need to internalize how to derive capacity numbers from first principles, not just state them.
  • I need a much cleaner mental model of ES: what it's good at, what it can't do, and where it fits in the data flow.
  • I need to understand spatial indexing properly — Geohash, S2, H3 — before I use geospatial data in any future design.
  • I need to design for failure, not just for the happy path. Every asynchronous update step needs a defined behavior on failure.
  • I need to treat availability targets as engineering commitments, not aspirational numbers.

The architecture we drew is directionally correct. The CQRS pattern, the CDC pipeline, the layered cache strategy — these are real solutions to real problems. But a correct direction with unexamined gaps is only halfway to a real design. The next session should revisit the failure modes, the capacity math, and the geospatial keying strategy with more rigor.

Resources I'm Going to Read

  • Designing Data-Intensive Applications — Martin Kleppmann. Covers CQRS, CDC, replication, and consistency models in depth. Most of the questions raised in this session are answered in this book. (need to buy this)
  • Elasticsearch: The Definitive Guide (https://www.elastic.co/docs/get-started). Will clear up my confusion about what ES is and isn't.
  • Uber Engineering Blog — H3: Hexagonal Hierarchical Spatial Index. The practical reference for geospatial discretization.
  • Martin Fowler — CQRS (martinfowler.com/bliki/CQRS.html). Short, authoritative, worth reading before the next session https://martinfowler.com/bliki/CQRS.html
  • Confluent Blog — Change Data Capture with Debezium. Will deepen my understanding of the CDC layer we designed.
  • Woowa Brothers Tech Blog (techblog.woowahan.com) — specifically the Store Exposure System series, again. Reading it after having designed our own version will make the differences much more instructive.
  • System Design Interview Vol. 2 — Alex Xu. The cache design chapters map directly onto what we discussed. (I have this book)
Comments