Overview
two types
- product
- infrastructure
Steps
- Requirements
- Core Entities
- API
- High-level Design
- Deep Dives
Requirements
Functional Requirements - features Non-functional Requirements - user experience
- Latency
- CAP Theorem - strong consistency vs high availability
- Strong consistency for xyz feature
- High availability for abc feature
- ex: booking system (working during network partition)
- strong consistency
- when user tries during parallel acid write, user sees error (lost availability)
- Read Write Ratio
- Scale (customize for app)
- Scalability to handle surges from popular events
- Fault tolerance
Core Entities
- 1 entity maps to 1 service in microservices
- ex: booking system
- Event
- Venue
- Ticket
- Performer
API
- METHOD host/resource?key=val
High Level Design
Real Time (kappa)
- Apache Kafka Broker
- Event stream
- Pubsub broadcast
- Flink
- Aggregator (real time, quick, may go down (use lambda to reconcile)) Batch (lambda)
- Spark
- Aggregator (map reduce) Databases (lambda)
- Cassandra
- Write heavy
- DynamoDB
- Read heavy
- Eventual Consistency
- Postgres
- ACID
- Strong Consistency
- OLAP
- Analytics
- ElasticSearch
- Search
- Inverted index
Deep Dives
Latency
- CDN -> cache api calls / static (common requests, bad for unique)
- ElasticSearch -> search
- gRPC -> server to server using protobufs
- SSE / WebSockets -> UI to server Fault Tolerance (data integrity)
- Message Queue (offset / cursor) Scale
- API GW -> auth / rate limiting / routing
- Horizontal scaling
- Shard
- Hot spots -> shard the shard
- Hot spots -> buffer / message queue (pq)
- Cache to DB (to reduce read load)
- UI to CDN -> cache api