In summary, Kafka streams provide a way to process and analyze data in motion, and topics serve as the channels through which the data is ingested, processed, and distributed. The combination of topics and streams allows for a powerful and flexible data processing architecture in Kafka.
Kafka broker
- Node
- Responsible for persistence
- Contains leader topics and replica topics
Kafka topic
- Logical channel in a Kafka broker
- Has multiple partitions
- Leader will copy to replica
Kafka topic partition
- Partition of a topic
- Producers can emit to the topic partition
- Consumers can consume from the topic partition
- (For more info: look into ad service system)
Kafka streams
- Consumes topic and streams in real time
Dead Letter Queue
- After a timeout of no acknowledge from consumer, send event to DLQ
Consumer Groups
- Group of Consumers
- Single consumer group pattern - distribute load (each partition is assigned a consumer from the cg)
- Multi consumer group pattern - broadcast (each consumer group will receive the same message)
Consumers
- Consumer service (could be fargate, etc)
- Each partition will be assigned a consumer
- Each consumer can take on multiple partitions
Amazon Managed Streaming (AWS MSK)
- Managed Streaming for Kafka service
- Auto scaling storage capacity
- Compute capacity automatically scales up (not horizontal)
- Manual scaling number of brokers
Avro
- Messages commonly use Avro schema
Offsets
- Must have, way of keeping track where the consumer's progress is in a partition
- Typical pattern is storing offset state for all consumers in another topic (a __consumer_offsets)
Apache Kafka
