Other metadata requiring extreme scalability and fault tolerance.
Posted: Thu May 22, 2025 6:30 am
While not a traditional database for persistent storage, Kafka is a distributed streaming platform that WhatsApp heavily uses as a message queue. This is fundamental for:
Ensuring message delivery order: Messages are sent to Kafka topics to maintain FIFO (First-In, First-Out) order.
Buffering messages: Handling the immense volume of messages before they are processed by the message delivery service.
Enabling asynchronous processing: Decoupling message sending from delivery logic.
Blob Storage / Object Storage (e.g., S3, Google Cloud Storage, or Meta's internal equivalents):
Used for storing large binary objects like images, videos, audio files, and documents slovenia phone number list in encrypted form. These are typically accessed via unique IDs stored in the relational databases.
Distributed Key-Value Stores (e.g., Cassandra): Given WhatsApp's massive scale and need for high availability and partition tolerance, a NoSQL distributed key-value store like Apache Cassandra is highly probable for:
Offline message storage: For the 30-day window, Cassandra's distributed nature makes it resilient and scalable for this purpose. Each message would be tagged with recipient ID, timestamp, etc.
User-to-server mapping across data centers: For complex routing at a global scale.
Some analyses suggest Cassandra is used for general metadata storage due to its horizontal scalability.
Ensuring message delivery order: Messages are sent to Kafka topics to maintain FIFO (First-In, First-Out) order.
Buffering messages: Handling the immense volume of messages before they are processed by the message delivery service.
Enabling asynchronous processing: Decoupling message sending from delivery logic.
Blob Storage / Object Storage (e.g., S3, Google Cloud Storage, or Meta's internal equivalents):
Used for storing large binary objects like images, videos, audio files, and documents slovenia phone number list in encrypted form. These are typically accessed via unique IDs stored in the relational databases.
Distributed Key-Value Stores (e.g., Cassandra): Given WhatsApp's massive scale and need for high availability and partition tolerance, a NoSQL distributed key-value store like Apache Cassandra is highly probable for:
Offline message storage: For the 30-day window, Cassandra's distributed nature makes it resilient and scalable for this purpose. Each message would be tagged with recipient ID, timestamp, etc.
User-to-server mapping across data centers: For complex routing at a global scale.
Some analyses suggest Cassandra is used for general metadata storage due to its horizontal scalability.