apache kafka – Distributed Event collection
I have software running on end users devices (laptop,phone,desktop,etc). (Number of devices is ~100K).
The software needs to send an Event[s] (a well defined data structure).
The BackEnd of my platform needs to collect the events and push them to Apache Kafka.
Few options came into my mind:
- Write (protobuf) to Kafka directly from the end user device
- Use gRPC (with protobuf) and have a gRPC server that will forward the incoming events to Kafka. The gRPC server should not be very “smart” – it will mainly deal with routing the events to the correct Kafka topic.
The gRPC layer will add another cluster + load balancer to the platform –> extra complexity. On the other hand it does not feel right to expose the kafka to the end user devices (not to mention security and authorization)
Read more here: Source link