apache kafka – Distributed Event collection

I have software running on end users devices (laptop,phone,desktop,etc). (Number of devices is ~100K).
The software needs to send an Event[s] (a well defined data structure).
The BackEnd of my platform needs to collect the events and push them to Apache Kafka.

Few options came into my mind:

  1. Write (protobuf) to Kafka directly from the end user device
  2. Use gRPC (with protobuf) and have a gRPC server that will forward the incoming events to Kafka. The gRPC server should not be very “smart” – it will mainly deal with routing the events to the correct Kafka topic.

The gRPC layer will add another cluster + load balancer to the platform –> extra complexity. On the other hand it does not feel right to expose the kafka to the end user devices (not to mention security and authorization)

Any ideas?

Read more here: Source link