You do not need multiple Producers in a single service

Producer is Thread Safe you can use its single instance. Now let’s talk about the benefits of single Producer instance. It batches your record and it also batches batches of records to minimize the number of request to the broker servers. Single Producer Instance gives you low number of requests (read it connection) to the broker server which means better performance. If you publish 100 records with many Producer Instance versus single Producer Instance you may be firing many requests to the broker which is giving you less performance.

Check % of records per request in Kafka Producer metrics

Adjust to get better percentage. Get the batch size higher in the beginning. Fie tune later.

Not enabling complression

Compressio is not enabled by default.

Not hadnling CallBack which is given to Producer.produce() method

You can figure out

  • message failures
  • back pressure
  • schema related issues
  • performance of Producer