Often we integrate two systems where one system is emitting events on some jms solution while another is listening a kafka topic and expecting source messages on kafka. In such cases, we write a bridge application which reads data from jms and writes on kafka.

alt text

This bridge application should satisfy following requirements -

  • It should send messages on kafka without breaking the sequence of message.
  • If jms broker goes down, it should keep trying to connect to that and whenever broker starts, application should reconnect and should start listening messages.
  • If jms broker balances its nodes (shift master node from one to another), application should reconnect and should start listening messages.
  • If kafka goes down, it should keep trying to connect to that and either should stop reading messages from queue or should keep holding those messages which couldn’t be send to kafka. Whenever kafka broker starts, application should reconnect and should start sending messages.
  • If application goes down, it should do that gracefully - should stop listening new messages and should finish processing of messages which are already under process.
  • In no case, a message should be lost.

Next, we’ll be discussing 3 examples using different technologies. We are using Apache Qpid as JMS broker.

  1. JMS to Kafka using Spring JMS
  2. JMS to Kafka using Apache Camel
  3. JMS to Kafka using Apache Storm

1. JMS to Kafka using Spring JMS

This code can be found here -
https://github.com/prashantbhardwaj/qpid-to-kafka-using-spring-jms

DefaultMessageListenerContainer is used to create connection with Qpid and KafkaTemplate is used to send message to Kafka.

Test case Pass/Fail Remark
no out of sequence message not tested yet
reconnect on jms broker start up not tested yet
reconnect on jms broker node shuffle up not tested yet
reconnect on kafka start up not tested yet
graceful application shutdown not tested yet
no message loss not tested yet
performance x msgs/sec

2. JMS to Kafka using Apache Camel

This code can be found here -
https://github.com/prashantbhardwaj/qpid-to-kafka-using-camel

Test case Pass/Fail Remark
no out of sequence message not tested yet
reconnect on jms broker start up not tested yet
reconnect on jms broker node shuffle up not tested yet
reconnect on kafka start up not tested yet
graceful application shutdown not tested yet
no message loss not tested yet
performance y msgs/sec

3. JMS to Kafka using Apache Storm

This code can be found here -
https://github.com/prashantbhardwaj/qpid-to-kafka-using-storm

Test case Pass/Fail Remark
no out of sequence message not tested yet
reconnect on jms broker start up not tested yet
reconnect on jms broker node shuffle up not tested yet
reconnect on kafka start up not tested yet
graceful application shutdown not tested yet
no message loss not tested yet
performance z msgs/sec