Jms to Kafka Bridge

Often we integrate two systems where one system is emitting events on some jms solution while another is listening a kafka topic and expecting source messages on kafka. In such cases, we write a bridge application which reads data from jms and writes on kafka.

This bridge application should satisfy following requirements -

• It should send messages on kafka without breaking the sequence of message.
• If jms broker goes down, it should keep trying to connect to that and whenever broker starts, application should reconnect and should start listening messages.
• If jms broker balances its nodes (shift master node from one to another), application should reconnect and should start listening messages.
• If kafka goes down, it should keep trying to connect to that and either should stop reading messages from queue or should keep holding those messages which couldn’t be send to kafka. Whenever kafka broker starts, application should reconnect and should start sending messages.
• If application goes down, it should do that gracefully - should stop listening new messages and should finish processing of messages which are already under process.
• In no case, a message should be lost.

Next, we’ll be discussing 3 examples using different technologies. We are using Apache Qpid as JMS broker.

1. JMS to Kafka using Spring JMS
2. JMS to Kafka using Apache Camel
3. JMS to Kafka using Apache Storm

1. JMS to Kafka using Spring JMS

This code can be found here - https://github.com/prashantbhardwaj/qpid-to-kafka-using-spring-jms

DefaultMessageListenerContainer is used to create connection with Qpid and KafkaTemplate is used to send message to Kafka.

2. JMS to Kafka using Apache Camel

This code can be found here - https://github.com/prashantbhardwaj/qpid-to-kafka-using-camel

3. JMS to Kafka using Apache Storm

This code can be found here - https://github.com/prashantbhardwaj/qpid-to-kafka-using-storm