Saturday, 14 March 2015

Cloud messaging with Camel and IronMQ

About 2 years ago I was on the lookout for a way of doing messaging between our internal integration infrastructure, and our Drupal cms system that runs on aws. Preferably the messaging solution should be SAAS.

Since we are using Camel for integration an obvious choice would be to use aws SQS, and the Camel sqs component for that, but message size limits and other missing QOS features that you are used to when working with message brokers made me google a bit around to look for other options.

I found IronMQ to be a good match since it had some nice features compared to SQS, especially FIFO and push queues.

Since Camel and the community around it is awesome, and it's fairly easy to create/share your own components I created a camel-ironmq component so IronMQ could be part of the camel route dsl like this simple file copy route shows.

About half a year ago another use case came up. We had to post time critical events to business partners for them to be able to start and stop recording live video broadcast streams. It should be easy to just add/remove authenticated partners as the need may arise.
The IronMQ push queue feature seemed a good fit, since you are able to fan out the messages you send to a IronMQ queue to a configurable number of http/webhook url's.

In the mean time IronMQ had moved somewhat, and they are working on a new V3 release with new features and better performance.
Some api changes had been introduced, and I had to brush up the component, and at the same time introduce some missing features.
Thats done on the v3 branch where also longPolling, batchDelete and concurrentConsumer support is added.
Christian Posta was so nice to make the concurrentConsumer support easy in the latest Camel 2.15.0 release as he blogged about here.    

Example of v3 route:
The tests I have done shows that you are able to consume 2500 msg/sec with 20 concurrent consumers and batchDelete turned on.

If you want to try out camel-ironmq v3 component, you have to compile the IronMQ 3.0.2-SNAPSHOT since all fixes havn't been released yet, and then upgrade to that version in camel-ironmq pom.

Currently I think only IronMQ v2 is public available for developer testing. For v3 you have to have a payed account.