Why should you store Integration

messages

When it comes to integration, message queues come in handy enabling us to decouple the systems, ensuring guaranteed delivery and helping rate control. In the figure below, the message path with the queue is more durable than the direct path.

However nothing is perfect, “Everything fails all the time,” says Werner Vogels the CTO of Amazon. So it’s good to have precautionary measures being taken to face any failures. The obvious requirement when dealing with failures is to know what, when and how it has happened and how to recover from it. This is where logs come in handy, If you have maintained the logs or have stored the messages it’s a matter of checking the logs, identifying the issue, correcting it and replay.

Problem : How to store messages while in transit?? One simple solution is keep another(mirror) message queue and store messages there. However message queues are not designed for storage, an integration flow is healthy if your queues are (nearly) empty. On the other hand this is not possible as many message queues does not support long message retention times. As for an example AWS SQS message retention period is 14 days.

One simple approach would be to store a copy of messages to a database or file. But how? In the AWS world we can configure a Lambda function to listen to queue and store the messages to a DB or blob storage. The figure below shows the flow digram and you can find the code at the bottom of this post.

Why is there a Topic? We talked about using a queue right?. Yes, this is because AWS Lambda does not support triggers from Queues, but it can be triggered by topics. We can implement the same with queue, however then we might have to schedule the Lambda function to periodically poll the queue. Also when we read the queue with Lambda we have to make sure NOT to acknowledge as read, and while Lambda reads this our intended endpoints would have to wait for a (Visibility Timeout) time to see the message again. Those were the reasons to go with a Topic (publish/subscribe) model here.