Problems of Event Sourcing Pattern

In one of our projects, our backend server is built by using the prevalent programming language Golang. 

Communicating to databases is costly in terms of performance and resources. We should not do it unless it is really essential. So we use an in-memory cache to store the most accessed data. But, it is not as easy as it sounds. We can not rely on cached data all the time. We realized that and built an event (our event framework is built by leveraging goroutine) that syncs all the database changes to the cache-store based on the event published.

Basically, published events are consumed within the local instances and an ideal solution until there is a single instance. The problem arose, we decided to go for multi-instances of our backend server. Since only local instances can consume an event published in the event framework, there was no way to communicate with other instances. 

How we can solve this using AWS SNS/SQS

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. Simple Notification Service (SNS) is a fully managed messaging service for both application-to-application and application-to-person communication.


Use Case of Broadcast Queue

It broadcasts the event to all replicas. The typical use case is updating the in-memory cache for all replicas.

Use Case of Event Queue

We publish and consume the event within a single replica using an internal goroutine.

The idea is to replace the internal publish with remote publish and consume events in FIFO order. By publishing to a remote queue, we ensure that the local replica is stable and the load is distributed across all the replicas. 

Ram Bhandari and Dibesh Man Shrestha

Ram Kumar Bhandari and Dibesh Man Singh Shrestha co-authored this blog. They are both software engineers at Leapfrog. 

More in Blogs

MVS: Digital Solution for COVID Vaccination Appointment Scheduling InsightsTechnology

MVS: Digital Solution for COVID Vaccination Appointment Scheduling

Problem Statement Mankind is facing one of the biggest challenges in recent history due to the spread of the Covid-19

Read more
How to Automate your Work From Home: Puppeteer Use Case InsightsTechnology

How to Automate your Work From Home: Puppeteer Use Case

In this article, we will achieve task automation by writing some code using Node and we will make use of the somewhat non-traditional method of using a headless browser.

Read more
8 DevOps Tools and Services We Love Technology

8 DevOps Tools and Services We Love

The core focus of any DevOps team is evolving and improving products rapidly. At Leapfrog, we value the speed at

Read more