In one of our projects, our backend server is built by using the prevalent programming language Golang.
Communicating to databases is costly in terms of performance and resources. We should not do it unless it is really essential. So we use an in-memory cache to store the most accessed data. But, it is not as easy as it sounds. We can not rely on cached data all the time. We realized that and built an event (our event framework is built by leveraging goroutine) that syncs all the database changes to the cache-store based on the event published.
Basically, published events are consumed within the local instances and an ideal solution until there is a single instance. The problem arose, we decided to go for multi-instances of our backend server. Since only local instances can consume an event published in the event framework, there was no way to communicate with other instances.
How we can solve this using AWS SNS/SQS
Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. Simple Notification Service (SNS) is a fully managed messaging service for both application-to-application and application-to-person communication.
Use Case of Broadcast Queue
It broadcasts the event to all replicas. The typical use case is updating the in-memory cache for all replicas.
Use Case of Event Queue
We publish and consume the event within a single replica using an internal goroutine.
The idea is to replace the internal publish with remote publish and consume events in FIFO order. By publishing to a remote queue, we ensure that the local replica is stable and the load is distributed across all the replicas.