Distributed Job Scheduler

Learn how to build an asynchronous distributed Job Scheduler that handles around 50k requests per day.

This architecture is built using four components-

1. Springboot app which sends the request for scheduling
2. SQS (Simple Queue Service) which is responsible for queueing the initial load
3. Dynamo DB for storing the job scheduling information
4. Kubernetes for managing the PODs responsible for scheduling the jobs

High-level archritechture

The complete application is divided into 3 parts. In the first part, we are going to learn about why and how the Spring boot application creates an item into SQS and then how store the Job information into Dynamo DB.

In the second part, we are going to learn again why and how Dynamo DB sharding is used and how Kubernetes PODs are mapped to those Dynamo DB shardings.

Dynamo DB based scheduling