We currently have a REST API service (let's call it as A) written in Python which triggers various heavy calculation jobs for Worker (W) written in Python as well. Both services are interconnected using a self-written PostgreSQL queue lib.

That's how they communicate at the moment:

```
 1. Call POST request to service A to create a Job in PG Queue
 2. Worker takes a job from Queue and runs calculations
 3. After a successful calculations the worker saves results in the shared DB 
- 
 1. External service B requests every N seconds service A to take the data calculated in the W by job id
 2. Service A observes shared DB and returns the data once the worker is finished (status=Done for a particular job)

```

[![old way architecture](https://i.stack.imgur.com/vz4QW.png)](https://i.stack.imgur.com/vz4QW.png)

**We have decided to rewrite the Python service A into Node.JS (NestJS) and improve the current implementation in any way.** As you may have noticed, there's line `External service B requests every N seconds`. We want to get rid of this pattern as well and use something more efficient and performant for this case.

### 1. Concept using Kafka

The first idea is to use some message queue service like Kafka:

[![concept-using-kafka](https://i.stack.imgur.com/6ADwp.png)](https://i.stack.imgur.com/6ADwp.png)

I like this approach since we have a separated worker which only executes some calculations (jobs) and Kafka which plays the role of distributing messages between services. This approach could also get rid of interval/polling requests for taking an actual data.

At the same time it has several possible drawbacks:

* Imagine the Worker runs only on single thread and, accordingly, can perform one job at a time. How to handle multiple messages at the same time from Kafka in the worker if there is already running job? Do we need to create a RAM Queue or Kafka can handle these cases?
* What if Kafka or service that consumes messages would suddenly shutdown, will the transmitted data be lost?
* Maybe use another appropriate MQ such as RabbitMQ instead of Kafka?

### 2. Concept using PG Queue ... with Kafka

Since we are rewriting the service from scratch using a new technology, we can improve the original approach by replacing the interval requests for getting the data with Kafka. So, the interaction scheme will look like this:

```
 1. External service B consumes all messages from a particular topic
 2. Service A observes DB and sends data to the topic once the worker is finished (status=Done on the job)

```

---

The approach using a shared database and a Postgres queue seems to me more safe and controllable, but at the same time the approach using Kafka as a common message broker seems more modern and sustainable.

Any thoughts on this? Maybe there's better approach to this problem.