How Drupal handles nodes 404 requests? I'm curious about how Drupal 8 handles nodes 404 requests. I think there might be a route of 404 and then an internal check - if node id doesn't exists, it simply calls 404 route path.

If this scenario is right, how can I achieve the same for custom entity?

I have a content entity type, but want a 404 redirection if someone access wrong entity id directly similar to any node page which doesn't exists.
Ah, seeing that this is a follow-up to How to check if entity id exists or valid?, I think I know what the problem is.

If you ensure upcasting through the route, then Drupal throws a not found exception automatically and shows the 404 page.

So, you need to define your route as /entityform/{your_entity_type_id} and in the controller, type hint the argument YourEntityTypeId $your_entity_type_id. Then Drupal automatically loads the object, gives it to you and returns a 404 page if not found.