Custom Ajax Route, I want to cache it
What kind of ajax exactly? I guess it returns JSON?

Drupal's standard ajax API currently always does POST requests, those are not cached. So make sure you are doing a GET request.

Render caching is also only available for render arrays/HTML.

Your response would be cached if you return an object that returns CacheableResponseInterface but only by the (anoymous) internal page cache. 

To have caching for all users, you likely need to implement your own logic  to cache the response object as you suggested. By default, the node_list cache tag can be used to have it invalidated automatically whenever any node is created or saved. You can define your own cache tag (any string works) that you invalidate yourself only when certain conditions are true (for example, a new node of a specific type, matching your entity query).