Is the X-csrf-token only available for authenticated users?
Drupal still doesn't use CSRF tokens for anonymous users. The CSRF token is also checked only when a session is available for the currently logged-in user, and only when the request method declared from the route is not one of the following: GET, HEAD, OPTIONS, TRACE.
There are some issues opened for Drupal core, which (at the moment I write this answer) still have to be fixed for Drupal 9.1.x, such as Allow form tokens to be used on anonymous forms in some cases, CSRF check always fails for users without a session, and X-CSRF-Token request header is missing when using Bearer authentication (which needs the steps to reproduce it).
The only way to add CSRF tokens for anonymous users is creating a new service that duplicates almost all the code in CsrfRequestHeaderAccessCheck, possibly using a route requirement that is different from _csrf_request_header_token used from Drupal core, and add that route requirement to the routes of your module that needs to be accessible also from anonymous users. Such service should also create a session, if it has not been already created.
Keep in mind that CSRF tokens are requested for those routes that alter data (for example, a route that deletes data from the database). If the route is simply showing data, the CSRF token is not requested.