Why are URLs with non-existing subdomains not triggering an error 404?
While you have stated that hosting is the same for both your sites it is most probably that this is either a server configuration issue or a DNS configuration issue rather than a Drupal setting issue.
In .htaccess the following catches 'wildcard' subdomains 

RewriteCond %{HTTP_HOST} ^[^.]+\.example\.com$ [NC]
RewriteCond %{HTTP_HOST} !^www\. [NC]
RewriteRule ^ https://example.com%{REQUEST_URI} [L,R=301]


In Apache configuration the following catches 'wildcard' subdomains

&lt;VirtualHost *:80&gt;
        ServerAlias *.example.com
        Redirect 301 / http://www.example.com
&lt;/VirtualHost&gt;


In Nginx confiuration the following catches 'wildcard' subdomains
(as you are using .htaccess you are obviously not using Nginx but for completeness..)

server {
  server_name *.example.com;
  return 301 $scheme://example.com$request_uri;
}

server {
  server_name www.example.com;

  #add in further directives to serve your content
}


These are the most common configurations that will catch wildcard sub-domains on the server.

It is possible to do redirects in settings.php also but not for wildcard.

The other possibility is that the issue is at the DNS entry. You may have set up a wildcard sub-domain which may also be known as a catch-all-subdomain. 

The layout of this will be determined by your DNS provider.
This Drupal guide illustrates how it would be acheived on cPanel: https://www.drupal.org/node/1068894