Issue title: Merged CSS/JS files require pub/static/_cache writeable, contrary to documentation
 Issue body: ### Preconditions
1. Have Magento installed with two filesystem owners [as documented](http://devdocs.magento.com/guides/v2.2/config-guide/prod/prod_file-sys-perms.html), with all code files (source and CLI-generated) read-only for web server user. This includes the full `pub/static` directory.
2. Turn on merging of CSS & JS files.

### Steps to reproduce
Clear static file cache and load home page.

### Expected result
No exception should be generated.

### Actual result
Exception is logged:

`[2018-01-16 16:18:56] main.CRITICAL: Directory "/var/www/html/___/pub/static/_cache/merged" cannot be created Warning!mkdir(): Permission denied {"exception":"[object] (Magento\\Framework\\Exception\\FileSystemException(code: 0): Directory \"/var/www/html/___/pub/static/_cache/merged\" cannot be created Warning!mkdir(): Permission denied at /var/www/html/___/vendor/magento/framework/Filesystem/Driver/File.php:225)"} []`

### Commentary
This could arguably be seen as a documentation issue in which case I can report it in the devdocs repo. However, as I understand it, it seems that the documentation does describe the intended state, meaning the code should be changed instead. I'm not sure if it's possible, but if it were, ideally the merged files would be generated by `static-content:deploy` as alluded to in #9542.
 Comments: 
 Comment 0: +1 here guys! Nothing should be written to pub/static when in production mode, and a setup:static-content:deploy _should_ write out merged files to static/_cache. This causes problems in load-balanced environments where a snapshot of a pre-compiled build is deployed.
 Comment 1: Hi @scottsb i cannot reproduce you scenario, which version of magento you've tested ?
 Comment 2: I see this behaviour in 2.1.
 Comment 3: This is a problem with 2.2.x as well. It starts with the files in static/_cache/merged not being written during static content deploy, and then the issue is compounded (on either a read-only file system or ) when Magento attempts to write merged JS / CSS files to disk during a web request.
 Comment 4: @scottsb , we are closing this issue due to inactivity. If you'd like to update it, please reopen the issue.
 Comment 5: @davidalger @scottsb Sorry for the confusion. This ticket was closed mistakenly. We will take a look at it again and let you know.
 Comment 6: hi @scottsb  , thank you for your report.
We've acknowledged the issue and added to our backlog.
 Comment 7: We also build separate from deploy and ran into this issue. We are expecting pub/static to actually be static and read only. We have also made the workaround to make the cache dir writable. Would be cleaner if these merged files generated as part of static content deploy.
 Comment 8: We are experiencing this same issue on our production Magento 2.2.6 server.  
 Comment 9: Looking forward to an update on this. Running 2.2.7 on a cluster and had to disable merging of css, not because of non writeable /pub/static/_cache folder, as I have allowed it, but individual servers don't seem to be triggering the generation of the cache files. I seem to get many different min.css files generated, with different hash's, but not always the one that the browser is requesting. I'd love to see these generated in the static:content:deploy phase.
 Comment 10: I think I understand the reasoning behind this. It's because it has to create the merged file for every possible frontend page, which would be tough to generate statically, especially without a full running app present to crawl. 

I honestly think documenting in Magento docs that pub/static/_cache needs to be writable and shared across all frontend servers would be enough. 
 Comment 11: Why shared. Does magento keep track of which ones have been generated, and in effect only generates it on one of the cluster nodes and 404s it from the rest of the cluster as all cluster nodes think its already generated, not knowing it is only present on one node.

That would certainly explain what i am seeing.
 Comment 12: I was mulling over this last night and was thinking of how this could be solved. This is rough and may not be workable, but it might be a start:

1. Change the generation of the files to happen on-demand rather than as part of the triggering page request. This would allow whatever server receives the request to generate the file locally, even if it's a different server than triggered it originally. This file could then be cached using the general cache config (such as in Redis or at least in the expected `var` cache directory if using disk). In order for this to work the URL either has to include sufficient data or reference some cache key (which would be stored during the initial page request).

2. The first step alone would avoid the present filesystem permissions issue but at the significant cost of requiring a Magento boot to serve the merged files, even if actual processing is quick once cached. So the next step would be changing the default Varnish VCL config to include these merged static files. (Currently all non-HTML assets are [excluded by default](https://devdocs.magento.com/guides/v2.3/config-guide/varnish/config-varnish-magento.html#cache-static-files).) This only solves the problem for those using Varnish, but that's recommended config for prod in any case.

A different, more exotic option would be to keep the merging in the initial page request but to push the data into the full page cache inline. This wouldn't be hard for Magento's built-in FPC, but for Varnish it would require us to push data out to it rather than waiting on it to cache while proxying. [This is possible](https://gist.github.com/lampeh/3896514) but is a rather non-standard approach.
 Comment 13: @gwharton Yes, Magento merges the file as part of the page request and saves it the local disk. It's expected that the file is then served directly by the web server as a static file from then on (it never routes through PHP/Magento). But if the merged asset request goes to a different server in the cluster, then it won't be present locally if the directory isn't shared.
 Comment 14: @scottsb Yeah, I just deployed a shared writable pub/static/_cache folder across my cluster and re-enabled merging of css files and all seems ok.
 Comment 15: I've also had to mount {{pub/static/_cache}} as a shared folder in my Kubernetes environment since any time more than one Pod (container) is running, the file will be generated on one, but not another, and that kinda messes things up—plus I have to make sure it's writeable (which it wasn't earlier).
 Comment 16: To follow up to the previous commit, I _can't_ actually mount `pub/static/_cache` into my containers because when Magento rebuilds static assets it tries to delete the `_cache` directory.

So the next best compromise is mounting all of `pub/static` on all my containers. But if you do this, make sure the `.htaccess` file that ships with Magento core (which has rewrite rules for the 'version' directory for assets) is copied into the shared `pub/static` directory, otherwise all your static assets will be available, but 404'ing because Magento loads them at paths like `pub/static/version1548342577/frontend/` (note that `versionXYZ` is not an actual directory).
 Comment 17: I'm having the same problem. I will put static/_cache in shared storage, writable by www-data.
static appears to be not static...
 Comment 18: Depending on how many modules/extensions you have, you may find it easier to disable merging of css (and thus there is no static/_cache folder) and manually streamline your css instead.

There may be limited benefit to merging css with http2 anyway.
 Comment 19: I have enabled the merge JS & CSS, minification JS & CSS and bundled JS options. I have set up multiple instances and turned on the cloud front caching in AWS. But whenever the deployment is initiated through AWS code pipeline and merged JS and CSS files are missing and the website page is broken.

**Current Flow:** 
- Checked the pub/static/_cache/merged/ folder from instance 1 and instance 2. But some merged JS 
 and CSS files are missing.

**Expected / Actual:** 
- Check the pub/static/_cache/merged/ folder from instance 1 and instance 2. Both instances should have the same merged JS and CSS files.

 Comment 20: What is the current status of this? I share pub/media across servers, but not pub/static.

After a deploy, I have to clear the cache on all servers otherwise CSS is not loaded.

 Comment 21: I'm also facing the same issue. We have multiple instances with AWS. Post deploy my website broken due to 404's on css and js files. 
Will the issue be solved when we share the pub/static/_cache across all instances ?
 Comment 22: @SeeniBorn 

Either turn off CSS merging, or pub/static/_cache needs to be writable by and shared across all instances.
 Comment 23: @gwharton 
Turning off CSS merging will affect the site performance right?
If so, I will go for shared across all instances.
 Comment 24: You can't share pub/static/_cache without sharing pub/static.

Is that what's needed?
 Comment 25: Depends on how many modules you have. I just put up with the fact that i have about 10 css files instead of 1 merged and dont need shared static. Has little effect with http/2.0 anyway. What has far more of an effect is not being able to merge js effectively as there are far far more of them.
 Comment 26: My pub/static folder is uploaded to a read only cdn drive at deploy time so i cant have a shared writeable area under pub/static using that deploy method. My frontends dont even have the pub/static folder on them.
 Comment 27: What is the cdn provider you are using?
 Comment 28: My servers are on google compute engine so i just upload static content to a google storage bucket configured for direct public access. Handles all the caching and scaling and offloads the majority from your server. Works quite well (given a few tweaks)
 Comment 29: Is this the kind of error everyone is seeing:

```
[2019-06-06 12:40:28] report.ERROR: Requested path '_cache/merged/3be83f6cdb1c6b4e61ac4c5ee753c201.min.css' is wrong. [] []
[2019-06-06 12:40:28] report.CRITICAL: Requested path '_cache/merged/3be83f6cdb1c6b4e61ac4c5ee753c201.min.css' is wrong. [] []
[2019-06-06 12:40:28] report.ERROR: Requested path '_cache/merged/5bebc9bda012302869ab3ecd7b2a9006.min.js' is wrong. [] []
[2019-06-06 12:40:28] report.CRITICAL: Requested path '_cache/merged/5bebc9bda012302869ab3ecd7b2a9006.min.js' is wrong. [] []
[2019-06-06 12:41:58] report.ERROR: Requested path '_cache/merged/3be83f6cdb1c6b4e61ac4c5ee753c201.min.css' is wrong. [] []
[2019-06-06 12:41:58] report.CRITICAL: Requested path '_cache/merged/3be83f6cdb1c6b4e61ac4c5ee753c201.min.css' is wrong. [] []
[2019-06-06 12:41:58] report.ERROR: Requested path '_cache/merged/5bebc9bda012302869ab3ecd7b2a9006.min.js' is wrong. [] []
[2019-06-06 12:41:58] report.CRITICAL: Requested path '_cache/merged/5bebc9bda012302869ab3ecd7b2a9006.min.js' is wrong. [] []
```
 Comment 30: +1. Any workaround ?
 Comment 31: @SeeniBorn / @craigcarnell, Solution for this one is that we need to share the same pub/static for the different instance in AWS. Is this solution will work. 
 Comment 32: +1 , This is still an issue with Magento 2.3.1
 Comment 33: I'm having this issue as wel on Magento 2.3.4 after enabling CSS merge. Because of this it looks like my images are dissapearing on PDP.
 Comment 34: Same here — issue is happening on 2.3.4.

I'm using https://github.com/jalogut/magento2-deployer-plus and it also expects pub/static not to be writable.
 Comment 35: Having the issue on 2.3.4 as well.
The static-content deployment happens on the CI, which will build a docker image. This image does not have the pub/static/_cache directory.
If I deploy a first container, it works well, and the _cache directory is generated somehow.
If I deploy a second container, from exactly the same image, the _cache directory is not there, so CSS and JS fails.
I think sharing the _cache is a workaround, but is not a good solution. The whole purpose of it is to increase performance, if this cache lives on NFS, it would defeat the purpose, independently of whether you have a CDN on top or not.
Also, even with HTTP2, downloading a single file is always better that downloading 15 files, so disabling bundling is not ideal either.
I think every instance should have it's own _cache directory, just like it's own static assets.
 Comment 36: > Having the issue on 2.3.4 as well.
> The static-content deployment happens on the CI, which will build a docker image. This image does not have the pub/static/_cache directory.
> If I deploy a first container, it works well, and the _cache directory is generated somehow.
> If I deploy a second container, from exactly the same image, the _cache directory is not there, so CSS and JS fails.
> I think sharing the _cache is a workaround, but is not a good solution. The whole purpose of it is to increase performance, if this cache lives on NFS, it would defeat the purpose, independently of whether you have a CDN on top or not.
> Also, even with HTTP2, downloading a single file is always better that downloading 15 files, so disabling bundling is not ideal either.
> I think every instance should have it's own _cache directory, just like it's own static assets.

+1
 Comment 37: static-content:deploy command doesn't generates any file under pub/static/_cache/merged in Magento 2.4.0
 Comment 38: @webkul-ajaysaini You are correct. The frontend does it as and when requests come in.
 Comment 39: > @webkul-ajaysaini You are correct. The frontend does it as and when requests come in.

That should be a problem, which forces us to share the static folder between web nodes => the bottle-neck is becoming to the storage sharing system (we - like others - are using NFS).
IMO, web nodes should have their own files system for its best performance.
 Comment 40: And if a folder does not exist (pub/static/_cache), Magento creates _cache folder in var/tmp directory which grows unlimited.

var/tmp $ ls
**_cache**

Is fix only to create the folder manually or?
 Comment 41: Not sure if my issue is slightly different, im on 2.3.5-p1 and have locked my permissions down for those folders as per DevDocs and for just one module I have the following error sometimes generated in exception.log:

`[2021-04-16 08:08:58] main.CRITICAL: The path "/home/sites/magento/public_html/var/view_preprocessed/pub/static/frontend/Buzz/buzzluma/en_GB/Harrigo_CallForPrice/css/style.min.css" is not writable. {"exception":"[object] (Magento\\Framework\\Exception\\FileSystemException(code: 0): The path \"/home/sites/magento/public_html/var/view_preprocessed/pub/static/frontend/Buzz/buzzluma/en_GB/Harrigo_CallForPrice/css/style.min.css\" is not writable. at /home/sites/magento/public_html/vendor/magento/framework/Filesystem/Directory/Write.php:57)"} []`

Seems the file is present in pub static and should also be merged as referenced in default_head_blocks.xml file not sure why Magento trying to write it in there?
 Comment 42: The solution for this is to run static-content:deploy with option "-s standard" - set deployment strategy to standard. In this case the pub/_cache/merged/... files are being generated during static-content:deploy

More informations:
https://devdocs.magento.com/guides/v2.4/config-guide/cli/config-cli-subcommands-static-view.html

By the way: It would be useful if this documentation mentioned that setting deployment strategy to standard (this is not default strategy) will generate also the merged JS / CSS files.
 Comment 43: Sometime we get empty minify JS cached on cloudfront (path is located at /pub/static/_cache) so ideally deploy-static process should build these ready
 Comment 44: `2.4.3`, 2022 issue is still there.
 Comment 45: Any solution? For me, it is happening in Magento 2.4.2. I don't see any merged CSS added inside the HTML head tag. Also, it creates files in the **var/tmp** folder on every request. It grows to 300GB within a day and it down the server (No Space left). Right now, we schedule cron to clear the **var/tmp** folder to fix the "No space left" issue.

**Load the home page first time**
<magento-root>/var/tmp/_cache$ find . -type f
./merged/496119470a3f8db183209c81713b1f0e.min.css_OarcqoTd0W5G4yREWzaRbLgkzP9XwoVf

**Load the same page scecond time**
<magento-root>/var/tmp/_cache$ find . -type f
./merged/496119470a3f8db183209c81713b1f0e.min.css_OarcqoTd0W5G4yREWzaRbLgkzP9XwoVf
./merged/84ca4336623d8f0129099e328566962c.min.css_WvL2yQC0FIhpRDk8CGlSguKDe69VGWny
 Comment 46: Magento 2.4.4 - Issue still present.

CSS/JS should never be generated on web-request, these are static files, they do not change, having this part of the web request is a critical design flaw. It severely affects containerized applications and forces infrastructure work-arounds.

Merged JS / CSS should be built in static content deploy and not generated by frontend request.
 Comment 47: With the current "PR Created" label being added I'm hoping we are close to a possible solution for this issue. However I can't seem to find any corresponding PR, does anyone have a link?
 Comment 48: @mattgrul if there is a PR, I would LOVE to see it.  This has been driving me nuts for a couple of years now, and it would really help my quality of life if I didn't have to keep selectively changing permissions on these static files.
 Comment 49: I do not see PR either. Would be very pleased to see such too. Same opinion as of @Stoyvo , it's really unacceptable and pretty strange such a flaw to exists so much time on such advanced system. Someone should add more labels, like "platform health"
 Comment 50: For those wondering, the PR is likely created on Adobe's internal VCS, so you know it's being worked on.
{'source': 'GithubIssues', 'repo_name_with_owner': 'magento/magento2', 'issue_id': 288982442, 'repo_stars': 11872.0}