Issue title: [Question] Large number of sub-directories in `requests.queue`
 Issue body: ## Motivation

I am currently running a broad crawl on ~3 Million starting URLs using the suggested settings from this [page](https://docs.scrapy.org/en/latest/topics/broad-crawls.html). Since pause and resume features are necessary for my crawl, I used the `JOBDIR` project setting. This has been working great so far.

Recently I observed that the directory count of the `requests.queue` directory in the `JOBDIR` keeps on increasing in size. For example, I currently have ~800,000 sub-directories in `requests.queue`. I am concerned that a constant increase in size of this directory might lead to Inodes running out in my (relatively small) server's partition.

## Question

Is there any way to have `scrapy` automatically prune the `requests.queue` directory once the requests in this directory are complete?

PS: I am not well versed with how `queuelib` works in `scrapy`, so my apologies if this is a naive question.

## Additional information

Upon closer inspection, I observed that the majority of sub-directories in `requests.queue` are either empty or contain empty files with a `q*` prefix; as described in #4842 as well. I am guessing that these correspond to completed requests. If this is correct, then these stale directories/files could be easily parsed and removed.
 Comments: 
 Comment 0: I did not look into it too closely, I simply confirmed that [disk-based queuelib _should_ be removing files once empty](https://github.com/scrapy/queuelib/blob/c135eda9a1837f17e0826f5edee6e6bd8c6469e6/queuelib/queue.py#L138).

It would probably be helpful if you could provide a [minimal, reproducible example](https://stackoverflow.com/help/minimal-reproducible-example).

I could _not_ reproduce the empty `q`-prefixed files with the spider below. I run `scrapy runspider <path>`, stopped mid-crawl with Ctrl+C, then resumed. By the end of the crawl, `requests.queue/toscrape.com-<hash>` was empty (maybe you would rather it be removed?).

```py
from scrapy import Spider


class MySpider(Spider):
    name = "myspider"
    start_urls = ["https://toscrape.com"]
    custom_settings = {
        "JOBDIR": "jobdir",
        "SCHEDULER_DISK_QUEUE": "scrapy.squeues.PickleFifoDiskQueue",
        "SCHEDULER_PRIORITY_QUEUE": "scrapy.pqueues.DownloaderAwarePriorityQueue",
    }

    def parse(self, response):
        for i in range(200):
            yield response.follow(
                self.start_urls[0],
                callback=self.ignore,
                dont_filter=True,
            )

    def ignore(self, response):
        pass
```
 Comment 1: I tested this example and I confirm that `requests.queue/toscrape.com-<hash>` was empty. 
But indeed it would be interesting that this empty directory is deleted at the  the end of the crawl.
 Comment 2: Hi @Gallaecio and @LeMoussel, thank you both for the prompt responses. Regarding the empty `q*` files, I did some debugging and found a bug on my end which caused this. I made a mistake of using `functools.partial` to manipulate an `errback` of a request, for example:

```python
yield Request(url, callback=self.parse, errback=partial(self.errback, some_boolean = True))
```

I should have used the `cb_kwargs` option instead of `partial`, sorry about that. Upon removing the `partial`, I now have the same results as you where the `requests.queue` directory contains only empty directories at the end of a crawl.

> But indeed it would be interesting that this empty directory is deleted at the the end of the crawl.

In my use case, I need to have these empty directories removed periodically rather than *only* at the end of a crawl, otherwise I would run out of `inodes` in my server's partition before the crawl even finishes.

So my question now is:

Is it safe to remove empty directories in `requests.queue` periodically during a crawl, or is there some process within the crawl which requires these empty directories?
 Comment 3: Here is a MWE which produces three sub-directories in `requests.queue`:

```python
import scrapy


class MySpider(scrapy.Spider):
    name = "myspider"
    start_urls = ["https://toscrape.com"]
    custom_settings = {
        "JOBDIR": "jobdir",
        "SCHEDULER_DISK_QUEUE": "scrapy.squeues.PickleFifoDiskQueue",
        "SCHEDULER_PRIORITY_QUEUE": "scrapy.pqueues.DownloaderAwarePriorityQueue"
    }
    link_extractor = scrapy.linkextractors.LinkExtractor()

    def parse(self, response):
        for link in self.link_extractor.extract_links(response):
            yield scrapy.Request(link.url, callback=self.ignore, dont_filter=True)

    def ignore(self, response):
        pass
```
 Comment 4: In [squeues.py](https://github.com/scrapy/scrapy/blob/master/scrapy/squeues.py#L128) found this 
```
_PickleFifoSerializationDiskQueue = _serializable_queue(
    _with_mkdir(queue.FifoDiskQueue),
    _pickle_serialize,
    pickle.loads
)
```
=> `with_mkdir(queue.FifoDiskQueue)` -> Create directory ?

but also in [queuelib/queue.py](https://github.com/scrapy/queuelib/blob/master/queuelib/queue.py#L96)  
```
    def __init__(self, path: str, chunksize: int = 100000) -> None:
        self.path = path
        if not os.path.exists(path):
            os.makedirs(path)
```

=> `os.makedirs(path)` -> Create directory ?

In `queue.FifoDiskQueue` there is [`close` method ](https://github.com/scrapy/queuelib/blob/master/queuelib/queue.py#L158)

Perhaps to create a `FifoDiskQueue` & `LifoDiskQueue` subclass that overrides the `close` method  to handle the remove directory `self.path`?
 Comment 5: To do what @atreyasha wants, which I think is for Scrapy to only keep those folders while they have content, we would need to go deeper than changing the close method. Those queues should not just create the folders when the queue is created and then remove them; they should create them while they have enqueued requests, remove them when they run out of requests, and then re-create them if they get more requests.

It’s probably not trivial to do, but it should be possible to implement. We can modify the existing queue classes or create new ones if the change is expected to have any significant performance hit (my guess would be no).
{'source': 'GithubIssues', 'repo_name_with_owner': 'scrapy/scrapy', 'issue_id': 1024107327, 'repo_stars': 45035.0}