HfHubHTTPError - exceeded our hourly quotas for action: commit
how is your dataset structured? (file types, how many commits and files are you trying to push, etc) I succeeded in uploading it after several attempts with an hour gap between each attempt (inconvenient but worked). The final dataset is [here](https://huggingface.co/datasets/yuvalkirstain/pickapic_v2), code and context to the dataset can be found [here](https://github.com/yuvalkirstain/PickScore/).
I can close the issue if this behavior is intended, as most users probably do not need to upload large-scale datasets. We could fix this by creating a single commit for all the (Parquet) shards in `push_to_hub` instead of one commit per shard, as we currently do. 

@Wauplin Any updates on the 2-step commit process suggested by you that we need to implement this? > Any updates on the 2-step commit process suggested by you that we need to implement this?

Re-prioritizing this, sorry. Will let you know but probably can be done this week.
### Describe the bug

I try to upload a very large dataset of images, and get the following error:
```
File /fsx-multigen/yuvalkirstain/miniconda/envs/pickapic/lib/python3.10/site-packages/huggingface_hub/hf_api.py:2712, in HfApi.create_commit(self, repo_id, operations, commit_message, commit_description, token, repo_type, revision, create_pr, num_threads, parent_commit, run_as_future)
   2710 try:
   2711     commit_resp = get_session().post(url=commit_url, headers=headers, data=data, params=params)
-> 2712     hf_raise_for_status(commit_resp, endpoint_name="commit")
   2713 except RepositoryNotFoundError as e:
   2714     e.append_to_message(_CREATE_COMMIT_NO_REPO_ERROR_MESSAGE)

File /fsx-multigen/yuvalkirstain/miniconda/envs/pickapic/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py:301, in hf_raise_for_status(response, endpoint_name)
    297     raise BadRequestError(message, response=response) from e
    299 # Convert `HTTPError` into a `HfHubHTTPError` to display request information
    300 # as well (request id and/or server error message)
--> 301 raise HfHubHTTPError(str(e), response=response) from e

HfHubHTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/api/datasets/yuvalkirstain/pickapic_v2/commit/main (Request ID: Root=1-65112399-12d63f7d7f28bfa40a36a0fd)

You have exceeded our hourly quotas for action: commit. We invite you to retry later.
```

this makes it much less convenient to host large datasets on HF hub.

### Steps to reproduce the bug

Upload a very large dataset of images

### Expected behavior

the upload to work well

### Environment info

- `datasets` version: 2.13.1
- Platform: Linux-5.15.0-1033-aws-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 1.5.3