Iterating over dataset produces different label distribution statistics.
Hi @WillRL ,

Thanks for reporting. Would you please provide some dummy dataset similar to yours for replicating the issue. It will help us for analysing the issue. Thanks! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60525">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60525">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

No

### Source

source

### Tensorflow Version

2.10.1

### Custom Code

No

### OS Platform and Distribution

Windows 10 GPU

### Mobile device

Windows 10 GPU

### Python version

3.10

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

cuda 11.8, cudaNN 8.6.0

### GPU model and memory

_No response_

### Current Behaviour?

Iterating over tf.data.Dataset object and counting the number of occurrences of each label produces different results each time.

I initially thought that this may be caused by batching and shuffling (With dropping remainders), however, there is no batching of the dataset and it is not shuffling after each iteration as well.


```python
datasets = [val_dataset, val_dataset, val_dataset] # Using same dataset three times. expect to see same results each three times.
table_data = [[],[],[]]
amounts = np.zeros(len(labels)) # Allocating array outside to be more memory efficient.
counter = 0

for i, ds in enumerate(datasets):
    ds_size = int(ds.__len__())
    amounts = amounts*0
    total = 0
    for _, label in ds:
        label = int(label)
        amounts[label] += 1
        counter += 1

    for amount in amounts:
        table_data[i].append(f"{amount} ({(100*amount/ds_size):.1f}%)")

table = pd.DataFrame(table_data, columns=labels, index=['ds 1', "ds 2", "ds 3"])
display(table)
```


Table displayed shows: 
![image](https://user-images.githubusercontent.com/62092803/236663799-260d48b1-6f4c-41dc-92b9-d8410066f251.png)

The name of columns are the classes, we expect to see each row to have the same value as other rows of the same column, however, we don't. The counter reports back the correct amount of images. For reference, the dataset used has 39209 images. split 0.8. 0.1 and 0.1 for train, val, test. It is a supervised dataset.




### Standalone code to reproduce the issue

```shell
dataset = keras.utils.image_dataset_from_directory("to_some_dir",
                                                interpolation='bilinear',
                                                shuffle=True,
                                                image_size=(64, 64),
                                                batch_size=None)

dataset_size = int(dataset.__len__())

train_size = int(0.8 * dataset_size)
val_size = int(0.1 * dataset_size)
test_size = int(0.1 * dataset_size)

train_dataset = dataset.take(train_size)
val_dataset = dataset.skip(train_size).take(val_size)
test_dataset = dataset.skip(train_size).skip(val_size)

datasets = [val_dataset, val_dataset, val_dataset] # Using the same dataset three times for test only.
table_data = [[],[],[]]
amounts = np.zeros(len(labels)) # Allocating array outside to be more memory efficient.
counter = 0

for i, ds in enumerate(datasets):
    ds_size = int(ds.__len__())
    amounts = amounts*0
    total = 0
    for _, label in ds:
        label = int(label)
        amounts[label] += 1
        counter += 1

    for amount in amounts:
        table_data[i].append(f"{amount} ({(100*amount/ds_size):.1f}%)")

table = pd.DataFrame(table_data, columns=labels, index=['Train', "Validation", "Test"])
display(table)
```


### Relevant log output

_No response_</details>