We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Error: number of embeddings does not match number of samples
I got an error while using `lightly-magic` on a larger dataset.
My gut feeling tells me that this could be related to the number of workers or the batch size. Let's debug and fix the issue :)

I used the following command:
```
lightly-magic token='TOKEN' dataset_id='DATASET_ID' input_dir=/datasets/data trainer.max_epochs=40 loader.batch_size=256  loader.num_workers=12
```

The dataset consists of `74202` images.

```
Epoch 39: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 289/289 [02:24<00:00,  2.00it/s, loss=4.66, v_num=0]
Best model is stored at: /datasets/lightly_outputs/2021-06-07/07-44-35/lightly_epoch_9.ckpt                                                                            
########## Starting to embed your dataset.
Compute efficiency: 0.03: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 290/290 [00:31<00:00,  9.25it/s]
Embeddings are stored at /datasets/lightly_outputs/2021-06-07/07-44-35/embeddings.csv                                                                                  
########## Starting to upload your dataset to the Lightly platform.
Uploading images (with 12 workers).
  0%|▏                                                                                                                                     | 99/74202 [00:09<1:31:21, 13.52imgs/s]/opt/conda/envs/lightly/lib/python3.7/site-packages/lightly_utils/image_processing/metadata.py:53: RuntimeWarning: divide by zero encountered in double_scalars                  
  return float(np.where(std == 0., 0, mean / std))
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 74202/74202 [3:47:52<00:00,  4.35imgs/s]Finished the upload of the dataset.
Starting upload of embeddings.
Traceback (most recent call last):
  File "/opt/conda/envs/lightly/lib/python3.7/site-packages/lightly/cli/lightly_cli.py", line 80, in lightly_cli                                                                 
    return _lightly_cli(cfg)
  File "/opt/conda/envs/lightly/lib/python3.7/site-packages/lightly/cli/lightly_cli.py", line 38, in _lightly_cli                                                                
    _upload_cli(cfg)
  File "/opt/conda/envs/lightly/lib/python3.7/site-packages/lightly/cli/upload_cli.py", line 76, in _upload_cli                                                                  
    path_to_embeddings_csv=path_to_embeddings, name=name
  File "/opt/conda/envs/lightly/lib/python3.7/site-packages/lightly/api/api_workflow_upload_embeddings.py", line 71, in upload_embeddings                                        
    path_to_embeddings_csv=path_to_embeddings_csv)
  File "/opt/conda/envs/lightly/lib/python3.7/site-packages/lightly/api/api_workflow_upload_embeddings.py", line 104, in _order_csv_by_filenames                                 
    raise ValueError(f'There are {len(filenames)} rows in the embedding file, but '
ValueError: There are 74202 rows in the embedding file, but 74208 filenames/samples on the server.                                                                               

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 74202/74202 [3:48:50<00:00,  5.40imgs/s]
```

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `lightly/api/api_workflow_upload_dataset.py`
Content:
```
1 import os
2 import warnings
3 from concurrent.futures.thread import ThreadPoolExecutor
4 from typing import Union
5 
6 import lightly_utils.image_processing
7 import tqdm
8 
9 from lightly.openapi_generated.swagger_client import TagCreator, SamplesApi, SampleWriteUrls
10 from lightly.openapi_generated.swagger_client.models.sample_create_request import SampleCreateRequest
11 from lightly.api.utils import check_filename, PIL_to_bytes
12 from lightly.openapi_generated.swagger_client.models.initial_tag_create_request import InitialTagCreateRequest
13 from lightly.data.dataset import LightlyDataset
14 
15 from lightly.api.utils import retry
16 
17 from lightly_utils import image_processing
18 
19 
20 
21 class _UploadDatasetMixin:
22 
23     def upload_dataset(self, input: Union[str, LightlyDataset], max_workers: int = 8,
24                        mode: str = 'thumbnails', verbose: bool = True):
25         """Uploads a dataset to to the Lightly cloud solution.
26 
27         Args:
28             input:
29                 one of the following:
30                     - the path to the dataset, e.g. "path/to/dataset"
31                     - the dataset in form of a LightlyDataset
32             max_workers:
33                 Maximum number of workers uploading images in parallel.
34             max_requests:
35                 Maximum number of requests a single worker can do before he has
36                 to wait for the others.
37             mode:
38                 One of [full, thumbnails, metadata]. Whether to upload thumbnails,
39                 full images, or metadata only.
40 
41         Raises:
42             ValueError if dataset is too large or input has the wrong type
43             RuntimeError if the connection to the server failed.
44 
45         """
46         no_tags_on_server = len(self._get_all_tags())
47         if no_tags_on_server > 0:
48             warnings.warn(f"Dataset with id {self.dataset_id} has already been completely uploaded to the platform. Skipping upload.")
49             return
50 
51         # Check input variable 'input'
52         if isinstance(input, str):
53             dataset = LightlyDataset(input_dir=input)
54         elif isinstance(input, LightlyDataset):
55             dataset = input
56         else:
57             raise ValueError(f"input must either be a LightlyDataset or the path to the dataset as str, "
58                              f"but is of type {type(input)}")
59 
60         # check the allowed dataset size
61         max_dataset_size_str = self.quota_api.get_quota_maximum_dataset_size()
62         max_dataset_size = int(max_dataset_size_str)
63         if len(dataset) > max_dataset_size:
64             msg = f'Your dataset has {len(dataset)} samples which'
65             msg += f' is more than the allowed maximum of {max_dataset_size}'
66             raise ValueError(msg)
67 
68         # handle the case where len(dataset) < max_workers
69         max_workers = min(len(dataset), max_workers)
70         max_workers = max(max_workers, 1)
71 
72         # upload the samples
73         if verbose:
74             print(f'Uploading images (with {max_workers} workers).', flush=True)
75 
76         pbar = tqdm.tqdm(unit='imgs', total=len(dataset))
77         tqdm_lock = tqdm.tqdm.get_lock()
78 
79         # calculate the files size more efficiently
80         lightly_utils.image_processing.metadata._size_in_bytes = lambda img: 0
81 
82         # define lambda function for concurrent upload
83         def lambda_(i):
84             # load image
85             image, label, filename = dataset[i]
86             filepath = dataset.get_filepath_from_filename(filename, image)
87             # try to upload image
88             try:
89                 self._upload_single_image(
90                     image=image,
91                     label=label,
92                     filename=filename,
93                     filepath=filepath,
94                     mode=mode,
95                 )
96                 success = True
97             except Exception as e:
98                 warnings.warn(f"Upload of image {filename} failed with error {e}")
99                 success = False
100 
101             # update the progress bar
102             tqdm_lock.acquire()  # lock
103             pbar.update(1)  # update
104             tqdm_lock.release()  # unlock
105             # return whether the upload was successful
106             return success
107 
108         with ThreadPoolExecutor(max_workers=max_workers) as executor:
109             results = list(executor.map(
110                 lambda_, [i for i in range(len(dataset))], chunksize=1))
111 
112         if not all(results):
113             msg = 'Warning: Unsuccessful upload(s)! '
114             msg += 'This could cause problems when uploading embeddings.'
115             msg += 'Failed at image: {}'.format(results.index(False))
116             warnings.warn(msg)
117 
118         # set image type of data and create initial tag
119         if mode == 'full':
120             img_type = 'full'
121         elif mode == 'thumbnails':
122             img_type = 'thumbnail'
123         else:
124             img_type = 'meta'
125 
126         initial_tag_create_request = InitialTagCreateRequest(img_type=img_type, creator=TagCreator.USER_PIP)
127         self.tags_api.create_initial_tag_by_dataset_id(body=initial_tag_create_request, dataset_id=self.dataset_id)
128 
129     def _upload_single_image(self, image, label, filename: str, filepath: str, mode):
130         """Uploads a single image to the Lightly platform.
131 
132         """
133         self.samples_api: SamplesApi
134 
135         # check whether the filename is too long
136         basename = filename
137         if not check_filename(basename):
138             msg = (f'Filename {basename} is longer than the allowed maximum of '
139                    'characters and will be skipped.')
140             warnings.warn(msg)
141             return False
142 
143         # calculate metadata, and check if corrupted
144         metadata = image_processing.Metadata(image).to_dict()
145         metadata["sizeInBytes"] = os.path.getsize(filepath)
146 
147         # try to get exif data
148         try:
149             exifdata = image_processing.Exifdata(image)
150         except Exception:
151             exifdata = None
152 
153         # generate thumbnail if necessary
154         thumbname = None
155         if not metadata['is_corrupted'] and mode in ["thumbnails", "full"]:
156             thumbname = '.'.join(basename.split('.')[:-1]) + '_thumb.webp'
157 
158         body = SampleCreateRequest(
159             file_name=basename,
160             thumb_name=thumbname,
161             meta_data=metadata,
162             exif=exifdata if exifdata is None else exifdata.to_dict(),
163         )
164         sample_id = retry(
165             self.samples_api.create_sample_by_dataset_id,
166             body=body,
167             dataset_id=self.dataset_id
168         ).id
169 
170         if not metadata['is_corrupted'] and mode in ["thumbnails", "full"]:
171 
172             def upload_thumbnail(image, signed_url):
173                 thumbnail = image_processing.Thumbnail(image)
174                 image_to_upload = thumbnail.to_bytes()
175                 retry(
176                     self.upload_file_with_signed_url,
177                     image_to_upload,
178                     signed_url
179                 )
180                 thumbnail.thumbnail.close()
181 
182             def upload_full_image(filepath, signed_url):
183                 with open(filepath, 'rb') as image_to_upload:
184                     retry(
185                         self.upload_file_with_signed_url,
186                         image_to_upload,
187                         signed_url
188                     )
189 
190             if mode == "thumbnails":
191                 thumbnail_url = retry(
192                     self.samples_api.get_sample_image_write_url_by_id,
193                     dataset_id=self.dataset_id,
194                     sample_id=sample_id,
195                     is_thumbnail=True
196                 )
197                 upload_thumbnail(image, thumbnail_url)
198             elif mode == "full":
199                 sample_write_urls: SampleWriteUrls = retry(
200                     self.samples_api.get_sample_image_write_urls_by_id, dataset_id=self.dataset_id, sample_id=sample_id
201                 )
202                 upload_thumbnail(image, sample_write_urls.thumb)
203                 upload_full_image(filepath, sample_write_urls.full)
204 
205 
206 
207         image.close()
208 
209 
210
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/lightly/api/api_workflow_upload_dataset.py b/lightly/api/api_workflow_upload_dataset.py
--- a/lightly/api/api_workflow_upload_dataset.py
+++ b/lightly/api/api_workflow_upload_dataset.py
@@ -1,12 +1,12 @@
 import os
 import warnings
 from concurrent.futures.thread import ThreadPoolExecutor
-from typing import Union
+from typing import Union, List
 
 import lightly_utils.image_processing
 import tqdm
 
-from lightly.openapi_generated.swagger_client import TagCreator, SamplesApi, SampleWriteUrls
+from lightly.openapi_generated.swagger_client import TagCreator, SamplesApi, SampleWriteUrls, SampleData
 from lightly.openapi_generated.swagger_client.models.sample_create_request import SampleCreateRequest
 from lightly.api.utils import check_filename, PIL_to_bytes
 from lightly.openapi_generated.swagger_client.models.initial_tag_create_request import InitialTagCreateRequest
@@ -73,16 +73,29 @@
         if verbose:
             print(f'Uploading images (with {max_workers} workers).', flush=True)
 
-        pbar = tqdm.tqdm(unit='imgs', total=len(dataset))
-        tqdm_lock = tqdm.tqdm.get_lock()
 
         # calculate the files size more efficiently
         lightly_utils.image_processing.metadata._size_in_bytes = lambda img: 0
 
+        # get the filenames of the samples already on the server
+        self.samples_api: SamplesApi
+        samples: List[SampleData] = self.samples_api.get_samples_by_dataset_id(dataset_id=self.dataset_id)
+        filenames = [sample.file_name for sample in samples]
+        if len(filenames) > 0:
+            print(f"Found {len(filenames)} images already on the server, they are skipped during the upload.")
+        filenames_set = set(filenames)
+
+        pbar = tqdm.tqdm(unit='imgs', total=len(dataset)-len(filenames))
+        tqdm_lock = tqdm.tqdm.get_lock()
+
         # define lambda function for concurrent upload
         def lambda_(i):
             # load image
             image, label, filename = dataset[i]
+            if filename in filenames_set:
+                # the sample was already uploaded
+                return True
+
             filepath = dataset.get_filepath_from_filename(filename, image)
             # try to upload image
             try:

{"golden_diff": "diff --git a/lightly/api/api_workflow_upload_dataset.py b/lightly/api/api_workflow_upload_dataset.py\n--- a/lightly/api/api_workflow_upload_dataset.py\n+++ b/lightly/api/api_workflow_upload_dataset.py\n@@ -1,12 +1,12 @@\n import os\n import warnings\n from concurrent.futures.thread import ThreadPoolExecutor\n-from typing import Union\n+from typing import Union, List\n \n import lightly_utils.image_processing\n import tqdm\n \n-from lightly.openapi_generated.swagger_client import TagCreator, SamplesApi, SampleWriteUrls\n+from lightly.openapi_generated.swagger_client import TagCreator, SamplesApi, SampleWriteUrls, SampleData\n from lightly.openapi_generated.swagger_client.models.sample_create_request import SampleCreateRequest\n from lightly.api.utils import check_filename, PIL_to_bytes\n from lightly.openapi_generated.swagger_client.models.initial_tag_create_request import InitialTagCreateRequest\n@@ -73,16 +73,29 @@\n         if verbose:\n             print(f'Uploading images (with {max_workers} workers).', flush=True)\n \n-        pbar = tqdm.tqdm(unit='imgs', total=len(dataset))\n-        tqdm_lock = tqdm.tqdm.get_lock()\n \n         # calculate the files size more efficiently\n         lightly_utils.image_processing.metadata._size_in_bytes = lambda img: 0\n \n+        # get the filenames of the samples already on the server\n+        self.samples_api: SamplesApi\n+        samples: List[SampleData] = self.samples_api.get_samples_by_dataset_id(dataset_id=self.dataset_id)\n+        filenames = [sample.file_name for sample in samples]\n+        if len(filenames) > 0:\n+            print(f\"Found {len(filenames)} images already on the server, they are skipped during the upload.\")\n+        filenames_set = set(filenames)\n+\n+        pbar = tqdm.tqdm(unit='imgs', total=len(dataset)-len(filenames))\n+        tqdm_lock = tqdm.tqdm.get_lock()\n+\n         # define lambda function for concurrent upload\n         def lambda_(i):\n             # load image\n             image, label, filename = dataset[i]\n+            if filename in filenames_set:\n+                # the sample was already uploaded\n+                return True\n+\n             filepath = dataset.get_filepath_from_filename(filename, image)\n             # try to upload image\n             try:\n", "issue": "Error: number of embeddings does not match number of samples\nI got an error while using `lightly-magic` on a larger dataset.\r\nMy gut feeling tells me that this could be related to the number of workers or the batch size. Let's debug and fix the issue :)\r\n\r\nI used the following command:\r\n```\r\nlightly-magic token='TOKEN' dataset_id='DATASET_ID' input_dir=/datasets/data trainer.max_epochs=40 loader.batch_size=256  loader.num_workers=12\r\n```\r\n\r\nThe dataset consists of `74202` images.\r\n\r\n```\r\nEpoch 39: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 289/289 [02:24<00:00,  2.00it/s, loss=4.66, v_num=0]\r\nBest model is stored at: /datasets/lightly_outputs/2021-06-07/07-44-35/lightly_epoch_9.ckpt                                                                            \r\n########## Starting to embed your dataset.\r\nCompute efficiency: 0.03: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 290/290 [00:31<00:00,  9.25it/s]\r\nEmbeddings are stored at /datasets/lightly_outputs/2021-06-07/07-44-35/embeddings.csv                                                                                  \r\n########## Starting to upload your dataset to the Lightly platform.\r\nUploading images (with 12 workers).\r\n  0%|\u258f                                                                                                                                     | 99/74202 [00:09<1:31:21, 13.52imgs/s]/opt/conda/envs/lightly/lib/python3.7/site-packages/lightly_utils/image_processing/metadata.py:53: RuntimeWarning: divide by zero encountered in double_scalars                  \r\n  return float(np.where(std == 0., 0, mean / std))\r\n100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 74202/74202 [3:47:52<00:00,  4.35imgs/s]Finished the upload of the dataset.\r\nStarting upload of embeddings.\r\nTraceback (most recent call last):\r\n  File \"/opt/conda/envs/lightly/lib/python3.7/site-packages/lightly/cli/lightly_cli.py\", line 80, in lightly_cli                                                                 \r\n    return _lightly_cli(cfg)\r\n  File \"/opt/conda/envs/lightly/lib/python3.7/site-packages/lightly/cli/lightly_cli.py\", line 38, in _lightly_cli                                                                \r\n    _upload_cli(cfg)\r\n  File \"/opt/conda/envs/lightly/lib/python3.7/site-packages/lightly/cli/upload_cli.py\", line 76, in _upload_cli                                                                  \r\n    path_to_embeddings_csv=path_to_embeddings, name=name\r\n  File \"/opt/conda/envs/lightly/lib/python3.7/site-packages/lightly/api/api_workflow_upload_embeddings.py\", line 71, in upload_embeddings                                        \r\n    path_to_embeddings_csv=path_to_embeddings_csv)\r\n  File \"/opt/conda/envs/lightly/lib/python3.7/site-packages/lightly/api/api_workflow_upload_embeddings.py\", line 104, in _order_csv_by_filenames                                 \r\n    raise ValueError(f'There are {len(filenames)} rows in the embedding file, but '\r\nValueError: There are 74202 rows in the embedding file, but 74208 filenames/samples on the server.                                                                               \r\n\r\nSet the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.\r\n100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 74202/74202 [3:48:50<00:00,  5.40imgs/s]\r\n```\n", "before_files": [{"content": "import os\nimport warnings\nfrom concurrent.futures.thread import ThreadPoolExecutor\nfrom typing import Union\n\nimport lightly_utils.image_processing\nimport tqdm\n\nfrom lightly.openapi_generated.swagger_client import TagCreator, SamplesApi, SampleWriteUrls\nfrom lightly.openapi_generated.swagger_client.models.sample_create_request import SampleCreateRequest\nfrom lightly.api.utils import check_filename, PIL_to_bytes\nfrom lightly.openapi_generated.swagger_client.models.initial_tag_create_request import InitialTagCreateRequest\nfrom lightly.data.dataset import LightlyDataset\n\nfrom lightly.api.utils import retry\n\nfrom lightly_utils import image_processing\n\n\n\nclass _UploadDatasetMixin:\n\n    def upload_dataset(self, input: Union[str, LightlyDataset], max_workers: int = 8,\n                       mode: str = 'thumbnails', verbose: bool = True):\n        \"\"\"Uploads a dataset to to the Lightly cloud solution.\n\n        Args:\n            input:\n                one of the following:\n                    - the path to the dataset, e.g. \"path/to/dataset\"\n                    - the dataset in form of a LightlyDataset\n            max_workers:\n                Maximum number of workers uploading images in parallel.\n            max_requests:\n                Maximum number of requests a single worker can do before he has\n                to wait for the others.\n            mode:\n                One of [full, thumbnails, metadata]. Whether to upload thumbnails,\n                full images, or metadata only.\n\n        Raises:\n            ValueError if dataset is too large or input has the wrong type\n            RuntimeError if the connection to the server failed.\n\n        \"\"\"\n        no_tags_on_server = len(self._get_all_tags())\n        if no_tags_on_server > 0:\n            warnings.warn(f\"Dataset with id {self.dataset_id} has already been completely uploaded to the platform. Skipping upload.\")\n            return\n\n        # Check input variable 'input'\n        if isinstance(input, str):\n            dataset = LightlyDataset(input_dir=input)\n        elif isinstance(input, LightlyDataset):\n            dataset = input\n        else:\n            raise ValueError(f\"input must either be a LightlyDataset or the path to the dataset as str, \"\n                             f\"but is of type {type(input)}\")\n\n        # check the allowed dataset size\n        max_dataset_size_str = self.quota_api.get_quota_maximum_dataset_size()\n        max_dataset_size = int(max_dataset_size_str)\n        if len(dataset) > max_dataset_size:\n            msg = f'Your dataset has {len(dataset)} samples which'\n            msg += f' is more than the allowed maximum of {max_dataset_size}'\n            raise ValueError(msg)\n\n        # handle the case where len(dataset) < max_workers\n        max_workers = min(len(dataset), max_workers)\n        max_workers = max(max_workers, 1)\n\n        # upload the samples\n        if verbose:\n            print(f'Uploading images (with {max_workers} workers).', flush=True)\n\n        pbar = tqdm.tqdm(unit='imgs', total=len(dataset))\n        tqdm_lock = tqdm.tqdm.get_lock()\n\n        # calculate the files size more efficiently\n        lightly_utils.image_processing.metadata._size_in_bytes = lambda img: 0\n\n        # define lambda function for concurrent upload\n        def lambda_(i):\n            # load image\n            image, label, filename = dataset[i]\n            filepath = dataset.get_filepath_from_filename(filename, image)\n            # try to upload image\n            try:\n                self._upload_single_image(\n                    image=image,\n                    label=label,\n                    filename=filename,\n                    filepath=filepath,\n                    mode=mode,\n                )\n                success = True\n            except Exception as e:\n                warnings.warn(f\"Upload of image {filename} failed with error {e}\")\n                success = False\n\n            # update the progress bar\n            tqdm_lock.acquire()  # lock\n            pbar.update(1)  # update\n            tqdm_lock.release()  # unlock\n            # return whether the upload was successful\n            return success\n\n        with ThreadPoolExecutor(max_workers=max_workers) as executor:\n            results = list(executor.map(\n                lambda_, [i for i in range(len(dataset))], chunksize=1))\n\n        if not all(results):\n            msg = 'Warning: Unsuccessful upload(s)! '\n            msg += 'This could cause problems when uploading embeddings.'\n            msg += 'Failed at image: {}'.format(results.index(False))\n            warnings.warn(msg)\n\n        # set image type of data and create initial tag\n        if mode == 'full':\n            img_type = 'full'\n        elif mode == 'thumbnails':\n            img_type = 'thumbnail'\n        else:\n            img_type = 'meta'\n\n        initial_tag_create_request = InitialTagCreateRequest(img_type=img_type, creator=TagCreator.USER_PIP)\n        self.tags_api.create_initial_tag_by_dataset_id(body=initial_tag_create_request, dataset_id=self.dataset_id)\n\n    def _upload_single_image(self, image, label, filename: str, filepath: str, mode):\n        \"\"\"Uploads a single image to the Lightly platform.\n\n        \"\"\"\n        self.samples_api: SamplesApi\n\n        # check whether the filename is too long\n        basename = filename\n        if not check_filename(basename):\n            msg = (f'Filename {basename} is longer than the allowed maximum of '\n                   'characters and will be skipped.')\n            warnings.warn(msg)\n            return False\n\n        # calculate metadata, and check if corrupted\n        metadata = image_processing.Metadata(image).to_dict()\n        metadata[\"sizeInBytes\"] = os.path.getsize(filepath)\n\n        # try to get exif data\n        try:\n            exifdata = image_processing.Exifdata(image)\n        except Exception:\n            exifdata = None\n\n        # generate thumbnail if necessary\n        thumbname = None\n        if not metadata['is_corrupted'] and mode in [\"thumbnails\", \"full\"]:\n            thumbname = '.'.join(basename.split('.')[:-1]) + '_thumb.webp'\n\n        body = SampleCreateRequest(\n            file_name=basename,\n            thumb_name=thumbname,\n            meta_data=metadata,\n            exif=exifdata if exifdata is None else exifdata.to_dict(),\n        )\n        sample_id = retry(\n            self.samples_api.create_sample_by_dataset_id,\n            body=body,\n            dataset_id=self.dataset_id\n        ).id\n\n        if not metadata['is_corrupted'] and mode in [\"thumbnails\", \"full\"]:\n\n            def upload_thumbnail(image, signed_url):\n                thumbnail = image_processing.Thumbnail(image)\n                image_to_upload = thumbnail.to_bytes()\n                retry(\n                    self.upload_file_with_signed_url,\n                    image_to_upload,\n                    signed_url\n                )\n                thumbnail.thumbnail.close()\n\n            def upload_full_image(filepath, signed_url):\n                with open(filepath, 'rb') as image_to_upload:\n                    retry(\n                        self.upload_file_with_signed_url,\n                        image_to_upload,\n                        signed_url\n                    )\n\n            if mode == \"thumbnails\":\n                thumbnail_url = retry(\n                    self.samples_api.get_sample_image_write_url_by_id,\n                    dataset_id=self.dataset_id,\n                    sample_id=sample_id,\n                    is_thumbnail=True\n                )\n                upload_thumbnail(image, thumbnail_url)\n            elif mode == \"full\":\n                sample_write_urls: SampleWriteUrls = retry(\n                    self.samples_api.get_sample_image_write_urls_by_id, dataset_id=self.dataset_id, sample_id=sample_id\n                )\n                upload_thumbnail(image, sample_write_urls.thumb)\n                upload_full_image(filepath, sample_write_urls.full)\n\n\n\n        image.close()\n\n\n", "path": "lightly/api/api_workflow_upload_dataset.py"}], "after_files": [{"content": "import os\nimport warnings\nfrom concurrent.futures.thread import ThreadPoolExecutor\nfrom typing import Union, List\n\nimport lightly_utils.image_processing\nimport tqdm\n\nfrom lightly.openapi_generated.swagger_client import TagCreator, SamplesApi, SampleWriteUrls, SampleData\nfrom lightly.openapi_generated.swagger_client.models.sample_create_request import SampleCreateRequest\nfrom lightly.api.utils import check_filename, PIL_to_bytes\nfrom lightly.openapi_generated.swagger_client.models.initial_tag_create_request import InitialTagCreateRequest\nfrom lightly.data.dataset import LightlyDataset\n\nfrom lightly.api.utils import retry\n\nfrom lightly_utils import image_processing\n\n\n\nclass _UploadDatasetMixin:\n\n    def upload_dataset(self, input: Union[str, LightlyDataset], max_workers: int = 8,\n                       mode: str = 'thumbnails', verbose: bool = True):\n        \"\"\"Uploads a dataset to to the Lightly cloud solution.\n\n        Args:\n            input:\n                one of the following:\n                    - the path to the dataset, e.g. \"path/to/dataset\"\n                    - the dataset in form of a LightlyDataset\n            max_workers:\n                Maximum number of workers uploading images in parallel.\n            max_requests:\n                Maximum number of requests a single worker can do before he has\n                to wait for the others.\n            mode:\n                One of [full, thumbnails, metadata]. Whether to upload thumbnails,\n                full images, or metadata only.\n\n        Raises:\n            ValueError if dataset is too large or input has the wrong type\n            RuntimeError if the connection to the server failed.\n\n        \"\"\"\n        no_tags_on_server = len(self._get_all_tags())\n        if no_tags_on_server > 0:\n            warnings.warn(f\"Dataset with id {self.dataset_id} has already been completely uploaded to the platform. Skipping upload.\")\n            return\n\n        # Check input variable 'input'\n        if isinstance(input, str):\n            dataset = LightlyDataset(input_dir=input)\n        elif isinstance(input, LightlyDataset):\n            dataset = input\n        else:\n            raise ValueError(f\"input must either be a LightlyDataset or the path to the dataset as str, \"\n                             f\"but is of type {type(input)}\")\n\n        # check the allowed dataset size\n        max_dataset_size_str = self.quota_api.get_quota_maximum_dataset_size()\n        max_dataset_size = int(max_dataset_size_str)\n        if len(dataset) > max_dataset_size:\n            msg = f'Your dataset has {len(dataset)} samples which'\n            msg += f' is more than the allowed maximum of {max_dataset_size}'\n            raise ValueError(msg)\n\n        # handle the case where len(dataset) < max_workers\n        max_workers = min(len(dataset), max_workers)\n        max_workers = max(max_workers, 1)\n\n        # upload the samples\n        if verbose:\n            print(f'Uploading images (with {max_workers} workers).', flush=True)\n\n\n        # calculate the files size more efficiently\n        lightly_utils.image_processing.metadata._size_in_bytes = lambda img: 0\n\n        # get the filenames of the samples already on the server\n        self.samples_api: SamplesApi\n        samples: List[SampleData] = self.samples_api.get_samples_by_dataset_id(dataset_id=self.dataset_id)\n        filenames = [sample.file_name for sample in samples]\n        if len(filenames) > 0:\n            print(f\"Found {len(filenames)} images already on the server, they are skipped during the upload.\")\n        filenames_set = set(filenames)\n\n        pbar = tqdm.tqdm(unit='imgs', total=len(dataset)-len(filenames))\n        tqdm_lock = tqdm.tqdm.get_lock()\n\n        # define lambda function for concurrent upload\n        def lambda_(i):\n            # load image\n            image, label, filename = dataset[i]\n            if filename in filenames_set:\n                # the sample was already uploaded\n                return True\n\n            filepath = dataset.get_filepath_from_filename(filename, image)\n            # try to upload image\n            try:\n                self._upload_single_image(\n                    image=image,\n                    label=label,\n                    filename=filename,\n                    filepath=filepath,\n                    mode=mode,\n                )\n                success = True\n            except Exception as e:\n                warnings.warn(f\"Upload of image {filename} failed with error {e}\")\n                success = False\n\n            # update the progress bar\n            tqdm_lock.acquire()  # lock\n            pbar.update(1)  # update\n            tqdm_lock.release()  # unlock\n            # return whether the upload was successful\n            return success\n\n        with ThreadPoolExecutor(max_workers=max_workers) as executor:\n            results = list(executor.map(\n                lambda_, [i for i in range(len(dataset))], chunksize=1))\n\n        if not all(results):\n            msg = 'Warning: Unsuccessful upload(s)! '\n            msg += 'This could cause problems when uploading embeddings.'\n            msg += 'Failed at image: {}'.format(results.index(False))\n            warnings.warn(msg)\n\n        # set image type of data and create initial tag\n        if mode == 'full':\n            img_type = 'full'\n        elif mode == 'thumbnails':\n            img_type = 'thumbnail'\n        else:\n            img_type = 'meta'\n\n        initial_tag_create_request = InitialTagCreateRequest(img_type=img_type, creator=TagCreator.USER_PIP)\n        self.tags_api.create_initial_tag_by_dataset_id(body=initial_tag_create_request, dataset_id=self.dataset_id)\n\n    def _upload_single_image(self, image, label, filename: str, filepath: str, mode):\n        \"\"\"Uploads a single image to the Lightly platform.\n\n        \"\"\"\n        self.samples_api: SamplesApi\n\n        # check whether the filename is too long\n        basename = filename\n        if not check_filename(basename):\n            msg = (f'Filename {basename} is longer than the allowed maximum of '\n                   'characters and will be skipped.')\n            warnings.warn(msg)\n            return False\n\n        # calculate metadata, and check if corrupted\n        metadata = image_processing.Metadata(image).to_dict()\n        metadata[\"sizeInBytes\"] = os.path.getsize(filepath)\n\n        # try to get exif data\n        try:\n            exifdata = image_processing.Exifdata(image)\n        except Exception:\n            exifdata = None\n\n        # generate thumbnail if necessary\n        thumbname = None\n        if not metadata['is_corrupted'] and mode in [\"thumbnails\", \"full\"]:\n            thumbname = '.'.join(basename.split('.')[:-1]) + '_thumb.webp'\n\n        body = SampleCreateRequest(\n            file_name=basename,\n            thumb_name=thumbname,\n            meta_data=metadata,\n            exif=exifdata if exifdata is None else exifdata.to_dict(),\n        )\n        sample_id = retry(\n            self.samples_api.create_sample_by_dataset_id,\n            body=body,\n            dataset_id=self.dataset_id\n        ).id\n\n        if not metadata['is_corrupted'] and mode in [\"thumbnails\", \"full\"]:\n\n            def upload_thumbnail(image, signed_url):\n                thumbnail = image_processing.Thumbnail(image)\n                image_to_upload = thumbnail.to_bytes()\n                retry(\n                    self.upload_file_with_signed_url,\n                    image_to_upload,\n                    signed_url\n                )\n                thumbnail.thumbnail.close()\n\n            def upload_full_image(filepath, signed_url):\n                with open(filepath, 'rb') as image_to_upload:\n                    retry(\n                        self.upload_file_with_signed_url,\n                        image_to_upload,\n                        signed_url\n                    )\n\n            if mode == \"thumbnails\":\n                thumbnail_url = retry(\n                    self.samples_api.get_sample_image_write_url_by_id,\n                    dataset_id=self.dataset_id,\n                    sample_id=sample_id,\n                    is_thumbnail=True\n                )\n                upload_thumbnail(image, thumbnail_url)\n            elif mode == \"full\":\n                sample_write_urls: SampleWriteUrls = retry(\n                    self.samples_api.get_sample_image_write_urls_by_id, dataset_id=self.dataset_id, sample_id=sample_id\n                )\n                upload_thumbnail(image, sample_write_urls.thumb)\n                upload_full_image(filepath, sample_write_urls.full)\n\n\n\n        image.close()\n\n\n", "path": "lightly/api/api_workflow_upload_dataset.py"}]}