We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
psycopg2.errors.UniqueViolation: duplicate key value error
When running the demo data commands that I have, if the celery worker is set up to run in the background, an integretiy error for duplicate keys happens on the `image_entry.save()` call here:

https://github.com/ResonantGeoData/ResonantGeoData/blob/998a6c3995b4421c3632979a249fb78d66e1108f/rgd/geodata/models/imagery/etl.py#L69

The error:

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "geodata_imageentry_image_file_id_key"
DETAIL:  Key (image_file_id)=(14) already exists.
```

This is making me think that when we create a new `ImageEntry` in the tasks, there is some sort of race condition between jobs for the same `ImageFile`... which shouldn't happen? I'm not really sure what is going on here.
 

## Steps to reproduce

1. Clear the database volume
2. Apply migrations: `docker-compose run --rm django ./manage.py migrate`
3. In one session, launch the celery worker: `docker-compose up celery` and wait until ready
4. In another session, run the Landsat demo data command: `docker-compose run --rm django ./manage.py landsat_data -c 3`
    - Use the changes from #296 
5. Observe the error

## Error Message

<details>

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "geodata_imageentry_image_file_id_key"
DETAIL:  Key (image_file_id)=(14) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./manage.py", line 28, in <module>
    main()
  File "./manage.py", line 24, in main
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/opt/django-project/rgd/geodata/management/commands/landsat_data.py", line 49, in handle
    helper.load_raster_files(_get_landsat_urls(count))
  File "/opt/django-project/rgd/geodata/management/commands/_data_helper.py", line 80, in load_raster_files
    imentries = load_image_files(
  File "/opt/django-project/rgd/geodata/management/commands/_data_helper.py", line 56, in load_image_files
    result = load_image_files(imfile)
  File "/opt/django-project/rgd/geodata/management/commands/_data_helper.py", line 60, in load_image_files
    read_image_file(entry)
  File "/opt/django-project/rgd/geodata/models/imagery/etl.py", line 129, in read_image_file
    _read_image_to_entry(image_entry, file_path)
  File "/opt/django-project/rgd/geodata/models/imagery/etl.py", line 69, in _read_image_to_entry
    image_entry.save()
  File "/opt/django-project/rgd/geodata/models/common.py", line 51, in save
    super(ModifiableEntry, self).save(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 726, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 763, in save_base
    updated = self._save_table(
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 868, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 906, in _do_insert
    return manager._insert(
  File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 1268, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1410, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "geodata_imageentry_image_file_id_key"
DETAIL:  Key (image_file_id)=(14) already exists.
```

</details>

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `rgd/geodata/management/commands/_data_helper.py`
Content:
```
1 from functools import reduce
2 import os
3 from urllib.request import urlopen
4 
5 from django.db.models import Count
6 
7 from rgd.geodata import models, tasks
8 from rgd.geodata.datastore import datastore, registry
9 from rgd.geodata.models.imagery.etl import read_image_file
10 from rgd.utility import get_or_create_no_commit
11 
12 
13 def _get_or_download_checksum_file(name):
14     # Check if there is already an image file with this sha or URL
15     #  to avoid duplicating data
16     try:
17         _ = urlopen(name)  # HACK: see if URL first
18         try:
19             file_entry = models.ChecksumFile.objects.get(url=name)
20         except models.ChecksumFile.DoesNotExist:
21             file_entry = models.ChecksumFile()
22             file_entry.url = name
23             file_entry.type = models.FileSourceType.URL
24             file_entry.save()
25     except ValueError:
26         sha = registry[name].split(':')[1]  # NOTE: assumes sha512
27         try:
28             file_entry = models.ChecksumFile.objects.get(checksum=sha)
29         except models.ChecksumFile.DoesNotExist:
30             path = datastore.fetch(name)
31             file_entry = models.ChecksumFile()
32             file_entry.name = name
33             file_entry.file.save(os.path.basename(path), open(path, 'rb'))
34             file_entry.type = models.FileSourceType.FILE_FIELD
35             file_entry.save()
36             tasks.task_checksum_file_post_save.delay(file_entry.id)
37     return file_entry
38 
39 
40 def _get_or_create_file_model(model, name, skip_signal=False):
41     # For models that point to a `ChecksumFile`
42     file_entry = _get_or_download_checksum_file(name)
43     entry, _ = model.objects.get_or_create(file=file_entry)
44     # In case the last population failed
45     if skip_signal:
46         entry.skip_signal = True
47     if entry.status != models.mixins.Status.SUCCEEDED:
48         entry.save()
49     return entry
50 
51 
52 def load_image_files(image_files):
53     ids = []
54     for imfile in image_files:
55         if isinstance(imfile, (list, tuple)):
56             result = load_image_files(imfile)
57         else:
58             # Run `read_image_file` sequentially to ensure `ImageEntry` is generated
59             entry = _get_or_create_file_model(models.ImageFile, imfile, skip_signal=True)
60             read_image_file(entry)
61             result = entry.imageentry.pk
62         ids.append(result)
63     return ids
64 
65 
66 def load_raster_files(raster_files):
67     ids = []
68     for rf in raster_files:
69         imentries = load_image_files(
70             [
71                 rf,
72             ]
73         )
74         for pks in imentries:
75             if not isinstance(pks, (list, tuple)):
76                 pks = [
77                     pks,
78                 ]
79             # Check if an ImageSet already exists containing all of these images
80             q = models.ImageSet.objects.annotate(count=Count('images')).filter(count=len(pks))
81             imsets = reduce(lambda p, id: q.filter(images=id), pks, q).values()
82             if len(imsets) > 0:
83                 # Grab first, could be N-many
84                 imset = models.ImageSet.objects.get(id=imsets[0]['id'])
85             else:
86                 images = models.ImageEntry.objects.filter(pk__in=pks).all()
87                 imset = models.ImageSet()
88                 imset.save()  # Have to save before adding to ManyToManyField
89                 for image in images:
90                     imset.images.add(image)
91                 imset.save()
92             # Make raster of that image set
93             raster, created = models.RasterEntry.objects.get_or_create(image_set=imset)
94             if not created and raster.status != models.mixins.Status.SUCCEEDED:
95                 raster.save()
96             ids.append(raster.pk)
97     return ids
98 
99 
100 def load_shape_files(shape_files):
101     ids = []
102     for shpfile in shape_files:
103         entry = _get_or_create_file_model(models.GeometryArchive, shpfile)
104         ids.append(entry.geometryentry.pk)
105     return ids
106 
107 
108 def load_fmv_files(fmv_files):
109     raise NotImplementedError('FMV ETL with Docker is still broken.')
110 
111 
112 def load_kwcoco_archives(archives):
113     ids = []
114     for fspec, farch in archives:
115         spec = _get_or_download_checksum_file(fspec)
116         arch = _get_or_download_checksum_file(farch)
117         ds, _ = get_or_create_no_commit(models.KWCOCOArchive, spec_file=spec, image_archive=arch)
118         ds.save()
119         ids.append(ds.id)
120     return ids
121
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/rgd/geodata/management/commands/_data_helper.py b/rgd/geodata/management/commands/_data_helper.py
--- a/rgd/geodata/management/commands/_data_helper.py
+++ b/rgd/geodata/management/commands/_data_helper.py
@@ -40,11 +40,12 @@
 def _get_or_create_file_model(model, name, skip_signal=False):
     # For models that point to a `ChecksumFile`
     file_entry = _get_or_download_checksum_file(name)
-    entry, _ = model.objects.get_or_create(file=file_entry)
+    # No commit in case we need to skip the signal
+    entry, created = get_or_create_no_commit(model, file=file_entry)
     # In case the last population failed
     if skip_signal:
         entry.skip_signal = True
-    if entry.status != models.mixins.Status.SUCCEEDED:
+    if created or entry.status != models.mixins.Status.SUCCEEDED:
         entry.save()
     return entry

{"golden_diff": "diff --git a/rgd/geodata/management/commands/_data_helper.py b/rgd/geodata/management/commands/_data_helper.py\n--- a/rgd/geodata/management/commands/_data_helper.py\n+++ b/rgd/geodata/management/commands/_data_helper.py\n@@ -40,11 +40,12 @@\n def _get_or_create_file_model(model, name, skip_signal=False):\n     # For models that point to a `ChecksumFile`\n     file_entry = _get_or_download_checksum_file(name)\n-    entry, _ = model.objects.get_or_create(file=file_entry)\n+    # No commit in case we need to skip the signal\n+    entry, created = get_or_create_no_commit(model, file=file_entry)\n     # In case the last population failed\n     if skip_signal:\n         entry.skip_signal = True\n-    if entry.status != models.mixins.Status.SUCCEEDED:\n+    if created or entry.status != models.mixins.Status.SUCCEEDED:\n         entry.save()\n     return entry\n", "issue": "psycopg2.errors.UniqueViolation: duplicate key value error\nWhen running the demo data commands that I have, if the celery worker is set up to run in the background, an integretiy error for duplicate keys happens on the `image_entry.save()` call here:\r\n\r\nhttps://github.com/ResonantGeoData/ResonantGeoData/blob/998a6c3995b4421c3632979a249fb78d66e1108f/rgd/geodata/models/imagery/etl.py#L69\r\n\r\nThe error:\r\n\r\n```\r\nTraceback (most recent call last):\r\n  File \"/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py\", line 84, in _execute\r\n    return self.cursor.execute(sql, params)\r\npsycopg2.errors.UniqueViolation: duplicate key value violates unique constraint \"geodata_imageentry_image_file_id_key\"\r\nDETAIL:  Key (image_file_id)=(14) already exists.\r\n```\r\n\r\nThis is making me think that when we create a new `ImageEntry` in the tasks, there is some sort of race condition between jobs for the same `ImageFile`... which shouldn't happen? I'm not really sure what is going on here.\r\n \r\n\r\n## Steps to reproduce\r\n\r\n1. Clear the database volume\r\n2. Apply migrations: `docker-compose run --rm django ./manage.py migrate`\r\n3. In one session, launch the celery worker: `docker-compose up celery` and wait until ready\r\n4. In another session, run the Landsat demo data command: `docker-compose run --rm django ./manage.py landsat_data -c 3`\r\n    - Use the changes from #296 \r\n5. Observe the error\r\n\r\n## Error Message\r\n\r\n<details>\r\n\r\n```\r\nTraceback (most recent call last):\r\n  File \"/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py\", line 84, in _execute\r\n    return self.cursor.execute(sql, params)\r\npsycopg2.errors.UniqueViolation: duplicate key value violates unique constraint \"geodata_imageentry_image_file_id_key\"\r\nDETAIL:  Key (image_file_id)=(14) already exists.\r\n\r\n\r\nThe above exception was the direct cause of the following exception:\r\n\r\nTraceback (most recent call last):\r\n  File \"./manage.py\", line 28, in <module>\r\n    main()\r\n  File \"./manage.py\", line 24, in main\r\n    execute_from_command_line(sys.argv)\r\n  File \"/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py\", line 419, in execute_from_command_line\r\n    utility.execute()\r\n  File \"/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py\", line 413, in execute\r\n    self.fetch_command(subcommand).run_from_argv(self.argv)\r\n  File \"/usr/local/lib/python3.8/site-packages/django/core/management/base.py\", line 354, in run_from_argv\r\n    self.execute(*args, **cmd_options)\r\n  File \"/usr/local/lib/python3.8/site-packages/django/core/management/base.py\", line 398, in execute\r\n    output = self.handle(*args, **options)\r\n  File \"/opt/django-project/rgd/geodata/management/commands/landsat_data.py\", line 49, in handle\r\n    helper.load_raster_files(_get_landsat_urls(count))\r\n  File \"/opt/django-project/rgd/geodata/management/commands/_data_helper.py\", line 80, in load_raster_files\r\n    imentries = load_image_files(\r\n  File \"/opt/django-project/rgd/geodata/management/commands/_data_helper.py\", line 56, in load_image_files\r\n    result = load_image_files(imfile)\r\n  File \"/opt/django-project/rgd/geodata/management/commands/_data_helper.py\", line 60, in load_image_files\r\n    read_image_file(entry)\r\n  File \"/opt/django-project/rgd/geodata/models/imagery/etl.py\", line 129, in read_image_file\r\n    _read_image_to_entry(image_entry, file_path)\r\n  File \"/opt/django-project/rgd/geodata/models/imagery/etl.py\", line 69, in _read_image_to_entry\r\n    image_entry.save()\r\n  File \"/opt/django-project/rgd/geodata/models/common.py\", line 51, in save\r\n    super(ModifiableEntry, self).save(*args, **kwargs)\r\n  File \"/usr/local/lib/python3.8/site-packages/django/db/models/base.py\", line 726, in save\r\n    self.save_base(using=using, force_insert=force_insert,\r\n  File \"/usr/local/lib/python3.8/site-packages/django/db/models/base.py\", line 763, in save_base\r\n    updated = self._save_table(\r\n  File \"/usr/local/lib/python3.8/site-packages/django/db/models/base.py\", line 868, in _save_table\r\n    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)\r\n  File \"/usr/local/lib/python3.8/site-packages/django/db/models/base.py\", line 906, in _do_insert\r\n    return manager._insert(\r\n  File \"/usr/local/lib/python3.8/site-packages/django/db/models/manager.py\", line 85, in manager_method\r\n    return getattr(self.get_queryset(), name)(*args, **kwargs)\r\n  File \"/usr/local/lib/python3.8/site-packages/django/db/models/query.py\", line 1268, in _insert\r\n    return query.get_compiler(using=using).execute_sql(returning_fields)\r\n  File \"/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py\", line 1410, in execute_sql\r\n    cursor.execute(sql, params)\r\n  File \"/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py\", line 98, in execute\r\n    return super().execute(sql, params)\r\n  File \"/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py\", line 66, in execute\r\n    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)\r\n  File \"/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py\", line 75, in _execute_with_wrappers\r\n    return executor(sql, params, many, context)\r\n  File \"/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py\", line 84, in _execute\r\n    return self.cursor.execute(sql, params)\r\n  File \"/usr/local/lib/python3.8/site-packages/django/db/utils.py\", line 90, in __exit__\r\n    raise dj_exc_value.with_traceback(traceback) from exc_value\r\n  File \"/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py\", line 84, in _execute\r\n    return self.cursor.execute(sql, params)\r\ndjango.db.utils.IntegrityError: duplicate key value violates unique constraint \"geodata_imageentry_image_file_id_key\"\r\nDETAIL:  Key (image_file_id)=(14) already exists.\r\n```\r\n\r\n</details>\n", "before_files": [{"content": "from functools import reduce\nimport os\nfrom urllib.request import urlopen\n\nfrom django.db.models import Count\n\nfrom rgd.geodata import models, tasks\nfrom rgd.geodata.datastore import datastore, registry\nfrom rgd.geodata.models.imagery.etl import read_image_file\nfrom rgd.utility import get_or_create_no_commit\n\n\ndef _get_or_download_checksum_file(name):\n    # Check if there is already an image file with this sha or URL\n    #  to avoid duplicating data\n    try:\n        _ = urlopen(name)  # HACK: see if URL first\n        try:\n            file_entry = models.ChecksumFile.objects.get(url=name)\n        except models.ChecksumFile.DoesNotExist:\n            file_entry = models.ChecksumFile()\n            file_entry.url = name\n            file_entry.type = models.FileSourceType.URL\n            file_entry.save()\n    except ValueError:\n        sha = registry[name].split(':')[1]  # NOTE: assumes sha512\n        try:\n            file_entry = models.ChecksumFile.objects.get(checksum=sha)\n        except models.ChecksumFile.DoesNotExist:\n            path = datastore.fetch(name)\n            file_entry = models.ChecksumFile()\n            file_entry.name = name\n            file_entry.file.save(os.path.basename(path), open(path, 'rb'))\n            file_entry.type = models.FileSourceType.FILE_FIELD\n            file_entry.save()\n            tasks.task_checksum_file_post_save.delay(file_entry.id)\n    return file_entry\n\n\ndef _get_or_create_file_model(model, name, skip_signal=False):\n    # For models that point to a `ChecksumFile`\n    file_entry = _get_or_download_checksum_file(name)\n    entry, _ = model.objects.get_or_create(file=file_entry)\n    # In case the last population failed\n    if skip_signal:\n        entry.skip_signal = True\n    if entry.status != models.mixins.Status.SUCCEEDED:\n        entry.save()\n    return entry\n\n\ndef load_image_files(image_files):\n    ids = []\n    for imfile in image_files:\n        if isinstance(imfile, (list, tuple)):\n            result = load_image_files(imfile)\n        else:\n            # Run `read_image_file` sequentially to ensure `ImageEntry` is generated\n            entry = _get_or_create_file_model(models.ImageFile, imfile, skip_signal=True)\n            read_image_file(entry)\n            result = entry.imageentry.pk\n        ids.append(result)\n    return ids\n\n\ndef load_raster_files(raster_files):\n    ids = []\n    for rf in raster_files:\n        imentries = load_image_files(\n            [\n                rf,\n            ]\n        )\n        for pks in imentries:\n            if not isinstance(pks, (list, tuple)):\n                pks = [\n                    pks,\n                ]\n            # Check if an ImageSet already exists containing all of these images\n            q = models.ImageSet.objects.annotate(count=Count('images')).filter(count=len(pks))\n            imsets = reduce(lambda p, id: q.filter(images=id), pks, q).values()\n            if len(imsets) > 0:\n                # Grab first, could be N-many\n                imset = models.ImageSet.objects.get(id=imsets[0]['id'])\n            else:\n                images = models.ImageEntry.objects.filter(pk__in=pks).all()\n                imset = models.ImageSet()\n                imset.save()  # Have to save before adding to ManyToManyField\n                for image in images:\n                    imset.images.add(image)\n                imset.save()\n            # Make raster of that image set\n            raster, created = models.RasterEntry.objects.get_or_create(image_set=imset)\n            if not created and raster.status != models.mixins.Status.SUCCEEDED:\n                raster.save()\n            ids.append(raster.pk)\n    return ids\n\n\ndef load_shape_files(shape_files):\n    ids = []\n    for shpfile in shape_files:\n        entry = _get_or_create_file_model(models.GeometryArchive, shpfile)\n        ids.append(entry.geometryentry.pk)\n    return ids\n\n\ndef load_fmv_files(fmv_files):\n    raise NotImplementedError('FMV ETL with Docker is still broken.')\n\n\ndef load_kwcoco_archives(archives):\n    ids = []\n    for fspec, farch in archives:\n        spec = _get_or_download_checksum_file(fspec)\n        arch = _get_or_download_checksum_file(farch)\n        ds, _ = get_or_create_no_commit(models.KWCOCOArchive, spec_file=spec, image_archive=arch)\n        ds.save()\n        ids.append(ds.id)\n    return ids\n", "path": "rgd/geodata/management/commands/_data_helper.py"}], "after_files": [{"content": "from functools import reduce\nimport os\nfrom urllib.request import urlopen\n\nfrom django.db.models import Count\n\nfrom rgd.geodata import models, tasks\nfrom rgd.geodata.datastore import datastore, registry\nfrom rgd.geodata.models.imagery.etl import read_image_file\nfrom rgd.utility import get_or_create_no_commit\n\n\ndef _get_or_download_checksum_file(name):\n    # Check if there is already an image file with this sha or URL\n    #  to avoid duplicating data\n    try:\n        _ = urlopen(name)  # HACK: see if URL first\n        try:\n            file_entry = models.ChecksumFile.objects.get(url=name)\n        except models.ChecksumFile.DoesNotExist:\n            file_entry = models.ChecksumFile()\n            file_entry.url = name\n            file_entry.type = models.FileSourceType.URL\n            file_entry.save()\n    except ValueError:\n        sha = registry[name].split(':')[1]  # NOTE: assumes sha512\n        try:\n            file_entry = models.ChecksumFile.objects.get(checksum=sha)\n        except models.ChecksumFile.DoesNotExist:\n            path = datastore.fetch(name)\n            file_entry = models.ChecksumFile()\n            file_entry.name = name\n            file_entry.file.save(os.path.basename(path), open(path, 'rb'))\n            file_entry.type = models.FileSourceType.FILE_FIELD\n            file_entry.save()\n            tasks.task_checksum_file_post_save.delay(file_entry.id)\n    return file_entry\n\n\ndef _get_or_create_file_model(model, name, skip_signal=False):\n    # For models that point to a `ChecksumFile`\n    file_entry = _get_or_download_checksum_file(name)\n    # No commit in case we need to skip the signal\n    entry, created = get_or_create_no_commit(model, file=file_entry)\n    # In case the last population failed\n    if skip_signal:\n        entry.skip_signal = True\n    if created or entry.status != models.mixins.Status.SUCCEEDED:\n        entry.save()\n    return entry\n\n\ndef load_image_files(image_files):\n    ids = []\n    for imfile in image_files:\n        if isinstance(imfile, (list, tuple)):\n            result = load_image_files(imfile)\n        else:\n            # Run `read_image_file` sequentially to ensure `ImageEntry` is generated\n            entry = _get_or_create_file_model(models.ImageFile, imfile, skip_signal=True)\n            read_image_file(entry)\n            result = entry.imageentry.pk\n        ids.append(result)\n    return ids\n\n\ndef load_raster_files(raster_files):\n    ids = []\n    for rf in raster_files:\n        imentries = load_image_files(\n            [\n                rf,\n            ]\n        )\n        for pks in imentries:\n            if not isinstance(pks, (list, tuple)):\n                pks = [\n                    pks,\n                ]\n            # Check if an ImageSet already exists containing all of these images\n            q = models.ImageSet.objects.annotate(count=Count('images')).filter(count=len(pks))\n            imsets = reduce(lambda p, id: q.filter(images=id), pks, q).values()\n            if len(imsets) > 0:\n                # Grab first, could be N-many\n                imset = models.ImageSet.objects.get(id=imsets[0]['id'])\n            else:\n                images = models.ImageEntry.objects.filter(pk__in=pks).all()\n                imset = models.ImageSet()\n                imset.save()  # Have to save before adding to ManyToManyField\n                for image in images:\n                    imset.images.add(image)\n                imset.save()\n            # Make raster of that image set\n            raster, created = models.RasterEntry.objects.get_or_create(image_set=imset)\n            if not created and raster.status != models.mixins.Status.SUCCEEDED:\n                raster.save()\n            ids.append(raster.pk)\n    return ids\n\n\ndef load_shape_files(shape_files):\n    ids = []\n    for shpfile in shape_files:\n        entry = _get_or_create_file_model(models.GeometryArchive, shpfile)\n        ids.append(entry.geometryentry.pk)\n    return ids\n\n\ndef load_fmv_files(fmv_files):\n    raise NotImplementedError('FMV ETL with Docker is still broken.')\n\n\ndef load_kwcoco_archives(archives):\n    ids = []\n    for fspec, farch in archives:\n        spec = _get_or_download_checksum_file(fspec)\n        arch = _get_or_download_checksum_file(farch)\n        ds, _ = get_or_create_no_commit(models.KWCOCOArchive, spec_file=spec, image_archive=arch)\n        ds.save()\n        ids.append(ds.id)\n    return ids\n", "path": "rgd/geodata/management/commands/_data_helper.py"}]}