prompt: We are currently solving the following issue within our repository. Here is the issue text:
  --- BEGIN ISSUE ---
  [Bug]: New SHA256 hash takes extremely long time up to a point of of model load being unusable
  ### Is there an existing issue for this?
  
  - [X] I have searched the existing issues and checked the recent builds/commits
  
  ### What happened?
  
  Newly added sha-256 hash takes extremely long time to calculate on model load up to a point where loading appears to hang (i've restarted server twice before i even let it run until completion)  
  
  Previously switching to a new model was sub 10 sec, now switching to a new model (that does not have hash stored already) takes 100-150 sec (and this is a high end system)!
  
  And to make it worse, messages about hash calculation are only printed **after** it has been calculated, there is no progress info or anything to indicate system is actually doing anything for 2 min!
  
  
  ### Steps to reproduce the problem
  
  1. Switch to a new model and wait for completion - it takes forever
  
  
  ### What should have happened?
  
  Model load should **never** take over 2 minutes to complete.
  
  ### Commit where the problem happens
  
  f8c512478568293155539f616dce26c5e4495055
  
  ### What platforms do you use to access UI ?
  
  Windows, Linux
  
  ### What browsers do you use to access the UI ?
  
  Google Chrome, Microsoft Edge
  
  ### Command Line Arguments
  
  ```Shell
  --api --xformers
  ```
  
  
  ### Additional information, context and logs
  
  Console log showing model load taking 142 seconds!
  
  ```text
  Calculating sha256 for /home/vlado/dev/automatic/models/Stable-diffusion/mood-beautyreal-v01.ckpt: bcc0afd3b264ea028928187f56f70840f8d87ccf283b020982beba35d9c7e4ef
  Loading weights [bcc0afd3b2] from /home/vlado/dev/automatic/models/Stable-diffusion/mood-beautyreal-v01.ckpt
  Couldn't find VAE named vae-ft-mse-840000-ema-pruned; using None instead
  Applying xformers cross attention optimization.
  Weights loaded in 142.6s.
  ```
  
  
  --- END ISSUE ---
  
  Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
  --- BEGIN FILES ---
  Path: `modules/hashes.py`
  Content:
  ```
  1 import hashlib
  2 import json
  3 import os.path
  4 
  5 import filelock
  6 
  7 
  8 cache_filename = "cache.json"
  9 cache_data = None
  10 
  11 
  12 def dump_cache():
  13     with filelock.FileLock(cache_filename+".lock"):
  14         with open(cache_filename, "w", encoding="utf8") as file:
  15             json.dump(cache_data, file, indent=4)
  16 
  17 
  18 def cache(subsection):
  19     global cache_data
  20 
  21     if cache_data is None:
  22         with filelock.FileLock(cache_filename+".lock"):
  23             if not os.path.isfile(cache_filename):
  24                 cache_data = {}
  25             else:
  26                 with open(cache_filename, "r", encoding="utf8") as file:
  27                     cache_data = json.load(file)
  28 
  29     s = cache_data.get(subsection, {})
  30     cache_data[subsection] = s
  31 
  32     return s
  33 
  34 
  35 def calculate_sha256(filename):
  36     hash_sha256 = hashlib.sha256()
  37 
  38     with open(filename, "rb") as f:
  39         for chunk in iter(lambda: f.read(4096), b""):
  40             hash_sha256.update(chunk)
  41 
  42     return hash_sha256.hexdigest()
  43 
  44 
  45 def sha256_from_cache(filename, title):
  46     hashes = cache("hashes")
  47     ondisk_mtime = os.path.getmtime(filename)
  48 
  49     if title not in hashes:
  50         return None
  51 
  52     cached_sha256 = hashes[title].get("sha256", None)
  53     cached_mtime = hashes[title].get("mtime", 0)
  54 
  55     if ondisk_mtime > cached_mtime or cached_sha256 is None:
  56         return None
  57 
  58     return cached_sha256
  59 
  60 
  61 def sha256(filename, title):
  62     hashes = cache("hashes")
  63 
  64     sha256_value = sha256_from_cache(filename, title)
  65     if sha256_value is not None:
  66         return sha256_value
  67 
  68     print(f"Calculating sha256 for {filename}: ", end='')
  69     sha256_value = calculate_sha256(filename)
  70     print(f"{sha256_value}")
  71 
  72     hashes[title] = {
  73         "mtime": os.path.getmtime(filename),
  74         "sha256": sha256_value,
  75     }
  76 
  77     dump_cache()
  78 
  79     return sha256_value
  80 
  81 
  82 
  83 
  84 
  85
  ```
  --- END FILES ---
  
  Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.
  
  Here is an example:
  
  ```diff
  diff --git a/examples/server_async.py b/examples/server_async.py
  --- a/examples/server_async.py
  +++ b/examples/server_async.py
  @@ -313,4 +313,4 @@
   
   
   if __name__ == "__main__":
  -    asyncio.run(run_async_server("."), debug=True)
  +    asyncio.run(run_async_server(), debug=True)
  diff --git a/examples/server_sync.py b/examples/server_sync.py
  --- a/examples/server_sync.py
  +++ b/examples/server_sync.py
  @@ -313,5 +313,5 @@
   
   
   if __name__ == "__main__":
  -    server = run_sync_server(".")
  +    server = run_sync_server()
       server.shutdown()
  
  ```
  
golden_diff: diff --git a/modules/hashes.py b/modules/hashes.py
  --- a/modules/hashes.py
  +++ b/modules/hashes.py
  @@ -34,9 +34,10 @@
   
   def calculate_sha256(filename):
       hash_sha256 = hashlib.sha256()
  +    blksize = 1024 * 1024
   
       with open(filename, "rb") as f:
  -        for chunk in iter(lambda: f.read(4096), b""):
  +        for chunk in iter(lambda: f.read(blksize), b""):
               hash_sha256.update(chunk)
   
       return hash_sha256.hexdigest()
  
verification_info: {"golden_diff": "diff --git a/modules/hashes.py b/modules/hashes.py\n--- a/modules/hashes.py\n+++ b/modules/hashes.py\n@@ -34,9 +34,10 @@\n \r\n def calculate_sha256(filename):\r\n     hash_sha256 = hashlib.sha256()\r\n+    blksize = 1024 * 1024\r\n \r\n     with open(filename, \"rb\") as f:\r\n-        for chunk in iter(lambda: f.read(4096), b\"\"):\r\n+        for chunk in iter(lambda: f.read(blksize), b\"\"):\r\n             hash_sha256.update(chunk)\r\n \r\n     return hash_sha256.hexdigest()\n", "issue": "[Bug]: New SHA256 hash takes extremely long time up to a point of of model load being unusable\n### Is there an existing issue for this?\r\n\r\n- [X] I have searched the existing issues and checked the recent builds/commits\r\n\r\n### What happened?\r\n\r\nNewly added sha-256 hash takes extremely long time to calculate on model load up to a point where loading appears to hang (i've restarted server twice before i even let it run until completion)  \r\n\r\nPreviously switching to a new model was sub 10 sec, now switching to a new model (that does not have hash stored already) takes 100-150 sec (and this is a high end system)!\r\n\r\nAnd to make it worse, messages about hash calculation are only printed **after** it has been calculated, there is no progress info or anything to indicate system is actually doing anything for 2 min!\r\n\r\n\r\n### Steps to reproduce the problem\r\n\r\n1. Switch to a new model and wait for completion - it takes forever\r\n\r\n\r\n### What should have happened?\r\n\r\nModel load should **never** take over 2 minutes to complete.\r\n\r\n### Commit where the problem happens\r\n\r\nf8c512478568293155539f616dce26c5e4495055\r\n\r\n### What platforms do you use to access UI ?\r\n\r\nWindows, Linux\r\n\r\n### What browsers do you use to access the UI ?\r\n\r\nGoogle Chrome, Microsoft Edge\r\n\r\n### Command Line Arguments\r\n\r\n```Shell\r\n--api --xformers\r\n```\r\n\r\n\r\n### Additional information, context and logs\r\n\r\nConsole log showing model load taking 142 seconds!\r\n\r\n```text\r\nCalculating sha256 for /home/vlado/dev/automatic/models/Stable-diffusion/mood-beautyreal-v01.ckpt: bcc0afd3b264ea028928187f56f70840f8d87ccf283b020982beba35d9c7e4ef\r\nLoading weights [bcc0afd3b2] from /home/vlado/dev/automatic/models/Stable-diffusion/mood-beautyreal-v01.ckpt\r\nCouldn't find VAE named vae-ft-mse-840000-ema-pruned; using None instead\r\nApplying xformers cross attention optimization.\r\nWeights loaded in 142.6s.\r\n```\r\n\n", "before_files": [{"content": "import hashlib\r\nimport json\r\nimport os.path\r\n\r\nimport filelock\r\n\r\n\r\ncache_filename = \"cache.json\"\r\ncache_data = None\r\n\r\n\r\ndef dump_cache():\r\n    with filelock.FileLock(cache_filename+\".lock\"):\r\n        with open(cache_filename, \"w\", encoding=\"utf8\") as file:\r\n            json.dump(cache_data, file, indent=4)\r\n\r\n\r\ndef cache(subsection):\r\n    global cache_data\r\n\r\n    if cache_data is None:\r\n        with filelock.FileLock(cache_filename+\".lock\"):\r\n            if not os.path.isfile(cache_filename):\r\n                cache_data = {}\r\n            else:\r\n                with open(cache_filename, \"r\", encoding=\"utf8\") as file:\r\n                    cache_data = json.load(file)\r\n\r\n    s = cache_data.get(subsection, {})\r\n    cache_data[subsection] = s\r\n\r\n    return s\r\n\r\n\r\ndef calculate_sha256(filename):\r\n    hash_sha256 = hashlib.sha256()\r\n\r\n    with open(filename, \"rb\") as f:\r\n        for chunk in iter(lambda: f.read(4096), b\"\"):\r\n            hash_sha256.update(chunk)\r\n\r\n    return hash_sha256.hexdigest()\r\n\r\n\r\ndef sha256_from_cache(filename, title):\r\n    hashes = cache(\"hashes\")\r\n    ondisk_mtime = os.path.getmtime(filename)\r\n\r\n    if title not in hashes:\r\n        return None\r\n\r\n    cached_sha256 = hashes[title].get(\"sha256\", None)\r\n    cached_mtime = hashes[title].get(\"mtime\", 0)\r\n\r\n    if ondisk_mtime > cached_mtime or cached_sha256 is None:\r\n        return None\r\n\r\n    return cached_sha256\r\n\r\n\r\ndef sha256(filename, title):\r\n    hashes = cache(\"hashes\")\r\n\r\n    sha256_value = sha256_from_cache(filename, title)\r\n    if sha256_value is not None:\r\n        return sha256_value\r\n\r\n    print(f\"Calculating sha256 for {filename}: \", end='')\r\n    sha256_value = calculate_sha256(filename)\r\n    print(f\"{sha256_value}\")\r\n\r\n    hashes[title] = {\r\n        \"mtime\": os.path.getmtime(filename),\r\n        \"sha256\": sha256_value,\r\n    }\r\n\r\n    dump_cache()\r\n\r\n    return sha256_value\r\n\r\n\r\n\r\n\r\n\r\n", "path": "modules/hashes.py"}], "after_files": [{"content": "import hashlib\r\nimport json\r\nimport os.path\r\n\r\nimport filelock\r\n\r\n\r\ncache_filename = \"cache.json\"\r\ncache_data = None\r\n\r\n\r\ndef dump_cache():\r\n    with filelock.FileLock(cache_filename+\".lock\"):\r\n        with open(cache_filename, \"w\", encoding=\"utf8\") as file:\r\n            json.dump(cache_data, file, indent=4)\r\n\r\n\r\ndef cache(subsection):\r\n    global cache_data\r\n\r\n    if cache_data is None:\r\n        with filelock.FileLock(cache_filename+\".lock\"):\r\n            if not os.path.isfile(cache_filename):\r\n                cache_data = {}\r\n            else:\r\n                with open(cache_filename, \"r\", encoding=\"utf8\") as file:\r\n                    cache_data = json.load(file)\r\n\r\n    s = cache_data.get(subsection, {})\r\n    cache_data[subsection] = s\r\n\r\n    return s\r\n\r\n\r\ndef calculate_sha256(filename):\r\n    hash_sha256 = hashlib.sha256()\r\n    blksize = 1024 * 1024\r\n\r\n    with open(filename, \"rb\") as f:\r\n        for chunk in iter(lambda: f.read(blksize), b\"\"):\r\n            hash_sha256.update(chunk)\r\n\r\n    return hash_sha256.hexdigest()\r\n\r\n\r\ndef sha256_from_cache(filename, title):\r\n    hashes = cache(\"hashes\")\r\n    ondisk_mtime = os.path.getmtime(filename)\r\n\r\n    if title not in hashes:\r\n        return None\r\n\r\n    cached_sha256 = hashes[title].get(\"sha256\", None)\r\n    cached_mtime = hashes[title].get(\"mtime\", 0)\r\n\r\n    if ondisk_mtime > cached_mtime or cached_sha256 is None:\r\n        return None\r\n\r\n    return cached_sha256\r\n\r\n\r\ndef sha256(filename, title):\r\n    hashes = cache(\"hashes\")\r\n\r\n    sha256_value = sha256_from_cache(filename, title)\r\n    if sha256_value is not None:\r\n        return sha256_value\r\n\r\n    print(f\"Calculating sha256 for {filename}: \", end='')\r\n    sha256_value = calculate_sha256(filename)\r\n    print(f\"{sha256_value}\")\r\n\r\n    hashes[title] = {\r\n        \"mtime\": os.path.getmtime(filename),\r\n        \"sha256\": sha256_value,\r\n    }\r\n\r\n    dump_cache()\r\n\r\n    return sha256_value\r\n\r\n\r\n\r\n\r\n\r\n", "path": "modules/hashes.py"}]}