title: How can I run a python script to generate/update a JSON file in an S3 bucket?
body: <p><strong>MY QUESTION: How can I run a python script to generate/update a JSON file in an S3 bucket? Also, should I run this script every time a file gets added to my bucket, or when someone visits my webpage?</strong></p>
  
  <p><strong>Read on for description...</strong></p>
  
  <p>I have a website hosted on AWS, my static pages are in a public S3 bucket. The purpose is to share music I've written with music students. I have a Python script which scans all the objects (sheet music PDFs) in a folder in the same S3 bucket. The Python script then creates a JSON file that has the names of all the s3 objects along with <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html" rel="nofollow noreferrer">generated URLs</a> for each object.</p>
  
  <p>Heres how that JSON file is formatted:</p>
  
  <pre><code>{
    "bass": [{
        "bass-song1": "http://www.website.com/bass-song1.pdf"
      }, {
        "bass-song2": "http://www.website.com/bass-song2.pdf"
      }],
    "drum": [{
        "drum-song1": "http://www.website.com/drum-song1.pdf"
      }, {
        "drum-song2": "http://www.website.com/drum-song2.pdf"
      }],
    "guitar": [{
        "guitar-song1": "http://www.website.com/guitar-song1.pdf"
      }, {
        "guitar-song2": "http://www.website.com/guitar-song2.pdf"
      }]
  }
  </code></pre>
  
  <p>My working Python script, for reference:</p>
  
  <pre><code>import boto3
  import json
  import pprint
  
  # This program creates a json file
  # with temporary URLs for bucket objects, organized by folder
  # For use with javascript that generates links to bucket objects for website
  
  # create a session to retrieve credentials from ~/.aws/credentials
  session = boto3.Session(profile_name='&lt;MY PROFILE&gt;')
  
  # use your credentials to create a low-level client with the s3 service
  s3 = session.client('s3')
  
  # Store dictionary of objects from bucket that starts with the prefix '__'
  response = s3.list_objects_v2(Bucket='&lt;MY BUCKET&gt;', Prefix='&lt;FOLDER IN BUCKET&gt;')
  
  folder_list = []
  url_json = {}
  
  # (the value of 'Contents' is a list of dictionaries)
  # For all the dictionaries in the 'Contents' list,
  # IF they don't end with '/' (meaning, if its not a directory)...
  for i in response['Contents']:
      if i['Key'].endswith('/') != True:
  
          # get the directory of the current Key, save as string in 'dir'
          full_path = i['Key']
          # this retrieves the folder after '__'
          dir = full_path.split("/")[1]
          # capitalize the directory name, update variable
          dir = dir.capitalize()
          # this retrieves the file name
          filename = full_path.split("/")[2]
  
          # if the name of the directory ('dir') is not in folder_list:
          # add it to folder_list,
          # and add an item to dictionary 'url_json' where key is current 'dir' and value is empty list
          if dir not in folder_list:
              folder_list.append(dir)
              url_json[folder_list[-1]] = []
  
          # generate a temporary URL for the current bucket object
          url = s3.generate_presigned_url('get_object', Params={'Bucket':'&lt;MY BUCKET&gt;', 'Key':i['Key']}, ExpiresIn=3600)
  
          # create a dictionary for each bucket object
          # store the object's name (Key) and URL (value)
          object_dict = {filename:url}
  
          # Append the newly created URL to a list in the 'url_json' dictionary,
          # whose key is the last directory in 'folder_list'
          url_json[folder_list[-1]].append(object_dict)
  
  # Dump content of 'url_json' directory to 'urls.json' file
  # if it already exists, overwrite it
  with open('url_list.json', mode='w') as outfile:
      json.dump(url_json, outfile)
  </code></pre>
  
  <p>Also, in the body of my webpage I wrote some in-line JavaScript that loads/reads the JSON file, and uses the URLs and their associated text to create links for people to access those files on my webpage.</p>
  
  <p>My working JavaScript, for reference:</p>
  
  <pre><code>&lt;script type="text/javascript"&gt;
  
  async function getData(url) {
    const response = await fetch(url);
    return response.json()
  }
  
  async function main() {
    const data = await getData('&lt;JSON FILE URL&gt;');
  
    // 'instrument' example: 'Bass'
    for (var instrument in data) {
  
      var h = document.createElement("H3"); // Create the H1 element
      var t = document.createTextNode(instrument); // Create a text element
      h.appendChild(t); // Append the text node to the H1 element
      document.body.appendChild(h); // Append the H1 element to the document body
  
      // store the list of bass songs
      var song_list = data[instrument]
      // for each song (list element),
      for (var song_object of song_list) {
        // for every song name in the object (in python, dictionary)
        for (var song_name in song_object) {
          // create a var with the name and URL of the PDF song file
          var link_str = song_name;
          var link_url = song_object[song_name];
  
          // Create link to appear on website
          const a = document.createElement("a");
          var lineBreak = document.createElement("br");
          a.href = link_url;
          a.innerText = link_str;
          a.style.backgroundColor="rgba(255, 255, 255, 0.7)"
          document.body.appendChild(a);
          document.body.append(lineBreak);
        }
      }
    }
  }
  main();
  
  &lt;/script&gt;
  </code></pre>
  
  <p><strong>So my end goal is this:</strong>
  I upload a PDF file to my S3 bucket, my Python script executes to update the JSON file in my S3 bucket, and when someone visits my webpage the JavaScript will parse the JSON file to create links to those PDF files. Should I have the python script create the JSON when I upload files into the bucket, or when someone visits the website? The issue is that these object URLs have a <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html" rel="nofollow noreferrer">time expiration</a>... but I also don't want to be running code every time someone loads a page (for $$ reasons).</p>
  
  <p>Any help would be greatly appreciated. Let me know if you need anymore info from me.
  Thank you,
  Sebastian.</p>