title: retrieve list of file names from publicly available Amazon S3 bucket using R
body: <p>My goal is to download all tifs in a directory in a public Amazon S3 bucket. USGS has started using the Amazon S3 bucket as a way to distribute their 3DEP elevation products. However, I do not want to have to right click and download each file individually. In the past I've retrieved a list of file names in a http folder, create a <code>df</code> of full path to files, then used <code>download.file</code> to loop through each file and save it. I'm currently having an issue retrieving the list of filenames in the S3 bucket. Here is what I've tried so far...</p>
  <pre><code>www &lt;- &quot;http://prd-tnm.s3.amazonaws.com/index.html?prefix=StagedProducts/Elevation/OPR/Projects/WI_AshlandIronFlorence_2019_D19/WI_Ashland_4_2019/TIFF/&quot;
  
  library(rvest)
  page &lt;- read_html(www)
  files1 &lt;- html_elements(page, xpath = &quot;.//a[contains(@href,'.tif')]&quot;) %&gt;% html_text()
  print(files1)
  
  library(XML)
  files2 &lt;- XML::getHTMLLinks(www, xpQuery = &quot;//a/@href['.tif'=substring(., string-length(.) - 3)]&quot;)
  print(files2)
  
  library(aws.s3)
  bucket_address &lt;- &quot;http://prd-tnm.s3.amazonaws.com/index.html?prefix=StagedProducts/Elevation/OPR/Projects/WI_AshlandIronFlorence_2019_D19/WI_Ashland_4_2019/TIFF/&quot;
  get_bucket_df(bucket = bucket_address)
  </code></pre>
  <p>Thanks</p>