title: How to run an AWS Glue Python Spark Job with the Current Version of boto3?
body: <p>I'm trying to run the latest version of boto3 in an AWS Glue spark job to access methods that aren't available in the default version in Glue.</p>
  <p>To get the default version of boto3 and verify the method I want to access isn't available I run this block of code which is all boilerplate except for my <code>print</code> statements:</p>
  <pre><code>import sys
  import boto3
  from awsglue.transforms import *
  from awsglue.utils import getResolvedOptions
  from pyspark.context import SparkContext
  from awsglue.context import GlueContext
  from awsglue.job import Job
  
  ## @params: [JOB_NAME]
  args = getResolvedOptions(sys.argv, ['JOB_NAME'])
  
  sc = SparkContext()
  glueContext = GlueContext(sc)
  spark = glueContext.spark_session
  job = Job(glueContext)
  job.init(args['JOB_NAME'], args)
  
  athena = boto3.client('athena')
  print(boto3.__version__) # verify the default version boto3 imports
  print(athena.list_table_metadata) # method I want to verify I can access in Glue
  
  job.commit()
  </code></pre>
  <p>which returns</p>
  <blockquote>
  <p>1.12.4</p>
  </blockquote>
  <blockquote>
  <p>Traceback (most recent call last): File &quot;/tmp/another_sample&quot;, line 20, in  print(athena.list_table_metadata) File &quot;/home/spark/.local/lib/python3.7/site-packages/botocore/client.py&quot;, line 566, in <strong>getattr</strong> self.<strong>class</strong>.<strong>name</strong>, item) AttributeError: 'Athena' object has no attribute 'list_table_metadata'</p>
  </blockquote>
  <p>Ok, as expected with an older version of boto3. Let's try and import the latest version...</p>
  <p>I perform the following steps:</p>
  <ol>
  <li>Go to <a href="https://pypi.org/project/boto3/#files" rel="nofollow noreferrer">https://pypi.org/project/boto3/#files</a></li>
  <li>Download the <strong>boto3-1.17.13-py2.py3-none-any.whl</strong> file</li>
  <li>Place it in S3 location</li>
  <li>Go back to the Glue Job and under the <strong>Security configuration, script libraries, and job parameters (optional)</strong> section I update the <strong>Python library path</strong> with the S3 location from step 3</li>
  <li>Rerun block of code from above</li>
  </ol>
  <p>which returns</p>
  <blockquote>
  <p>1.17.9</p>
  </blockquote>
  <blockquote>
  <p>Traceback (most recent call last): File &quot;/tmp/another_sample&quot;, line 20, in  print(athena.list_table_metadata) File &quot;/home/spark/.local/lib/python3.7/site-packages/botocore/client.py&quot;, line 566, in <strong>getattr</strong> self.<strong>class</strong>.<strong>name</strong>, item) AttributeError: 'Athena' object has no attribute 'list_table_metadata'</p>
  </blockquote>
  <p>If I run this same script locally, which is running 1.17.9 I can find the method:</p>
  <blockquote>
  <p>1.17.9</p>
  </blockquote>
  <blockquote>
  <p>&lt;bound method ClientCreator._create_api_method.._api_call of &lt;botocore.client.Athena object at 0x7efd8a4f4710&gt;&gt;</p>
  </blockquote>
  <p>Any ideas on what's going on here and how to access the methods that I would expect should be imported in the upgraded version?</p>