title: Why am I getting an exception when using a Range Join hint?
body: <p>I'm trying to use the <a href="https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame.hint" rel="nofollow noreferrer">DataFrame.hint()</a> method to add a <a href="https://docs.databricks.com/delta/join-performance/range-join.html#enable-range-join-using-a-range-join-hint" rel="nofollow noreferrer">Range Join hint</a> to my join.</p>
  
  <p>I have two tables: <code>minutes</code> and <code>events</code>.</p>
  
  <p>The minutes table has the <code>minute_start</code> and <code>minute_end</code> columns that are time in seconds since a fixed moment in time. Naturally, their values are multiples of 60.</p>
  
  <p>The events table has similar <code>event_start</code> and <code>event_end</code> columns, only for events. Events can start and end at any second.</p>
  
  <p>For each event, I need to find all the minutes it overlaps with.</p>
  
  <p>I'm trying this on Databricks (runtime 5.1, Python 3.5):</p>
  
  <pre class="lang-py prettyprint-override"><code># from pyspark.sql.types import StructType, StructField, IntegerType
  
  # minutes = spark.sparkContext\
  #                .parallelize(((0,  60),
  #                              (60, 120)))\
  #                .toDF(StructType([
  #                          StructField('minute_start', IntegerType()),
  #                          StructField('minute_end', IntegerType())
  #                        ]))
  
  # events = spark.sparkContext\
  #               .parallelize(((12, 33),
  #                             (0,  120),
  #                             (33, 72),
  #                             (65, 178)))\
  #               .toDF(StructType([
  #                         StructField('event_start', IntegerType()),
  #                         StructField('event_end', IntegerType())
  #                       ]))
  
  events.hint("range_join", "60")\
        .join(minutes,
              on=[events.event_start   &lt; minutes.minute_end,
                  minutes.minute_start &lt; events.event_end])\
        .orderBy(events.event_start,
                 events.event_end,
                 minutes.minute_start)\
        .show()
  </code></pre>
  
  <p>Without the <code>hint</code> call, the result is as expected:</p>
  
  <pre><code>+-----------+---------+------------+----------+
  |event_start|event_end|minute_start|minute_end|
  +-----------+---------+------------+----------+
  |          0|      120|           0|        60|
  |          0|      120|          60|       120|
  |         12|       33|           0|        60|
  |         33|       72|           0|        60|
  |         33|       72|          60|       120|
  |         65|      178|          60|       120|
  +-----------+---------+------------+----------+
  </code></pre>
  
  <p>With the <code>hint</code>, I get the exception:</p>
  
  <pre><code>AnalysisException: 'Range join hint: invalid arguments Buffer(60);'
  </code></pre>
  
  <p>When I tried passing the <code>60</code> in the hint as a number as opposed to a string, it complained that a parameter of a hint must be a string.</p>
  
  <p>I'm <strong>not on Azure</strong>, but I expect the outcome would be the same.</p>
  
  <p>Has anyone had a similar issue and found a solution or knows where I'm making a mistake?</p>
  
  <p><strong>UPDATE 1</strong></p>
  
  <p>(Currently, I'm trying it on Databricks Runtime 6.1, Python 3.7.3, Spark 2.4.4)</p>
  
  <p>I thought I missed that the parameters are expected as an iterable, so I tried again, with <code>events.hint("range_join", [60])</code>. Same complaint about the argument not being a string: <code>TypeError: all parameters should be str, got 60 of type &lt;class 'int'&gt;</code>.</p>
  
  <p>I'm wondering if Databricks' version of Spark is behind.</p>
  
  <p>This is in Spark <a href="https://github.com/apache/spark/blob/master/python/pyspark/sql/dataframe.py#L470" rel="nofollow noreferrer">source code on GitHub</a>:</p>
  
  <pre class="lang-py prettyprint-override"><code>def hint(self, name, *parameters):
      ... (no checks on `parameters` up to here)
  
      allowed_types = (basestring, list, float, int)
      for p in parameters:
          if not isinstance(p, allowed_types):
              raise TypeError(
                  "all parameters should be in {0}, got {1} of type {2}".format(
                          allowed_types, p, type(p)))
  
      ... (no checks beyond this point)
  </code></pre>
  
  <p>so a list of <code>int</code>s should be allowed.</p>
  
  <p>What I'm getting is <code>all parameters should be str</code>, but the GitHub version would say <code>all parameters should be in (basestring, list, float, int)</code> if I passed a parameter of a wrong type.</p>
  
  <p><strong>UPDATE 2</strong></p>
  
  <p><code>hint("skew", "col_name")</code> appears to be working.</p>