title: How to make a pyarrow.DictionaryArray with ExtensionType? Using from_buffers? Using cast?
body: <p>Ultimately, my goal is to make a <a href="https://arrow.apache.org/docs/python/generated/pyarrow.DictionaryArray.html" rel="nofollow noreferrer">pyarrow.DictionaryArray</a> with an <a href="https://arrow.apache.org/docs/python/generated/pyarrow.ExtensionType.html" rel="nofollow noreferrer">ExtensionType</a>.</p>
  <p>For all other kinds of Arrow arrays, I can use the <a href="https://arrow.apache.org/docs/python/generated/pyarrow.Array.html#pyarrow.Array.from_buffers" rel="nofollow noreferrer">Array.from_buffers</a> static method to construct it and pass the ExtensionType as its first argument. However, I don't see a way to use from_buffers on DictionaryArray because I need to pass its <a href="https://arrow.apache.org/docs/python/generated/pyarrow.DictionaryArray.html#pyarrow.DictionaryArray.dictionary" rel="nofollow noreferrer">dictionary</a>, but it takes zero children.</p>
  <p>Given a DicationaryArray <code>a</code> and naively using its from_buffers (assuming the dictionary is in its <a href="https://arrow.apache.org/docs/python/generated/pyarrow.Array.html#pyarrow.Array.type" rel="nofollow noreferrer">type</a>, which I'm pretty sure it's not) results in a segfault.</p>
  <pre class="lang-py prettyprint-override"><code>&gt;&gt;&gt; import pyarrow as pa
  &gt;&gt;&gt; a = pa.array([&quot;one&quot;, &quot;two&quot;, &quot;three&quot;, &quot;two&quot;, &quot;one&quot;]).dictionary_encode()
  &gt;&gt;&gt; b = pa.DictionaryArray.from_buffers(a.type, len(a), a.indices.buffers())
  /arrow/cpp/src/arrow/array/array_dict.cc:83:  Check failed: (data-&gt;dictionary) != (nullptr) 
  /home/jpivarski/miniconda3/lib/python3.9/site-packages/pyarrow/libarrow.so.600(+0xf17fe8)[0x7f9dc4008fe8]
  /home/jpivarski/miniconda3/lib/python3.9/site-packages/pyarrow/libarrow.so.600(_ZN5arrow4util8ArrowLogD1Ev+0xed)[0x7f9dc400978d]
  /home/jpivarski/miniconda3/lib/python3.9/site-packages/pyarrow/libarrow.so.600(_ZN5arrow15DictionaryArrayC2ERKSt10shared_ptrINS_9ArrayDataEE+0x11f)[0x7f9dc427d00f]
  /home/jpivarski/miniconda3/lib/python3.9/site-packages/pyarrow/libarrow.so.600(_ZN5arrow9MakeArrayERKSt10shared_ptrINS_9ArrayDataEE+0x2c6)[0x7f9dc41381b6]
  /home/jpivarski/miniconda3/lib/python3.9/site-packages/pyarrow/lib.cpython-39-x86_64-linux-gnu.so(+0x213a77)[0x7f9dc5504a77]
  python(+0x15f995)[0x55b45da91995]
  python(_PyObject_MakeTpCall+0x316)[0x55b45da783d6]
  python(_PyEval_EvalFrameDefault+0x52de)[0x55b45db162ce]
  python(+0x138e20)[0x55b45da6ae20]
  python(_PyEval_EvalCodeWithName+0x47)[0x55b45db4f977]
  python(PyEval_EvalCodeEx+0x39)[0x55b45db4f9b9]
  python(PyEval_EvalCode+0x1b)[0x55b45db4f9db]
  python(+0x2506c9)[0x55b45db826c9]
  python(+0x28b994)[0x55b45dbbd994]
  python(+0x1142bf)[0x55b45da462bf]
  python(PyRun_InteractiveLoopFlags+0xeb)[0x55b45da4646a]
  python(+0x11487a)[0x55b45da4687a]
  python(+0x114e14)[0x55b45da46e14]
  python(Py_BytesMain+0x39)[0x55b45dbc4329]
  /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f9dc6abd0b3]
  python(+0x20aa51)[0x55b45db3ca51]
  Aborted (core dumped)
  </code></pre>
  <p>I didn't think that would work, anyway.</p>
  <p>For completeness, I also thought that there might be casting rules from each storage type to an ExtensionType with that storage type, but no, <a href="https://arrow.apache.org/docs/python/generated/pyarrow.Array.html#pyarrow.Array.cast" rel="nofollow noreferrer">cast</a> doesn't work, either.</p>
  <pre class="lang-py prettyprint-override"><code>&gt;&gt;&gt; import json
  &gt;&gt;&gt; import pyarrow as pa
  &gt;&gt;&gt; a = pa.array([&quot;one&quot;, &quot;two&quot;, &quot;three&quot;, &quot;two&quot;, &quot;one&quot;]).dictionary_encode()
  &gt;&gt;&gt; 
  &gt;&gt;&gt; class AnnotatedType(pa.ExtensionType):
  ...     def __init__(self, storage_type, annotation):
  ...         self.annotation = annotation
  ...         super().__init__(storage_type, &quot;my:app&quot;)
  ...     def __arrow_ext_serialize__(self):
  ...         return json.dumps(self.annotation).encode()
  ...     @classmethod
  ...     def __arrow_ext_deserialize__(cls, storage_type, serialized):
  ...         annotation = json.loads(serialized.decode())
  ...         return cls(storage_type, annotation)
  ...     @property
  ...     def num_buffers(self):
  ...         return self.storage_type.num_buffers
  ...     @property
  ...     def num_fields(self):
  ...         return self.storage_type.num_fields
  ... 
  &gt;&gt;&gt; b = a.cast(AnnotatedType(a.type, {&quot;some&quot;: &quot;data&quot;}))
  Traceback (most recent call last):
    File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
    File &quot;pyarrow/array.pxi&quot;, line 825, in pyarrow.lib.Array.cast
    File &quot;/home/jpivarski/miniconda3/lib/python3.9/site-packages/pyarrow/compute.py&quot;, line 309, in cast
      return call_function(&quot;cast&quot;, [arr], options)
    File &quot;pyarrow/_compute.pyx&quot;, line 528, in pyarrow._compute.call_function
    File &quot;pyarrow/_compute.pyx&quot;, line 327, in pyarrow._compute.Function.call
    File &quot;pyarrow/error.pxi&quot;, line 143, in pyarrow.lib.pyarrow_internal_check_status
    File &quot;pyarrow/error.pxi&quot;, line 120, in pyarrow.lib.check_status
  pyarrow.lib.ArrowNotImplementedError: Unsupported cast from dictionary&lt;values=string, indices=int32, ordered=0&gt; to extension&lt;my:app&lt;AnnotatedType&gt;&gt; (no available cast function for target type)
  </code></pre>
  <p><strong>Can DictionaryArrays have ExtensionType?</strong> (Calling <a href="https://arrow.apache.org/docs/python/generated/pyarrow.Array.html#pyarrow.Array.dictionary_encode" rel="nofollow noreferrer">dictionary_encode</a> on an array with ExtensionType is not an option. My DictionaryArrays will come fully built; I'm hoping to rebuild them with a new type without expanding them out.)</p>