title: How do you use the Elasticsearch Ingest Attachment Processor Plugin with the Python package elasticsearch-dsl
body: <p>I'm having trouble trying to use the <a href="https://www.elastic.co/guide/en/elasticsearch/plugins/current/ingest-attachment.html#ingest-attachment" rel="nofollow noreferrer">Ingest Attachment Processor Plugin</a> with ElasticSearch  (5.5 on AWS, 5.6 local). I'm developing in Python (3.6) and am using the <a href="https://github.com/elastic/elasticsearch-dsl-py" rel="nofollow noreferrer">elasticsearch-dls library</a>.</p>
  
  <p>I'm using <a href="http://elasticsearch-dsl.readthedocs.io/en/latest/persistence.html" rel="nofollow noreferrer">Persistence</a> and have my class set-up like this:</p>
  
  <pre><code>import base64
  from elasticsearch_dsl.field import Attachment, Text
  from elasticsearch_dsl import DocType, analyzer
  
  lower_keyword = analyzer('keyword', tokenizer="keyword", filter=["lowercase"])
  
  class ExampleIndex(DocType):
  class Meta:
      index = 'example'
      doc_type = 'Example'
  
      id = Text()
      name = Text(analyzer=lower_keyword)
      my_file = Attachment()
  </code></pre>
  
  <p>I then have a function like this that I call to create the index and save the document.</p>
  
  <pre><code>def index_doc(a_file):
      # Ensure that the Index is created before any documents are saved
      try:
          i = Index('example')
          i.doc_type(ExampleIndex)
          i.create()
  
          # todo - Pipeline creation needs to go here - But how do you do it?
  
      except Exception:
          pass
  
      # Check for existing index
      indices = ExampleIndex()
      try:
          s = indices.search()
          r = s.query('match', name=a_file.name).execute()
          if r.success():
              for h in r:
                  indices = ExampleIndex.get(id=h.meta.id)
                  break
      except NotFoundError:
          pass
      except Exception:
          logger.exception("Something went wrong")
          raise
  
      # Populate the document   
      indices.name = a_file.name
      with open(a_file.path_to_file, 'rb') as f:
          contents = f.read()
      indices.my_file = base64.b64encode(contents).decode("ascii")
  
      indices.save(pipeline="attachment") if indices.my_file else indices.save()
  </code></pre>
  
  <p>I have a text file with the contents <em>This is a test document</em>. When it's contents are base64 encoded they become <em>VGhpcyBpcyBhIHRlc3QgZG9jdW1lbnQK</em></p>
  
  <p>If I use CURL directly then it works:</p>
  
  <p>Create the pipline:</p>
  
  <pre><code>curl -XPUT 'localhost:9200/_ingest/pipeline/attachment?pretty' -H 'Content-Type: application/json' -d' {   "description" : "Extract attachment information",   "processors" : [
      {
        "attachment" : {
          "field" : "my_file"
        }
      }   ] }
  </code></pre>
  
  <p>Put the data</p>
  
  <pre><code>curl -XPUT 'localhost:9200/example/Example/AV9nkyJMZAQ2lQ3CtsLb?pipeline=attachment&amp;pretty'\
  -H 'Content-Type: application/json' \
  -d '{"my_file": "VGhpcyBpcyBhIHRlc3QgZG9jdW1lbnQK"}'
  </code></pre>
  
  <p>Fetch the data
  <a href="http://localhost:9200/example/Example/AV9nkyJMZAQ2lQ3CtsLb?pretty" rel="nofollow noreferrer">http://localhost:9200/example/Example/AV9nkyJMZAQ2lQ3CtsLb?pretty</a></p>
  
  <pre><code>{
      "_index" : "example",
      "_type" : "Example",
      "_id" : "AV9nkyJMZAQ2lQ3CtsLb",
      "_version" : 4,
      "found" : true,
      "_source" : {
          "my_file" : "VGhpcyBpcyBhIHRlc3QgZG9jdW1lbnQK",
          "attachment" : {
              "content_type" : "text/plain; charset=ISO-8859-1",
              "language" : "en",
              "content" : "This is a test document",
              "content_length" : 25
          }
      }
  }
  </code></pre>
  
  <p>The trouble is I can not see how to recreate this using the elasticsearch-dsl Python library</p>
  
  <p><strong>UPDATE:</strong>
  I can get everything working now other than the initial creation of the pipeline. If I create the pipeline using CURL then I can use it by simply changing the .save() method call to .save(pipeline="attachment"). I've updated my earlier function to show this as well as to have a comment on where the pipline creation would need to go.</p>
  
  <p>Here is an example of the CURL implementation of creating the pipeline</p>
  
  <pre><code>curl - XPUT 'localhost:9200/_ingest/pipeline/attachment?pretty' \
       - H 'Content-Type: application/json' \
       - d '"description": "Extract attachment information","processors": [{"attachment": {"field": "my_field"}}]}'
  </code></pre>