title: ElasticSearch nested query filters and dynamic data
body: <p>I have an index created using the following pyelasticsearch code:</p>
  
  <p><strong>EDIT: UPDATED AGAIN 11/12/13 18:31 GMT</strong></p>
  
  <pre><code>    entry_mapping = {
          'product': {
              'properties': {
                  '_advertiser_id': {'type': 'integer'},
                  'advertiser': {'type': 'string'},
                  'category': {'type': 'string'},
                  'created_at': {'type': 'date'},
                  'description': {'type': 'string'},
                  'fields': {
                      'type': 'nested',
                      'properties': {
                          'gender': {'type': 'string'},
                          'short_type': {'type': 'string'}
                      }
                  },
                  'id': {'type': 'string'},
                  'name': {'type': 'string'},
                  'price': {'type': 'float'},
                  'product_type': {'type': 'string'},
                  'updated_at': {'type': 'date'},
                  'variations': {'type': 'nested'},
              }
          }
      }
  
      es.create_index('product', settings={'mappings': entry_mapping})
  </code></pre>
  
  <p>Query mapping returned using <code>curl -XGET localhost:9200/products/_mapping</code> after data has been imported:</p>
  
  <pre><code>{
    "product" : {
      "product" : {
        "properties" : {
          "_advertiser_id" : {
            "type" : "integer"
          },
          "advertiser" : {
            "type" : "string"
          },
          "category" : {
            "type" : "string"
          },
          "created_at" : {
            "type" : "date",
            "format" : "dateOptionalTime"
          },
          "description" : {
            "type" : "string"
          },
          "fields" : {
            "type" : "nested",
            "properties" : {
              "gender" : {
                "type" : "string"
              },
              "short_type" : {
                "type" : "string"
              }
            }
          },
          "id" : {
            "type" : "string"
          },
          "images" : {
            "properties" : {
              "id" : {
                "type" : "string"
              },
              "url" : {
                "type" : "string"
              }
            }
          },
          "name" : {
            "type" : "string"
          },
          "price" : {
            "type" : "float"
          },
          "product_type" : {
            "type" : "string"
          },
          "updated_at" : {
            "type" : "date",
            "format" : "dateOptionalTime"
          },
          "variations" : {
            "type" : "nested",
            "properties" : {
              "colour" : {
                "type" : "string"
              },
              "female_tops" : {
                "type" : "string"
              },
              "image" : {
                "type" : "string"
              },
              "length" : {
                "type" : "string"
              },
              "size" : {
                "type" : "string"
              },
              "sleeve_length" : {
                "type" : "string"
              },
              "type" : {
                "type" : "string"
              },
              "zip_type" : {
                "type" : "string"
              }
            }
          }
        }
      }
    }
  }
  </code></pre>
  
  <p>I am successfully querying using the following query:</p>
  
  <pre><code>curl -XGET 'http://127.0.0.1:9200/products/_search?size=100' -d '{"query": {"filtered": {"query": {"query_string": {"query": "t-shirt"}}}}}'
  </code></pre>
  
  <p>The following is an example result:</p>
  
  <pre><code>{
     "_index":"product",
     "_type":"product",
     "_id":"525adf3fd1f4677e32d0f996",
     "_score":0.034907393,
     "_source":{
        "category":"T-Shirts",
        "advertiser":"string",
        "product_type":"Clothing",
        "description":"string",
        "fields":{
           "gender":"M"
        },
        "created_at":"2013-10-07T13:24:03.182000",
        "variations":[
           {
              "colour":"Grey",
              "sleeve_length":"Short sleeved",
              "size":"S"
           },
           {
              "colour":"Grey",
              "sleeve_length":"Short sleeved",
              "size":"M"
           },
           {
              "colour":"Grey",
              "sleeve_length":"Short sleeved",
              "size":"L"
           }
        ],
        "updated_at":"2013-10-19T13:54:29.796000",
        "price":12.0,
        "images":[
           {
              "url":"https://s3-eu-west-1.amazonaws.com/...",
              "id":"525adf23d1f4677e32d0f994",
              "resource_uri":""
           },
           {
              "url":"https://s3-eu-west-1.amazonaws.com/...",
              "id":"525adf30d1f4677e32d0f995",
              "resource_uri":""
           }
        ],
        "_advertiser_id":4,
        "id":"525adf3fd1f4677e32d0f996",
        "name":"Fresh Charcoal"
     }
  }
  </code></pre>
  
  <p>I am trying to execute the following query using pyelsticsearch.</p>
  
  <pre><code>    self.query = {
          'query': {
              'filtered': {
                  'query': {
                      'query_string': {'query': self.query_str}
                  },
                  'filter': {
                      'and': [
                          {
                              'range': {
                                  'price': {
                                      'gte': self.min_price,
                                      'lte': self.max_price
                                  }
                              },
                          },
                          {
                              'terms': {
                                  '_advertiser_id': self.advertisers,
                              },
                          },
                          {
                              'term': {
                                  'fields.gender': self.gender.lower(),
                              },
                          },
                          {
                              'nested': {
                                  'path': 'variations',
                                  'query': {'match_all': {}},
                                  'filter': {
                                      'and': [
                                          {
                                              'terms': {
                                                  'variations.size': [s.lower() for s in self.sizes]
                                              },
                                          },
                                          {
                                              'term': {
                                                  'variations.colour': self.colour.lower(),
                                              }
                                          }
                                      ]
                                  }
                              }
                          },
                      ]
                  },
              }
          }
      }
  </code></pre>
  
  <p>Unfortunately it fails to return any results when there is data matching the query. Any help would be greatly appreciated.</p>
  
  <p><strong>UPDATE: 12/12/13 11:40 GMT</strong></p>
  
  <p>Below is an example of the JSON produced by the query code above.</p>
  
  <pre><code>curl -XGET 'http://127.0.0.1:9200/product/_search?size=100' -d '
  {
     "query":{
        "filtered":{
           "filter":{
              "and":[
                 {
                    "range":{}
                 },
                 {
                    "terms":{
                       "_advertiser_id":[
                          7,
                          4
                       ]
                    }
                 },
                 {
                    "term":{
                       "fields.gender":"m"
                    }
                 },
                 {
                    "nested":{
                       "filter":{
                          "and":[
                             {
                                "terms":{
                                   "variations.size":[
                                      "xs",
                                      "s",
                                      "m",
                                      "l",
                                      "xl",
                                      "xxl"
                                   ]
                                }
                             },
                             {
                                "term":{
                                   "variations.colour":"black"
                                }
                             }
                          ]
                       },
                       "path":"variations",
                       "query":{
                          "match_all":{
  
                          }
                       }
                    }
                 }
              ]
           },
           "query":{
              "query_string":{
                 "query":"t-shirt"
              }
           }
        }
     }
  }'
  </code></pre>
  
  <p><strong>UDPATED: 12/12/13 11:51 GMT</strong></p>
  
  <p>Things get stranger. Having stripped down the query the following gives results.</p>
  
  <pre><code>curl -XGET 'http://127.0.0.1:9200/product/_search?size=100' -d '{
     "query":{
        "filtered":{
           "filter":{
              "and":[
                 {
                    "nested":{
                       "filter":{
                          "an":[
                             {
                                "terms":{
                                   "variations.size":[
                                      "xs",
                                      "s",
                                      "m",
                                      "l",
                                      "xl",
                                      "xxl"
                                   ]
                                }
                             },
                             {
                                "term":{
                                   "variations.colour":"black"
                                }
                             }
                          ]
                       },
                       "path":"variations",
                       "query":{
                          "match_all":{
  
                          }
                       }
                    }
                 }
              ]
           },
           "query":{
              "query_string":{
                 "query":"t-shirt"
              }
           }
        }
     }
  }'
  </code></pre>
  
  <p>Example result data from above query: </p>
  
  <pre><code>{
     "_index":"product",
     "_type":"product",
     "_id":"525ade5ad1f4677e32d0f993",
     "_score":0.10493462,
     "_source":{
        "category":"T-Shirts",
        "advertiser":"...",
        "product_type":"Clothing",
        "description":"...",
        "fields":{
           "gender":"M"
        },
        "created_at":"2013-10-07T13:24:03.182000",
        "variations":[
           {
              "colour":"Black",
              "sleeve_length":"Short sleeved",
              "size":"S"
           },
           {
              "colour":"Black",
              "sleeve_length":"Short sleeved",
              "size":"M"
           },
           {
              "colour":"Black",
              "sleeve_length":"Short sleeved",
              "size":"L"
           }
        ],
        "updated_at":"2013-10-19T14:05:34.299000",
        "price":0.0,
        "images":[
           {
              "url":"...",
              "id":"525ade50d1f4677e30a2cb3a",
              "resource_uri":""
           }
        ],
        "_advertiser_id":4,
        "id":"525ade5ad1f4677e32d0f993",
        "name":"..."
     }
  }
  </code></pre>
  
  <p>*<em>UPDATED: 21/12/2012 10:48 GMT *</em></p>
  
  <p>I have isolated the part of the query that's being problematic—i.e. not returning any results—when combined with the entirety of the query.</p>
  
  <pre><code>{
      'term': {
          'fields.gender': self.gender.lower(),
      },
  }
  </code></pre>
  
  <p>Exemplar working query:</p>
  
  <pre><code>curl -XGET 'http://127.0.0.1:9200/product/_search?size=100' -d '{
     "query":{
        "filtered":{
           "filter":{
              "and":[
                 {
                    "range":{
                       "price":{
                          "gte":0.0,
                          "lte":200.0
                       }
                    }
                 },
                 {
                    "terms":{
                       "_advertiser_id":[
                          7,
                          4
                       ]
                    }
                 },
                 {
                    "nested":{
                       "filter":{
                          "and":[
                             {
                                "terms":{
                                   "variations.size":[
                                      "xs",
                                      "s",
                                      "m",
                                      "l",
                                      "xl",
                                      "xxl"
                                   ]
                                }
                             },
                             {
                                "term":{
                                   "variations.colour":"black"
                                }
                             }
                          ]
                       },
                       "path":"variations",
                       "query":{
                          "match_all":{
  
                          }
                       }
                    }
                 }
              ]
           },
           "query":{
              "query_string":{
                 "query":"t-shirt"
              }
           }
        }
     }
  }'
  </code></pre>
  
  <p>Exemplar unworking query:</p>
  
  <pre><code>curl -XGET 'http://127.0.0.1:9200/product/_search?size=100' -d '{
     "query":{
        "filtered":{
           "filter":{
              "and":[
                 {
                    "range":{
                       "price":{
                          "gte":0.0,
                          "lte":200.0
                       }
                    }
                 },
                 {
                    "terms":{
                       "_advertiser_id":[
                          7,
                          4
                       ]
                    }
                 },
                 {
                    "term":{
                       "fields.gender":"m"
                    }
                 },
                 {
                    "nested":{
                       "filter":{
                          "and":[
                             {
                                "terms":{
                                   "variations.size":[
                                      "xs",
                                      "s",
                                      "m",
                                      "l",
                                      "xl",
                                      "xxl"
                                   ]
                                }
                             },
                             {
                                "term":{
                                   "variations.colour":"black"
                                }
                             }
                          ]
                       },
                       "path":"variations",
                       "query":{
                          "match_all":{
  
                          }
                       }
                    }
                 }
              ]
           },
           "query":{
              "query_string":{
                 "query":"t-shirt"
              }
           }
        }
     }
  }'
  </code></pre>