TransportError(503, u'') when tying to use a recently created elasticsearch index
Question: I'm creating an Elasticseach index using Python API like this:

    
    
    from elasticsearch import Elasticsearch
    
    es = Elasticsearch()
    
    index_body = {"mappings": {".percolator": {"properties": {"message": {"type": "string", "analyzer": "english"}}}}}
    # Creates the index if it doesn't exist
    if not es.indices.exists('test'):
        es.indices.create(index='test', body=index_body)
    
    print es.exists(index='test', id='1')
    

The index is created successfully, but when I check for the existence of a
document inside the index it fails with this error:

    
    
    Traceback (most recent call last):
      File "main.py", line 12, in <module>
        print es.exists(index='test', id='1')
      File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 68, in _wrapped
        return func(*args, params=params, **kwargs)
      File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py", line 282, in exists
        self.transport.perform_request('HEAD', _make_path(index, doc_type, id), params=params)
      File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 307, in perform_request
        status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
      File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.py", line 86, in perform_request
        self._raise_error(response.status, raw_data)
      File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py", line 102, in _raise_error
        raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
    elasticsearch.exceptions.TransportError: TransportError(503, u'')
    

If I run this script a second time, with the index already created it works
just fine. Does anyone have an idea of what could be going wrong?


Answer: When creating a new index, you need to wait until all shards are allocated.

The best way I know of how to do is this:

  1. fetch `<your_index>/_status`
  2. iterate over all `indices.<your_index>.shards` and verify that `routing.state = STARTED` everywhere
  3. Go to 1) unless all shards are started

[Here's a (PHP)
project](https://github.com/ruflin/Elastica/blob/master/test/lib/Elastica/Test/Base.php#L81-L92)
which did this for unit tests:

    
    
    protected function _waitForAllocation(Index $index)
    {
        do {
            $settings = $index->getStatus()->get();
            $allocated = true;
            foreach ($settings['shards'] as $shard) {
                if ($shard[0]['routing']['state'] != 'STARTED') {
                    $allocated = false;
                }
            }
        } while (!$allocated);
    }
    

**Wrong answer:**

You've to give ES _some space_ before your continue. ES is _near realtime_ ,
so delays are to be expected. Especially when you're running your code
sequentially with next to no delay.

I think you simply need to call the [`_refresh`
endpoint](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-
refresh.html) and you're covered.

In practice I've to do the same in my unit tests. They execute very fast and
creating/pumping data/destroying indices takes time so in my `setUp()` I've a
call to `_refresh` before handing over to the respective `test*()` method. And
in some tests, where I index data, I also have to place `_refresh` calls.

Usually, during _normal operation_ you don't need to call it shouldn't. Keep
in mind that the default `refresh_interval` is `1s`. If you're regularly
update your index and expect sub-second updates to be reflected (I'm talking
about e.g. `_search`), that's where you need to start.

