1. A method comprising: partitioning a distributed file system into a plurality of shards; for each of the plurality of shards, creating a shard queue corresponding to the shard to improve indexing speed to near real-time; receiving, via a near real time publish application program interface (API), data to be indexed in the distributed file system; determining a shard, from the plurality of shards, responsible for the data to be indexed, the determining the shard responsible for the data to be indexed comprising: retrieving a mapping between tenant identifications and shards from a distributed database; and identifying a shard that corresponds to the tenant identification received with the data to be indexed from the mapping; and generating a message in a shard queue corresponding to the shard responsible for the data to be indexed, the message indicating that data needs to be urgently indexed, the detection of the message in the shard queue by a near real time manager corresponding to the shard responsible for the data to be indexed causing the near real time manager to cause the data to be indexed.