class_docstr: Responsible for providing the indices of all files for current shard.
  If it is the path of hdfs, there is also responsible for
  downloading files and organizing resources.
docstr: Download all training set for i-th shard from hdfs
code: def _prepare_sharded_trainset_from_hdfs(self, i):
      """Download all training set for i-th shard from hdfs"""
      if is_main_worker():
          itr = self.shard_ids[i]
          # Not downloading already downloaded files
          if self.cur_shard is not None and i == 0:
              itr = set(itr) - set(self.cur_shard)
          lpath, rpath = (
              self.data_para["local_path"],
              self.data_para["real_path"],
          )
          procs = []
          for k in itr:
              split_k = "train" + (str(k) if k > 0 else "")
              procs.append(
                  set_subprocess(
                      f"hadoop fs -get {rpath}/{split_k}.* {lpath}"
                  )
              )
          self.pre_procs = procs
      sync_all_workers()
  
imports: import itertools
  import json
  import logging
  import os
  import shutil
  import subprocess
  from pathlib import Path
  from argparse import Namespace
  import numpy
  from fairseq import metrics, options, utils
  from fairseq.data import AppendTokenDataset, ConcatDataset, LanguagePairDataset, PrependTokenDataset, StripTokenDataset, TruncateDataset, data_utils, encoders, indexed_dataset
  import torch.distributed
  from fairseq.tasks import LegacyFairseqTask, register_task
  from fairseq.tasks.translation import TranslationTask
label_desc: Use image node_id 4 for calling the ShardIterator obj's underlying member method code with example usage: obj._prepare_sharded_trainset_from_hdfs(i) without return types