title: sge All queues dropped because of overload or full
body: <p>I'm going to run a million batch jobs with " sge ".</p>
  
  <p>Approximately 10,000 jobs are well executed, but after an hour of execution, they stop running.</p>
  
  <p>After about an hour's run, the process slows down and eventually stops.</p>
  
  <p>Checking the error message does not confirm any errors.</p>
  
  <p>i can check the message below only.</p>
  
  <p>"All queues dropped because of overload or full" </p>
  
  <p>How do I set up the layout to run normally?</p>
  
  <p>there is one master server and four clients and files share using nfs
  and every system run on docker and docker-swirm</p>
  
  <p>do qstat when job execution speed was slow down</p>
  
  <pre><code>$qstat -j
  queue instance "peteris.q@sge00" dropped because it is full
  queue instance "peteris.q@sge02" dropped because it is full
  queue instance "peteris.q@sge03" dropped because it is full
  queue instance "peteris.q@sge01" dropped because it is full
  All queues dropped because of overload or full
  </code></pre>
  
  <p>detail messages</p>
  
  <pre><code>$qstat -j 1595799
  =============================================================
  job_number:                 1595799
  exec_file:                  job_scripts/1595799
  submission_time:            Sun May 27 08:08:10 2018
  owner:                      root
  uid:                        0
  group:                      root
  gid:                        0
  sge_o_home:                 /root
  sge_o_path:                         /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  sge_o_workdir:              /data/23andMe
  sge_o_host:                 sge
  account:                    sge
  cwd:                        /data/23andMe
  mail_list:                  root@sge
  notify:                     FALSE
  job_name:                   python3
  jobshare:                   0
  env_list:
  job_args:                   lineage.py,makeShell/1009_user3130_user3600.list
  script_file:                python3
  usage    1:                 cpu=00:00:02, mem=0.59503 GBs, io=0.03963,     vmem=493.180M, maxvmem=493.180M
  scheduling info:            queue instance "peteris.q@sge00" dropped because     it is full
                          queue instance "peteris.q@sge02" dropped because it is full
                          queue instance "peteris.q@sge03" dropped because it is full
                          queue instance "peteris.q@sge01" dropped because it is full
                          All queues dropped because of overload or full
  </code></pre>
  
  <p>sge config</p>
  
  <pre><code>algorithm default
  schedule_interval 0:0:10
  maxujobs 0
  queue_sort_method load
  job_load_adjustments np_load_avg=100.0
  load_adjustment_decay_time 0:7:30
  load_formula np_load_avg
  schedd_job_info true
  flush_submit_sec 2
  flush_finish_sec 2
  params none
  reprioritize_interval 0:0:0
  halftime 168
  usage_weight_list cpu=1.000000,mem=0.000000,io=0.000000
  compensation_factor 5.000000
  weight_user 0.250000
  weight_project 0.250000
  weight_department 0.250000
  weight_job 0.250000
  weight_tickets_functional 0
  weight_tickets_share 0
  share_override_tickets TRUE
  share_functional_shares TRUE
  max_functional_jobs_to_schedule 200
  report_pjob_tickets TRUE
  max_pending_tasks_per_job 50
  halflife_decay_list none
  policy_hierarchy OFS
  weight_ticket 0.500000
  weight_waiting_time 0.278000
  weight_deadline 3600000.000000
  weight_urgency 0.500000
  weight_priority 0.000000
  max_reservation 0
  default_duration INFINITY
  </code></pre>
  
  <p>sge queue config</p>
  
  <pre><code>qname                 peteris.q
  hostlist              @allhosts
  seq_no                0
  load_thresholds       NONE
  suspend_thresholds    NONE
  nsuspend              1
  suspend_interval      00:00:05
  priority              0
  min_cpu_interval      00:00:05
  processors            UNDEFINED
  qtype                 BATCH INTERACTIVE
  ckpt_list             NONE
  pe_list               make
  rerun                 FALSE
  slots                 20
  tmpdir                /tmp
  shell                 /bin/bash
  prolog                NONE
  epilog                NONE
  shell_start_mode      posix_compliant
  starter_method        NONE
  suspend_method        NONE
  resume_method         NONE
  terminate_method      NONE
  notify                00:00:01
  owner_list            NONE
  user_lists            NONE
  xuser_lists           NONE
  subordinate_list      NONE
  complex_values        NONE
  projects              NONE
  xprojects             NONE
  calendar              NONE
  initial_state         default
  s_rt                  INFINITY
  h_rt                  INFINITY
  s_cpu                 INFINITY
  h_cpu                 INFINITY
  s_fsize               INFINITY
  h_fsize               INFINITY
  s_data                INFINITY
  h_data                INFINITY
  s_stack               INFINITY
  h_stack               INFINITY
  s_core                INFINITY
  h_core                INFINITY
  s_rss                 INFINITY
  h_rss                 INFINITY
  s_vmem                INFINITY
  h_vmem                INFINITY
  </code></pre>