title: How to identify, what is stalling the system in Linux?
body: <p>I have an embedded system, when I do the user i/o operations, the system just stalls. It does the action after a long time. This system is quite complex and has many process running. My question is how can I identify what is making the system stall - it does nothing literally for 5 minutes. After 5 minutes, I see the outcome. I really don't know what is stalling the system. Any inputs on how to debug this issue. I have run the top on the system. However, it doesn't lead to any issue. See here, the jup_render is just taking 30% of CPU, which is not enough to stall the system. So, I am not sure whether top is useful here or  not.</p>
  
  <p>~ # top</p>
  
  <p>top - 12:01:05 up 21 min,  1 user,  load average: 1.49, 1.26, 0.87
  Tasks: 116 total,   2 running, 114 sleeping,   0 stopped,   0 zombie
  Cpu(s): 44.4%us, 13.9%sy,  0.0%ni, 40.3%id,  0.0%wa,  0.0%hi,  1.4%si,  0.0%st
  Mem:    822572k total,   389640k used,   432932k free,     1980k buffers
  Swap:        0k total,        0k used,        0k free,   227324k cached</p>
  
  <pre><code> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND          
  850 root      20   0  309m  32m  16m S   30  4.0   3:10.88 jup_render                                                                            
    870 root      20   0  221m  13m  10m S   27  1.7   2:28.78 jup_render                                                                            
    688 root      20   0 1156m 4092 3688 S   11  0.5   1:25.49 rxserver                                                                               
      9 root      20   0     0    0    0 S    2  0.0   0:06.81 ksoftirqd/1                                                                            
     16 root      20   0     0    0    0 S    1  0.0   0:06.87 ksoftirqd/3                                                                            
   9294 root      20   0  1904  616  508 R    1  0.1   0:00.10 top                                                                                    
    812 root      20   0  865m  85m  46m S    1 10.7   1:21.17 lippo_main                                                                               
     13 root      20   0     0    0    0 S    1  0.0   0:06.59 ksoftirqd/2                                                                            
    800 root      20   0  223m 8316 6268 S    1  1.0   0:08.30 rat-cadaemon                                                                        
      3 root      20   0     0    0    0 S    1  0.0   0:05.94 ksoftirqd/0                                                                            
   1456 root      20   0 80060  10m 8208 S    1  1.2   0:04.82 jup_render                                                                            
   1330 root      20   0  202m  10m 8456 S    0  1.3   0:06.08 jup_render                                                                            
   8905 root      20   0  1868  556  424 S    0  0.1   0:02.91 dropbear                                                                               
   1561 root      20   0 80084  10m 8204 S    0  1.2   0:04.92 jup_render                                                                            
    753 root      20   0 61500 7376 6184 S    0  0.9   0:04.06 ale_app                                                                                
   1329 root      20   0 79908   9m 8208 S    0  1.2   0:04.77 jup_render                                                                            
    631 dbus      20   0  3248 1636  676 S    0  0.2   0:13.10 dbus-daemon                                                                            
   1654 root      20   0 80068  10m 8204 S    0  1.2   0:04.82 jup_render                                                                            
    760 root      20   0  116m  15m  12m S    0  1.9   0:10.19 jup_server                                                                            
      8 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/1:0                                                                            
      2 root      20   0     0    0    0 S    0  0.0   0:00.00 kthreadd                                                                               
      7 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/1                                                                            
    170 root       0 -20     0    0    0 S    0  0.0   0:00.00 kblockd                                                                                
      6 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/0                                                                            
    167 root      20   0     0    0    0 S    0  0.0   0:00.00 sync_supers                                                                            
    281 root       0 -20     0    0    0 S    0  0.0   0:00.00 nfsiod   
  </code></pre>