title: How to debug a hanging job resulting from reading from lustre?
body: <p>I have a job in interruptible sleep state (S), hanging for a few hours.</p>
  <ul>
  <li>can't use gdb (gdb will hang when attaching to the PID).</li>
  <li>can't use strace, strace will resume the hanging job =(</li>
  </ul>
  <p>WCHAN field shows the PID is waiting for ptlrpc. After some search online, it looks like this is a lustre operation. The print files also revealed the program is stuck in reading data from lustre. Any idea or suggestion on how to proceed the diagnose? Or possible reason why the hanging happens?</p>