title: Making your .NET language step correctly in the debugger
body: <p>Firstly, I apologize for the length of this question. </p>
  
  <p>I am the author of <a href="http://ironscheme.codeplex.com/" rel="nofollow noreferrer">IronScheme</a>. Recently I have been working hard on emitting decent debug info, so that I can use the 'native' .NET debugger. </p>
  
  <p>While this has been partly successful, I am running into some teething problems.</p>
  
  <p>The first problem is related to stepping.</p>
  
  <p>Due to Scheme being an expression language, everything tends to be wrapped in parenthesis, unlike the major .NET languages which seems to be statement (or line) based.</p>
  
  <p>The original code (Scheme) looks like:</p>
  
  <pre><code>(define (baz x)
    (cond
      [(null? x) 
        x]
      [(pair? x) 
        (car x)]
      [else
        (assertion-violation #f "nooo" x)]))
  </code></pre>
  
  <p>I have on purpose laid out each expression on a newline.</p>
  
  <p>The emitted code transforms to C# (via ILSpy) looks like:</p>
  
  <pre><code>public static object ::baz(object x)
  {
    if (x == null)
    {
      return x;
    }
    if (x is Cons)
    {
      return Builtins.Car(x);
    }
    return #.ironscheme.exceptions::assertion-violation+(
       RuntimeHelpers.False, "nooo", Builtins.List(x));
  }
  </code></pre>
  
  <p>As you can see, pretty simple. </p>
  
  <p>Note: If the code was transformed into a conditional expression (?:) in C#, the whole thing would just be one debug step, keep that in mind.</p>
  
  <p>Here is IL output with source and line numbers:</p>
  
  <pre><code>  .method public static object  '::baz'(object x) cil managed
    {
      // Code size       56 (0x38)
      .maxstack  6
      .line 15,15 : 1,2 ''
  //000014: 
  //000015: (define (baz x)
      IL_0000:  nop
      .line 17,17 : 6,15 ''
  //000016:   (cond
  //000017:     [(null? x) 
      IL_0001:  ldarg.0
      IL_0002:  brtrue     IL_0009
  
      .line 18,18 : 7,8 ''
  //000018:       x]
      IL_0007:  ldarg.0
      IL_0008:  ret
  
      .line 19,19 : 6,15 ''
  //000019:     [(pair? x) 
      .line 19,19 : 6,15 ''
      IL_0009:  ldarg.0
      IL_000a:  isinst [IronScheme]IronScheme.Runtime.Cons
      IL_000f:  ldnull
      IL_0010:  cgt.un
      IL_0012:  brfalse    IL_0020
  
      IL_0017:  ldarg.0
      .line 20,20 : 7,14 ''
  //000020:       (car x)]
      IL_0018:  tail.
      IL_001a:  call object [IronScheme]IronScheme.Runtime.Builtins::Car(object)
      IL_001f:  ret
  
      IL_0020:  ldsfld object 
           [Microsoft.Scripting]Microsoft.Scripting.RuntimeHelpers::False
      IL_0025:  ldstr      "nooo"
      IL_002a:  ldarg.0
      IL_002b:  call object [IronScheme]IronScheme.Runtime.Builtins::List(object)
      .line 22,22 : 7,40 ''
  //000021:     [else
  //000022:       (assertion-violation #f "nooo" x)]))
      IL_0030:  tail.
      IL_0032:  call object [ironscheme.boot]#::
         'ironscheme.exceptions::assertion-violation+'(object,object,object)
      IL_0037:  ret
    } // end of method 'eval-core(033)'::'::baz'
  </code></pre>
  
  <p><em>Note:</em> To prevent the debugger from simply highlighting the entire method, I make the method entry point just 1 column wide.</p>
  
  <p>As you can see, each expression maps correctly to a line.</p>
  
  <p>Now the problem with stepping (tested on VS2010, but same/similar issue on VS2008):</p>
  
  <p>These are with <code>IgnoreSymbolStoreSequencePoints</code> not applied.</p>
  
  <ol>
  <li>Call baz with null arg, it works correctly. (null? x) followed by x.</li>
  <li>Call baz with Cons arg, it works correctly. (null? x) then (pair? x) then (car x).</li>
  <li>Call baz with other arg, it fails. (null? x) then (pair? x) then (car x) then (assertion-violation ...).</li>
  </ol>
  
  <p>When applying <code>IgnoreSymbolStoreSequencePoints</code> (as recommended):</p>
  
  <ol>
  <li>Call baz with null arg, it works correctly. (null? x) followed by x.</li>
  <li>Call baz with Cons arg, it fails. (null? x) then (pair? x).</li>
  <li>Call baz with other arg, it fails. (null? x) then (pair? x) then (car x) then (assertion-violation ...).</li>
  </ol>
  
  <p>I also find in this mode that some lines (not shown here) are incorrectly highlighted, they are off by 1.</p>
  
  <p>Here are some ideas what could be the causes:</p>
  
  <ul>
  <li>Tailcalls confuses the debugger</li>
  <li>Overlapping locations (not shown here) confuses the debugger (it does so very well when setting a breakpoint)</li>
  <li>????</li>
  </ul>
  
  <p>The second, but also serious, issue is the debugger failing to break/hit breakpoints in some cases.</p>
  
  <p>The only place where I can get the debugger to break correctly (and consistantly), is at the method entry point. </p>
  
  <p>The situation gets a bit better when <code>IgnoreSymbolStoreSequencePoints</code> is not applied.</p>
  
  <p><strong>Conclusion</strong></p>
  
  <p>It might be that the VS debugger is just plain buggy  :(</p>
  
  <p><strong>References:</strong></p>
  
  <ol>
  <li><a href="https://stackoverflow.com/questions/4357420/making-a-clr-net-language-debuggable">Making a CLR/.NET Language Debuggable</a></li>
  </ol>
  
  <p><strong>Update 1:</strong></p>
  
  <p>Mdbg does not work for 64-bit assemblies. So that is out. I have no more 32-bit machines to test it on. <em>Update:</em> I am sure this is no big problem, does anyone have a fix? <em>Edit:</em> Yes, silly me, just start mdbg under the x64 command prompt :)</p>
  
  <p><strong>Update 2:</strong></p>
  
  <p>I have created a C# app, and tried to dissect the line info. </p>
  
  <p>My findings:</p>
  
  <ul>
  <li>After any <code>brXXX</code> instruction you need to have a sequence point (if not valid aka '#line hidden', emit a <code>nop</code>).</li>
  <li>Before any <code>brXXX</code> instruction,  emit a '#line hidden' and a <code>nop</code>.</li>
  </ul>
  
  <p>Applying this, does not however fix the issues (alone?). </p>
  
  <p>But adding the following, gives the desired result  :)</p>
  
  <ul>
  <li>After <code>ret</code>, emit a '#line hidden' and a <code>nop</code>.</li>
  </ul>
  
  <p>This is using the mode where <code>IgnoreSymbolStoreSequencePoints</code> is not applied. When applied, some steps are still skipped :(</p>
  
  <p>Here is the IL output when above has been applied:</p>
  
  <pre><code>  .method public static object  '::baz'(object x) cil managed
    {
      // Code size       63 (0x3f)
      .maxstack  6
      .line 15,15 : 1,2 ''
      IL_0000:  nop
      .line 17,17 : 6,15 ''
      IL_0001:  ldarg.0
      .line 16707566,16707566 : 0,0 ''
      IL_0002:  nop
      IL_0003:  brtrue     IL_000c
  
      .line 16707566,16707566 : 0,0 ''
      IL_0008:  nop
      .line 18,18 : 7,8 ''
      IL_0009:  ldarg.0
      IL_000a:  ret
  
      .line 16707566,16707566 : 0,0 ''
      IL_000b:  nop
      .line 19,19 : 6,15 ''
      .line 19,19 : 6,15 ''
      IL_000c:  ldarg.0
      IL_000d:  isinst     [IronScheme]IronScheme.Runtime.Cons
      IL_0012:  ldnull
      IL_0013:  cgt.un
      .line 16707566,16707566 : 0,0 ''
      IL_0015:  nop
      IL_0016:  brfalse    IL_0026
  
      .line 16707566,16707566 : 0,0 ''
      IL_001b:  nop
      IL_001c:  ldarg.0
      .line 20,20 : 7,14 ''
      IL_001d:  tail.
      IL_001f:  call object [IronScheme]IronScheme.Runtime.Builtins::Car(object)
      IL_0024:  ret
  
      .line 16707566,16707566 : 0,0 ''
      IL_0025:  nop
      IL_0026:  ldsfld object 
        [Microsoft.Scripting]Microsoft.Scripting.RuntimeHelpers::False
      IL_002b:  ldstr      "nooo"
      IL_0030:  ldarg.0
      IL_0031:  call object [IronScheme]IronScheme.Runtime.Builtins::List(object)
      .line 22,22 : 7,40 ''
      IL_0036:  tail.
      IL_0038:  call object [ironscheme.boot]#::
        'ironscheme.exceptions::assertion-violation+'(object,object,object)
      IL_003d:  ret
  
      .line 16707566,16707566 : 0,0 ''
      IL_003e:  nop
    } // end of method 'eval-core(033)'::'::baz'
  </code></pre>
  
  <p><strong>Update 3:</strong></p>
  
  <p>Problem with above 'semi-fix'. Peverify reports errors on all methods due to the <code>nop</code> after <code>ret</code>. I dont understand the problem really. How can a <code>nop</code> break verification after a <code>ret</code>. It is like dead code (except that it is NOT even code) ... Oh well, experimentation continues.</p>
  
  <p><strong>Update 4:</strong></p>
  
  <p>Back at home now, removed the 'unverifiable' code, running on VS2008 and things are a lot worse. Perhaps running unverifiable code for the sake of proper debugging might be the answer. In 'release' mode, all output would still be verifiable.</p>
  
  <p><strong>Update 5:</strong></p>
  
  <p>I have now decided my above idea is the only viable option for now. Although the generated code is unverifiable, I have yet to find any <code>VerificationException</code>'s. I dont know what the impact will be on the end user with this scenario. </p>
  
  <p>As a bonus, my second issue has also be solved. :)</p>
  
  <p>Here is a little <a href="http://www.screencast.com/t/Tjrkjl7By" rel="nofollow noreferrer">screencast</a> of what I ended up with. It hits breakpoints, does proper stepping (in/out/over), etc. All in all, the desired effect.</p>
  
  <p>I, however, am still not accepting this as the way to do it. It feel overly-hacky to me. Having a confirmation on the real issue would be nice.</p>
  
  <p><strong>Update 6:</strong></p>
  
  <p>Just had the change to test the code on VS2010, there seems to be some problems:</p>
  
  <ol>
  <li><strike>The first call now does not step correctly. (assertion-violation ...) is hit. Other cases works fine.</strike> Some old code emitted unnecessary positions. Removed the code, works as expected. :)</li>
  <li>More seriously, breakpoints fail on the second invocation of the program (using in-memory compilation, dumping assembly to file seems to make breakpoints happy again).</li>
  </ol>
  
  <p>Both these cases work correctly under VS2008. The main difference is that under VS2010, the entire application is compiled for .NET 4 and under VS2008, compiles to .NET 2. Both running 64-bit.</p>
  
  <p><strong>Update 7:</strong></p>
  
  <p>Like mentioned, I got mdbg running under 64-bit. Unfortunately, it also have the breakpoint issue where it fails to break if I rerun the program (this implies it gets recompiled, so not using the same assembly, but still using the same source).</p>
  
  <p><strong>Update 8:</strong></p>
  
  <p>I have <a href="https://connect.microsoft.com/VisualStudio/feedback/details/684089/" rel="nofollow noreferrer">filed a bug</a> at the MS Connect site regarding the breakpoint issue. </p>
  
  <p>Update: Fixed</p>
  
  <p><strong>Update 9:</strong></p>
  
  <p>After some long thinking, the only way to make the debugger happy seems to be doing SSA, so every step can be isolated and sequential. I am yet to prove this notion though. But it seems logical. Obviously, cleaning up temps from SSA will break debugging, but that is easy to toggle, and leaving them does not have much overhead.</p>