text: # Licensed to the .NET Foundation under one or more agreements.
  # The .NET Foundation licenses this file to you under the Apache 2.0 License.
  # See the LICENSE file in the project root for more information.
  
  '''
  -------------------------------------------------------------------------------
  INTRODUCTION
  This test plan addresses debugging test coverage for the 2.x series of 
  IronPython interpreter releases.
  
  EDITORS NOTES
  Throughout this document you will find references to documentation in
  other packages similar to "See documentation for debugging.pdb_module".  This simply
  means that relative to this package (i.e., debugging), you should follow
  the 'pdb_module (package)' link at the bottom of this page.
  
  -------------------------------------------------------------------------------
  FEATURE GOALS
  The goals of IP's debugging feature is to provide IronPython users with 
  complete use of CPython's existing debugger functionality, the "pdb" and
  "bdb" modules respectively, while also providing a great interactive debugging 
  experience using Microsoft tools such as mdbg and the Visual Studio IDE.
  
  -------------------------------------------------------------------------------
  PRIMARY TESTING CONCERNS
  - new builtin module attributes added to IP to support CPython's "bdb" and "pdb"
    modules
  - compatibility with CPython's use of the "bdb" and "pdb" modules
  - what happens breakpoint-wise when IP steps into .NET code from "pdb"?
  - .NET applications hosting IP should be able to debug Python code
  - the experience of debugging IP under Microsoft's mdbg and Visual 
    Studio tools is what existing CSharp and VB.Net would expect; but still
    Pythonic
  - "-D" command-line flag
  
  -------------------------------------------------------------------------------
  PRIMARY TESTING FOCUS
  Our principal testing focus will be on testing that IP's support of the
  "pdb" module is completely compatible with that of CPython by running existing
  CPython tests, and also by creating a new generic side-by-side testing framework.
  
  The secondardy focus will be testing that debugging IP under mdbg is a
  pleasant experience.  This will initially be acheived by using the CLR Team's
  existing debugging test framework, but in the long run we will move to a
  new generic, side-by-side testing framework which can be reused for other 
  side-by-side testing areas.
  
  -------------------------------------------------------------------------------
  REFERENCES
  * IronPython Test Plan              Internal IP SharePoint site
  * Feature Specifications            None...
  * Development Documents             None...
  * Schedule Documents                None...
  * CPython pdb module                http://docs.python.org/library/pdb.html
  * CPython bdb module                http://docs.python.org/library/bdb.html
  * pdb tutorial                      http://onlamp.com/pub/a/python/2005/09/01/debugger.html
  * MDbg.exe                          http://msdn.microsoft.com/en-us/library/ms229861.aspx
  * Debugging IP in Visual Studio     http://devhawk.net/2008/05/08/Debugging+IronPython+Code+In+Visual+Studio.aspx
  * Bug queries                       http://ironpython.codeplex.com/WorkItem/AdvancedList.aspx
                                      and set the component to "Debugging"
  * Build drop location               Updated at http://ironpython.codeplex.com/SourceControl/ListDownloadableCommits.aspx
                                      on weekdays
  * source file tree                  $/.../dlr/Debugging/ and also $/.../dlr/.../IronPython/*
  
  -------------------------------------------------------------------------------
  PERSONNEL
  Program Manager:    hpierson
  Developer:          dinov
  Tester:             dfugate
  
  -------------------------------------------------------------------------------
  TESTING SCHEDULE
      
      Phase I - Test Plan and Enabling Existing Tests
          - Write test plan for IronPython Debugging feature
          - Enable any existing CPython test cases for "pdb" and "bdb" in SNAP
          - Enable any existing CPython test cases for Python features supporting 
            "pdb" and "bdb" in SNAP
          
      Phase II - Examine Coverage of Existing Cases and Write High-priority Cases
          - Verify that developer tests for new additions to IP 
            supporting "pdb" and "bdb" modules are sufficient. Write more test
            cases if not
          - Write additional test cases for the "pdb" module if CPython's test cases 
            are insufficient
          - Examine our existing mdbg test (cases) looking for holes. Write new test
            cases if needed
          - Author a few manual sanity test cases for verifying IP can 
            be debugged under the Visual Studio IDE. These should be folded into
            the CodePlex test pass signoff procedure for major releases
            
      Phase III - Design and Implement Side-by-Side Testing Framework
          - Based on the needs of test cases added in the previous phase either:
              1. Design a side-by-side testing infrastructure to be used with testing
                 IP's CPython debugging compatibility OR
              2. Look for existing side-by-side testing frameworks we could reuse.
                 The pexpect package might be useful here
          - Implement (or add existing) side-by-side testing framework
      
      Phase IV - Implement High-priority Cases
          - Implement test cases for additions to IP which support the
            "pdb" module IF NEEDED
          - Implement test cases for the "pdb" module IF NEEDED
          - Implement test cases for mdbg IF NEEDED
          
      Phase IV - Write Lower Priority Cases
          - Write test cases around stepping into (hosted) Python code from a 
            .NET application
          - Write test cases around stepping into .NET code from an ipy.exe
            process
          - Write performance test cases
                      
      Phase V - Implement Lower Priority Cases
          - Implement test cases around stepping into (hosted) Python code from a 
            .NET application
          - Implement test cases around stepping into .NET code from an ipy.exe
            process
          - Implement performance test cases
      
      Phase VI - Future
          - Develop test cases for the "bdb" module if CPython's test cases are
            insufficient
          - Automate manual sanity Visual Studio debugging tests
          - Migrate existing mdbg tests to the new side-by-side testing framework
          - Low-resource test cases and implementation
          - Reliability test cases and implementation
          - Scalability test cases and implementation
          - Stress test cases and implementation
          - Internationalization test cases and implementation
  
      Phase P - All Public Releases
          - Verify all user documentation with respect to debugging
          - Setup testing - verify pdb.py and all supporting modules are installed by the MSI
          - Generate code coverage report and verify block coverage as greater
            than 80%
  
  -------------------------------------------------------------------------------
  FEATURE HISTORY
  2.0:     Tests implemented for IronPython under mdbg. Scenarios which still 
           need to be covered are:
           * Attach scenario with and without the -D option
           * Step into/out when there is no pdb file for IP 
             binaries
           * Multiple statements in one line (e.g., "x = 1;y=2"
           * if x and \\n y: print 1
           * dict/tuple spanning multiple lines
           * break/continue, explicit raise
           * other multi-line statement scenarios such as:
             if True: print 1
             if True:\\n    print 1
           * MdbgInterop
             able to break into the C# code from Python
             able to break back into Python code from C#
  
  2.6B2:   First release of IronPython to include support of CPython's pdb module.
           Only works if -X:Frames or -X:FullFrames option supplied to ipy.exe.
           No pdb-related tests running in SNAP at this point, but we are running
           all of CPython's tests WRT supporting functions in the sys module.
           Debugging test plan completely rewritten in the form of pydoc strings,
           and made public.
  
  -------------------------------------------------------------------------------
  FEATURES:
  - sys.settrace, sys.gettrace, and sys.call_tracing.  NOTE: while each of these
    three builtin module members needs to be tested within unit tests on their
    own, the real use case here is through the "bdb" and "pdb" modules
  - ipy.exe's generation of PDBs
  - ipy.exe's PDB visualizers
  
  -------------------------------------------------------------------------------
  FILES AND MODULES:
  
  - FILES LIST:
      * everything under Microsoft.Scripting.Debugging, although the DLR 
        officially owns this
      * sys.cs
      * IronPython.Runtime.PythonTracebackListener
  - REGISTRY, INI SETTINGS: None
  - SETUP PROCEDURES: 
      CPython standard library must be installed and present in sys.path to be 
      usable
  - DE-INSTALLATION PROCEDURES
      N/A
  - DATABASE SETUP AND PROCEDURES
      N/A
  - NETWORK DOMAIN/TOPOLOGIES CONFIGURATION PROCEDURES
      N/A
  - PERFORMANCE MONITORING COUNTERS SETUP AND CONFIGURATIONS
      N/A
  
  -------------------------------------------------------------------------------
  OPERATIONAL ISSUES
  N/A.  This feature of IP is not being monitored/maintained by operational 
  staff, and IP is provided on an as-is basis.
  
  -------------------------------------------------------------------------------
  SCOPE OF TEST CASES
  We'll be getting most of our test coverage on the debugging feature of 
  IP by unit testing the CPython 'pdb' module.  Doing this thoroughly
  should guarantee we hit nearly all blocks of code supporting debugging in 
  IP DLLs. This will be validated and verified through monthly code 
  coverage runs.  As an aside, this should in theory hit most of the DLR's
  Microsoft.Scripting.Debugging.dll as well.
  
  As IP currently has no tie-in into the Visual Studio IDE, this is 
  arguably far less important to test than our support of the 'pdb' module or 
  even mdbg.  As a result, the Microsoft toolset testing emphasis will involve
  exahaustively covering all aspects of mdbg. That said, some of our users do in
  fact have VS Pro installed implying that we should perform at least minimal,
  manual sanity tests before every major public release of IP.  Should we
  ever provide visualizers for our debug symbols we'll need much more 
  comprehensive and automated tests for VS of course.
   
  -------------------------------------------------------------------------------
  ACCEPTANCE CRITERIA
  - no debugging related feature should be checked into the internal source 
    repository without some form of unit test hooked into SNAP checked in as
    well
  - we should not publically advertise the existance of any new debugging features
    in an IP release unless:
      - if the feature is CPython-based; the majority of it's corresponding 
        CPython unit test cases have been enabled in SNAP and pass
      - if the feature is novel to IP; block coverage should be greater
        than 70%
      - the feature needs to be documented in some form other than a blog
      - performance of the feature needs to be "within reason"
      - if the feature is not CPython-based; it needs a Pythonic feel to it
      - at least half of all bugs opened on the feature since its inception are
        fixed
  
  -------------------------------------------------------------------------------
  KEY FEATURE ISSUES
  - no spec on debugging IP from mdbg, VS, or what debugging symbols 
    should be generated for various Python constructs
  - insufficient test resources to adequately test this feature
  - insufficient resources to implement all debugging features called out in this
    test plan
  - existing mdbg tests are based on CLR infrastructure and require a Perl 
    installation. We cannot redistribute this publically
  - existing mdbg tests in SNAP are flakey
  
  -------------------------------------------------------------------------------
  TEST APPROACH
  - DESIGN VALIDATION
      We have no part in the design of the "pdb" and "bdb" standard modules. 
      
      As for debugging under mdbg...there is no current design document on 
      debugging Python sessions.  Generally speaking test will simply ensure that
      whatever mdbg functionality exists with regard to IP is "Pythonic",
      but at the same time familiar to CSharp and VB developers.
  
  - DATA VALIDATION
      pdb/bdb: at a very low-level, all we need to validate is that parameters passed
      to a trace function we set via sys.settrace are as expected.  In particular,
      the first parameter passed in is the current stack frame which we have a 
      very solid chance of getting wrong.
  
      mdbg/VS: need to validate that generation of PDB symbols "make sense"
      with respect to whatever Python source is being compiled. There are at 
      least two ways to accomplish this:
      1. Create a regression test in which we compare the generated PDB of a 
         known Python script to an expected PDB file.  The Python script should
         exhaustively cover everything that's possible in the Python grammar.
         Testing in this manner is not really maintainable in the long run as we
         expect the PDB generated for a given Python script to change over time
         reflecting optimizations and new features added to IP
      2. Test that the behavior of mdbg, given a known Python script, does not 
         regress.  Again, this Python script should exhaustively cover everything
         that's possible in Python's grammar, but this time around we'll also
         need to cover everything that's possible under mdbg
  
  - API TESTING
      We'll exhaustively cover the entire "pdb" standard module without ever 
      touching the CSharp APIs directly.  Prior coverage runs have shown that 
      (generally speaking) any block of code that is not directly hittable through 
      the Python API is likely dead code.  Also, testing "bdb" module is lower 
      priority as "pdb" uses this for its own implementation.  The final 
      justification for this is that there is actually very little in terms of 
      debugging APIs in IP DLLs.  Most of the debugging support is built
      directly into the DLR's Microsoft.Scripting.Debugging.dll.
      
      For mdbg/VS, we'll exhaustively cover all mdbg commands.
      
  - CONTENT TESTING
    All debugging documentation should be thoroughly reviewed by Test before any
    public release of IronPython.  Careful attention will be given to duplicating
    exactly any command fed into ipy.exe or mdbg.exe, and this should be 
    automated entirely if feasible.
    
  - LOW-RESOURCE TESTING
    Use of the debugging feature has a few side effects which might be interesting
    from a limited resources perspective.  First and foremost, use of the -X:Frames
    and -X:FullFrames IP console flags imply that IP will consume more 
    memory than under normal circumstances.  We should check that IP does not 
    consume "too much" extra memory.  Next, use of the debugging flag, '-D', in
    conjunction with the '-X:SaveAssemblies' flag will generate 
    Snippets.debug.scripting.pdb in the %TMP% directory. What occurs if the drive
    containing %TMP% is full?
  
    NOTE: while this is an interesting test area to explore, there are no current
    plans to test it.  We should perform general low-resource testing of 
    IP before tackling it for this specific feature.
    
  - SETUP TESTING
    The only option of IP's MSI installer capable of affecting
    the debugging feature is the ability to selectively install CPython's standard
    library.  As we generate the list of CPython modules to include in the MSI
    dynamically, it would be worthwhile to always check that pdb.py is included.
    
    NOTE: IronPython currently has no automated tests for the MSI. Once we
    do, we need to add a check to ensure pdb.py is always included.  For the time
    being this will have to be done manually.
    
  - MODES AND RUNTIME OPTIONS
    We should focus our efforts on testing IP with the following modes:
    * -D -X:Frames
    * -D -X:FullFrames
    * -D -X:Frames -X:SaveAssemblies
    * -D
    * -X:Frames
    * 
    using both debug and release IP assemblies.  Also note that this
    needs to be performed for both Python interactive sessions and Python scripts
    passed into ipy.exe (e.g., "ipy.exe test_str.py").  Last but
    not least, there's the %PYTHONDEBUG% environment variable to consider.
    
  - INTEROPERABILITY
    Of primary concern is that IP produces an identical experience as
    CPython when given input which utilizes the pdb module. This will be 
    accomplished via side-by-side testing and verification that the output of 
    IP is the same as CPython given an identical input.
    
    We will also verify that IP can be debugged from mdbg and the Visual
    Studio IDE.
    
  - INTEGRATION TESTING
    * breaking into .NET code from pdb
    * breaking into COM from pdb
    * breaking into IronRuby from pdb
    * breaking into Python code from DLR hosting APIs
    * breaking into Python code from IronRuby
  
  - BETA TESTING
    The second beta release of IronPython 2.6 will support the debugging feature
    in the form of very limited pdb module support with the -X:Frames flag 
    passed to ipy.exe. As with all beta releases of IP, this will be 
    released on CodePlex to the general public. We will then use feedback
    from the IronPython Community to determine the amount of effort that goes 
    into fully testing this feature.
    
  - ENVIRONMENT/SYSTEM - GENERAL
    The %PYTHONDEBUG% environment variable may have some impact on this feature.
    Other than this, the VS IDE will of course need to be installed for testing
    the VS debugging experience.
    
  - CONFIGURATION
    Need .NET 2.0 Service Pack 1 installed to run IP, and .NET 3.5
    Service Pack 1 to build the feature.
    
  - USER INTERFACE
    IP provides no user interfaces.
    
  - PERFORMANCE & CAPACITY TESTING
    Minimally, we need to run one or more tests in the perf lab under the -X:Frames
    and -X:FullFrames test modes.  We should also measure the end-to-end run time
    of a complete, yet minimal, debugging sample utilizing the pdb module in the 
    lab.  The focus here will simply be on ensuring there are no perf regressions
    which are greater than 15% for any given checkin, and also that overall 
    performance of the debugging feature remains palatable to our userbase.
    
    In terms of capacity testing we should look at:
    - if perf gets affected when breaking into deeply nested functions (e.g., 
      recursive functions)
    - anything else?
    
  - PRIVACY
    Does %TMP%\Snippets.debug.scripting.pdb contain sensitive customer data? If 
    so, does IP disclose the existence of this file?
    Is remote debugging supported? If so, what safeguards are in place?
    
  - RELIABILITY
    Most end-users are expected to enter into debugging sessions for short 
    periods of time to diagnose issues with Python code.  From this perspective
    the so-called "up time" of debugging sessions is less important than the 
    overall sporadic failure frequency. We can get a good sense of how often 
    sporadic failures occur by automatically running debugging tests in SNAP 
    for every IP checkin.
    
  - SCALABILITY
    Does debugging still work correctly when stepping into deeply nested (e.g., 1000)
    functions?
    
  - STRESS TESTING
    - gcstress
    - MDA
    - assemblies in the GAC
    - call pdb functions multiple times and look for memory leaks
    - what happens when stepping into unreasonably deeply nested functions?
    
  - VOLUME TESTING
    App building exercise or try utilizing pdb from some of our internal Python 
    tools. It would be interesting to use this from gopackage when exceptions 
    are encountered. 
    
  - INTERNATIONAL ISSUES
    As Python is not a localized language and available only in English, we 
    simply need to confirm that nothing gets localized.  This can be done by
    running a simple debugbing test against non-English OSes such as Deutsche
    Vista.
    
  - ROBUSTNESS
    We'll need a dedicated test or two in the stress lab to ensure there are no
    memory leaks.  Also, we'll need to keep an eye out for sporadic failures of
    debugging tests in SNAP.
    
  - ERROR TESTING
    What happens when invalid commands are fed to the pdb debugger? Is this 
    handled identically to CPython?
    
    Is there any conceivable way to fully break debugging without actually 
    breaking ipy.exe in general?  What happens under the mode 
    "-D -X:SaveAssemblies" when the current user doesn't have write 
    permissions on %TMP%?  What happens if MS.Scripting.Debugging.dll is 
    removed outright?
    
  - USABILITY
    There are three major usability issues with debugging:
    * IP offers no visualizer for the VS IDE
    * we're largely incompatible with CPython's pdb module
    * debugging Python code from .NET apps isn't currently possible
    
    Test will depend on feedback from the IronPython Mailing List and blogs to
    determine more usability issues. Resource permitting, we may also do 
    app building exercises.
    
    The usability goal of this feature is that there are no complaints about
    the IP debugging experience from our users.
    
  - ACCESSIBILITY
    As this feature provides no new user interfaces to IronPython. and simply 
    emits output to the stdout stream of command prompts, there should be no 
    need to invest into accessibility testing.
    
  - USER SCENARIOS
    It's anticipated that users of this feature will be limited almost exclusively to
    developers. Expert Python users will likely prefer using the pdb
    module or debugging their applications directly via the "-i" ipy.exe 
    option.  Existing Microsoft customers with little Python background will 
    probably be more comfortable with the VS IDE debugging experience. Due to 
    this we must make sure both experiences are great!
    
    The types of applications being debugged will likely:
    * be non-trivial
    * consist of many different modules/packages
    * make heavy use of the CPython standard library and/or third party Python
      packages
    * be debugged often during the application development phase
    * be debugged infrequently in a production environment when something goes 
      awry
    
    With these constraints in mind, we should try to debug existing major third
    party Python applications and/or add debugging support to our own internal
    Python tools.
    
  - BOUNDARIES AND LIMITS
    What happens when stepping into a recursive function at the maximum recursion
    level?
    
  - OPERATIONAL ISSUES
    None
    
  - SPECIAL CODE PROFILING AND OTHER METRICS
    Overall block coverage of all assemblies with "ipy" or "IronPython" in their
    names should be above 80% and file coverage should stay above 97%. Visual 
    Studio's mstest will be used to measure code coverage.
  
  -------------------------------------------------------------------------------
  TEST ENVIRONMENT
  
  - OPERATING SYSTEMS
    32-bit Windows XP
    32-bit Windows 2003
    64-bit Windows 2003
    32-bit Windows Vista
    64-bit Windows Vista
    32-bit Windows Vista (Deutsche)
  - NETWORKS
    General Intranet network connection required for SNAP.
    May be other special network needs if remote debugging is/becomes a supported scenario.
  - HARDWARE
    - MACHINES
      At least two machines of every OS variety called out above with at least
      10 gigs of free hard disk space, 2 gigs of RAM, and a modern CPU
  - SOFTWARE
    * Visual Studio 2008 Team System Service Pack 1 installed to run the 
      test suite
    * PowerShell 1.0 to run any supporting test scripts
    * CPython for side-by-side tests
    * mdbg
  
  -------------------------------------------------------------------------------
  UNIQUE TESTING CONCERNS FOR SPECIFIC FEATURES
  It should be relatively easy to drive pdb or mdbg via the command line in an
  automated fashion, but the same cannot be said about the VS IDE debugging 
  experience.  We'll need to reuse the VS team's automation infrastructure and 
  ensure any machines running these tests in SNAP are never locked at the 
  username/password screen which would cause the test to fail.  Also, should 
  visualization ever be a supported scenario, we may have no other option than to 
  test this manually (note - this needs to be investigated).
  
  -------------------------------------------------------------------------------
  AREA BREAKDOWN
  - CPython's "pdb" module
    See debugging.pdb_mod.
  
  - mdbg tool support
    See debugging.mdbg_tool.
        
  - VS IDE
    See debugging.vs.
    
  - sys module extensions
    - sys.settrace
      settrace(function) 
      Set the system's trace function, which allows you to implement a Python 
      source code debugger in Python. The function is thread-specific; for a 
      debugger to support multiple threads, it must be registered using 
      settrace() for each thread being debugged.
  
      Trace functions should have three arguments: frame, event, and arg. frame 
      is the current stack frame. event is a string: 'call', 'line', 'return', 
      'exception', 'c_call', 'c_return', or 'c_exception'. arg depends on the 
      event type.
  
      The trace function is invoked (with event set to 'call') whenever a new 
      local scope is entered; it should return a reference to a local trace 
      function to be used that scope, or None if the scope shouldn't be traced.
  
      The local trace function should return a reference to itself (or to another 
      function for further tracing in that scope), or None to turn off tracing 
      in that scope.
  
      The events have the following meaning:
      'call'
          A function is called (or some other code block entered). The global 
          trace function is called; arg is None; the return value specifies the 
          local trace function.
      'line'
          The interpreter is about to execute a new line of code (sometimes 
          multiple line events on one line exist). The local trace function is 
          called; arg is None; the return value specifies the new local trace 
          function.
      'return'
          A function (or other code block) is about to return. The local trace 
          function is called; arg is the value that will be returned. The trace 
          function's return value is ignored.
      'exception'
          An exception has occurred. The local trace function is called; arg is 
          a tuple (exception, value, traceback); the return value specifies the 
          new local trace function.
      'c_call'
          A C function is about to be called. This may be an extension function 
          or a builtin. arg is the C function object.
      'c_return'
          A C function has returned. arg is None.
      'c_exception'
          A C function has thrown an exception. arg is None.
      
      Note that as an exception is propagated down the chain of callers, an 
      'exception' event is generated at each level.
    
    - sys.gettrace
      Get the trace function as set by settrace().
      
    - sys.call_tracing
      call_tracing(func, args) -> object
  
      Call func(*args), while tracing is enabled.  The tracing state is
      saved, and restored afterwards.  This is intended to be called from
      a debugger from a checkpoint, to recursively debug some other code.
  
  -------------------------------------------------------------------------------
  TEST CASE STRUCTURE
  Test cases will be stored directly in the pydoc strings of the functions 
  implementing the tests.  In turn, the modules containing these test case 
  functions can be found under the "debugging" test package. Generating pydoc for
  the "debugging" test package and all sub-modules/sub-packages will generate 
  this test plan.
  
  -------------------------------------------------------------------------------
  SPEC REVIEW ISSUES
  There is no IronPython spec for this feature.  Any issues observed in the pdb 
  module, which acts as a spec, and reproducible under CPython should be reported 
  to http://bugs.python.org.
  
  -------------------------------------------------------------------------------
  TEST TOOLS
  One of the following:
  - reuse CLR's debugging test infrastructure.  NOTE: this is only suitable for 
    mdbg tests
  - write our own new side-by-side testing infrastructure entirely in Python
    so that we can eventually contribute pdb tests back to CPython
  - write our own new side-by-side testing infrastructure in any
    language using any framework. http://sourceforge.net/projects/pexpect/ could
    be useful as would be PowerShell
  - re-use some internal tool that might not be redistributable on CodePlex
  
  which will minimally need to support the following features:
  - ability to ignore lines (matching some regular expression) entirely
  - ability to substitute strings in lines
  - ability to run arbitrary commands (e.g., not limited to "ipy.exe ..." or "mdbg ...")
  - ability to match the current output against a reference output stored on disk
  - ability to run two or more arbitrary commands and compare their output against each other
  
  As per usual, the plan is to automate all tests using the SNAP checkin system
  and VS's mstest tool.
  
  -------------------------------------------------------------------------------
  SMOKE TEST (ACCEPTANCE TEST, BUILD VERIFICATION, ETC.)
  
  pdb acceptance test
      ipy.exe -c "import pdb"
  
  mdbg
      TODO (July 30, 2009)
      
  Visual Studio IDE
      TODO (July 30, 2009)
  
  
  -------------------------------------------------------------------------------
  AUTOMATED TESTS
  pdb and mdbg tests will be 100% automated in SNAP.
  
  
  -------------------------------------------------------------------------------
  MANUAL TESTS
  Setup testing will be manual for now, and we'll eventually move to automated 
  testing in SNAP using PowerShell support scripts.
  
  Visual Studio IDE testing is also manual for now.  Sometime in the 
  future we'll migrate to using the VS team's automation technologies.
  
  Manual testing will only occur for major IP releases and the 
  instructions will be documented on the internal IP website under the 
  "Release Process" wiki.
  
  
  -------------------------------------------------------------------------------
  REGRESSION TESTS
  Regression tests will be added to existing test modules under the "debugging" 
  package before any CodePlex or internal bug dealing with debugging is closed.
  As such, the majority of these regressions will be automated and run on every
  developer or test checkin.  Developers should preferably add a regression test
  for every bug fix, and Test will verify the sufficiency of the test, extending
  it if necessary.
  
  -------------------------------------------------------------------------------
  BUG BASHES
  While considered to be extremely useful, we have no current plans for internal 
  bug bashes on this feature.  Quite simply put, we need more IronPython headcount 
  and/or interest from the rest of Visual Studio Languages to accomplish this.
  
  -------------------------------------------------------------------------------
  BUG REPORTING
  All bug reports on this feature which do not include information consided to be
  Microsoft confidential are to be filed at 
  http://ironpython.codeplex.com/WorkItem/AdvancedList.aspx under the "Debugging"
  component using the bug template found at 
  http://ironpython.codeplex.com/Wiki/View.aspx?title=IronPython%20Bug%20Template.
  Bugs will be triaged on a weekly basis by the entire IronPython Team.
  
  -------------------------------------------------------------------------------
  PLAN CONTINGENCIES
  Without at least one additional SDET or other test tasks getting dropped, only 
  phases I and II of this plan are implementable prior to the release of 
  IP 2.6 RTM.
  
  -------------------------------------------------------------------------------
  EXTERNAL DEPENDENCIES
  We depend on the DLR for their implementation of Microsoft.Scripting.Debugging.dll 
  which forms the basis of our support of the CPython pdb module.  This plan 
  assumes that this DLL has been adequately tested.
  
  There are no teams/projects we know about which are taking a dependency on 
  IP's debugging functionality.
  
  -------------------------------------------------------------------------------
  HEADCOUNT REQUIREMENTS
  At least two FTE IronPython SDETs will be needed to implement this test plan 
  fully: one to handle general, day-to-day IP test operations
  (i.e., passes/investigations/issues/regressions, etc), and another to work exclusively on the 
  debugging feature.  A loose approximation is that the SDET working full-time
  on debugging would need anywhere from one to three months to completely 
  implement all aspects of this plan.  This estimate varies considerably 
  depending upon how much emphasis is to be given to pdb versus mdbg versus the 
  VS IDE.  Having a second full-time SDET on the debugging feature would have a
  not positive effect as the division of labor between testing command-line IP
  debugging support and VS IDE support is quite clear.
  
  -------------------------------------------------------------------------------
  PRODUCT SUPPORT
  IP is not a supported product in the conventional Microsoft sense. 
  Support for IP is freely given directly by the IP Team via the 
  IronPython Mailing List.
  
  -------------------------------------------------------------------------------
  DROP PROCEDURES
  All developer checkins must go through the SNAP checkin system => IronPython
  sources are always in a "good" state.  Test will build IP from sources
  using developer instructions.
  
  -------------------------------------------------------------------------------
  RELEASE PROCEDURES
  See the IP "Release Process" wiki on the internal IP website.
  
  -------------------------------------------------------------------------------
  ALIAS/NEWSGROUPS AND COMMUNICATION CHANNELS
  Major changes around IP debugging support should be announced on the 
  IronPython Mailing List, users@lists.ironpython.com.
  
  -------------------------------------------------------------------------------
  REGULAR MEETINGS
  - FEATURE TEAM MEETINGS
    There is no feature team for debugging.
  - PROJECT TEST TEAM MEETINGS
    IronPython weekly team meeting. Day and time are subject to change.
    We'll triage debugging work items during this meeting and discuss IP 
    debugging issues as necessary.
  - FEATURE TEAM TEST MEETINGS
    There is no feature test team for debugging.
  
  -------------------------------------------------------------------------------
  DECISION MAKING PROCEDURES
  Decisions on this feature will be driven by the following criteria:
  - Microsoft business needs
  - IronPython Community feature requests
  - Previous decisions made by CPython. E.g., the pdb module
  - Existing debugging support in Microsoft products such as VS
  In the event a conscensus cannot be reached by the team on something, Jim
  Hugunin's input should be sought.
  
  Miscellaneous Procedures:
  - the gopackage tool is responsible for validating all source pushes to 
    CodePlex
  - at least two IP team members need to sanity check public builds
    before they can be released
  - full test pass signoff on this feature will be needed for every public, signed
    release
  - bug triages will be performed with all three software disciplines in attendance
  - development design should be reviewed both by the PM and Test disciplines
  
  -------------------------------------------------------------------------------
  NOTES
  
  MDBG TEST CASE WRITING TIPS
  - use different language structures as the last line in the Python file
  - when only one line lives inside a block of code, but the code should not be
    stepped through
  - correct step-through: no extra step-throughs or lack thereof
  - 
  '''