1. At a computer system including one or more processors and system memory, the computer system also including a compiler, a method for recording debug information for an application, the debugging information mapping between a unified address space and a plurality of target address spaces, including at least a first address space and a second different address space, the method comprising: an act of accessing source code for the application, the source code declaring a plurality of variables denoting storage locations, the source code utilizing unified address space pointers to provide a unified address space abstraction, the unified address space pointers referencing storage locations in the unified address space, the source code including statements and expressions for assigning the address of one or more of the plurality of variables to one or more corresponding unified address space pointers, the source code including statements and expressions for dereferencing the one or more corresponding unified address space pointers; an act of compiling the source code into lower level code for execution on a target system, compilation of the source code including: an act of allocating at least one of the plurality of variables to the first address space and allocating at least one other of the plurality of variables to the second different address space such that the first address space is referenced through the at least one variable allocated to the first address space and the second different address space is referenced through the at least one other variable allocated to the second different address space; an act of representing each unified address space pointer using a two component pointer including a tag component and an offset component, the tag component configured to store an identifier value representing a target address space, the offset component configured to store an offset value indicating a relative offset within the represented target address space; and an act of accumulating debug information for the application that can be used to translate between the unified address space and the plurality of target address spaces, including: for each unified address space pointer variable in the source code, an act of recording a designated storage location for the tag component and offset component of the corresponding two component pointer; an act of generating a table, the table for storing mapping entries, each mapping entry for mapping an identifier value to a storage location for a target address space; and for each assignment of a program variable address to a unified address space pointer in the source code, an act of inserting a mapping entry into the table, the mapping entry mapping an identifier value for the program variable to the storage location of the target address space for the program variable, the identifier value also assigned to the tag component of the two component pointer corresponding to the unified address pointer; and an act of storing the accumulated debug information for use in a debugging session.