Manipulating a Python file from C#
Question: I'm working on some tools for a game I'm making. The tools serve as a front
end to making editing game files easier. Several of the files are python
scripting files. For instance, I have an Items.py file that contains the
following (minimalized for example)

    
    
    from ItemModule import *
    
    import copy
    
    class ScriptedItem(Item):
        def __init__(self, name, description, itemtypes, primarytype, flags, usability, value, throwpower):
            Item.__init__(self, name, description, itemtypes, primarytype, flags, usability, value, throwpower, Item.GetNextItemID())
        def Clone(self):
            return copy.deepcopy(self)
    
    ItemLibrary.AddItem(ScriptedItem("Abounding Crystal", "A colourful crystal composed of many smaller crystals. It gives off a warm glow.", ItemType.SynthesisMaterial, ItemType.SynthesisMaterial, 0, ItemUsage.Unusable, 0, 50))
    

As I Mentioned, I want to provide a front end for editing this file without
requring an editor to know python/edit the file directly. My editor needs to
be able to:

  1. Find and list all the class types (in this example, it'd be only Scripted Item)
  2. Find and list all created items (in this case there'd only be one, Abounding Crystal). I'd need to find the type (in this caseScriptedItem) and all the parameter values
  3. Allow editing of parameters and the creation/removal of items.

To do this, I started writing my own parser, looking for the class keyword and
when these recorded classes are use to construct objects. This worked for
simple data, but when I started using classes with complex constructors
(lists, maps, etc.) it became increasing difficult to correctly parse.

After searching around, I found IronPython made it easy to parse python files,
so that's what I went about doing. Once I built the Abstract Syntax Tree I
used PythonWalkers to identify and find all the information I need. This works
perfectly for reading in data, but I don't see an easy way to push updated
data into the Python file. As far as I can tell, there's no way to change the
values in the AST and much less so to convert the AST back into a script file.
If I'm wrong, I'd love for someone to tell me how I could do this. What I'd
need to do now is search through the file until I find the correctly line,
then try to push the data into the constructor, ensuring correct ordering.

Is there some obvious solution I'm not seeing? Should I just keeping working
on my parser and make it support more complex data types? I really though I
had it with the IronPython parser, but I didn't think about how tricky it'd be
to push modified data back into the file.

Any suggestions would be appreciated


Answer: You want a source-to-source [program transformation
tool](http://en.wikipedia.org/wiki/Program_transformation).

Such a tool parses a language to an internal data structure (invariably an
AST), allows you to modify the AST, and then can regenerate source text from
the modified AST without changing essentially anything about the source except
where the AST changes were made.

Such a program transformation tool has to parse text to ASTs, and "anti-parse"
(called "Prettyprint") ASTs to text. If IronPython has a prettyprinter, that's
what you need. If it doesn't, you can build one with some (maybe a lot) of
effort; as you've observed, this isn't as easy as one might think. See my
answer [Compiling an AST back to source
code](http://stackoverflow.com/questions/5832412/compiling-an-ast-back-to-
source-code/5834775#5834775)

If that doesn't work, our [DMS Software Reengineering
Toolkit](http://www.semanticdesigns.com/Products/DMS/DMSToolkit.html) with its
[Python front
end](http://www.semanticdesigns.com/Products/FrontEnds/PythonFrontEnd.html)
might do the trick. It has all the above properties.

