How to prevent the decimate modifier from working on vertices outside the vertex group (with python)?
Old answer below!

My solution was terrible slow. So I got the blender sources and changed the modifier. It was fairly easy!

I added following code on line 311 (https://git.blender.org/gitweb/gitweb.cgi/blender.git/blob/master:/source/blender/bmesh/tools/bmesh_decimate_collapse.c#l311)

// Only work on vertex groups
// vweights is false when no vertex groups are used
    if (vweights) {
        float weight_v1 = roundf(vweights[BM_elem_index_get(e-&gt;v1)]);
        float weight_v2 = roundf(vweights[BM_elem_index_get(e-&gt;v2)]);

        if (weight_v1 != 0.0f &amp;&amp; weight_v2 != 0.0f)
        {
            eheap_table[BM_elem_index_get(e)] = BLI_heap_insert(eheap, cost, e);
            return;
        }
        else
            return;
    }


As you can see I used the weight to find out if the vertex is really in the group. I need to do that because in deform.c (https://git.blender.org/gitweb/gitweb.cgi/blender.git/blob/HEAD:/source/blender/blenkernel/intern/deform.c#l742) the weight is set to 0.0 even if the vertex isn't inside the group.
If you need this you need to add vertices to a group and set any value between 0.01 and 1 as vertex weight.

Old answer

I wrote a simple (stupid) decimate algorithm which only works on a vertex group or the whole mesh. It is only collapsing edges and the angles are totally ignored. It is not trying to preserve the shape really well.

Here is the code, it is not optimized (I am pretty new to python) and I know I could optimze it here and there (all the bpy.ops api calls could have faster api methods) but for now I am happy. Nevertheless: Here is the code!

If you have an idea how the algorithm would be faster, spit it out. Thanks!

# how_often: how often should a edge be collapsed
# group_name: if it is None it will work on the whole mesh or on a given group.
def decimate_mesh(self, how_often=0, group_name=None):
    import bpy
    ob = bpy.context.active_object
    mesh = ob.data

    # go to edit mode and deselect everything .. just to be sure
    bpy.ops.object.mode_set(mode = 'EDIT')
    bpy.ops.mesh.select_all(action='DESELECT')
    bpy.ops.object.mode_set(mode = 'OBJECT')
    how_often_decimate = how_often

    if group_name is None:
        # decimate all edges
        while how_often_decimate != 0:
            edge_length = dict()
            # calculate edge lengths
            for edge in mesh.edges:
                edge_length[(mesh.vertices[edge.vertices[0]].co-mesh.vertices[edge.vertices[1]].co).length] = edge
            # pick the shortest edge, collapse it and make all polygons concave
            min_key = min(edge_length.keys())
            edge_to_collapse = edge_length[min_key]
            edge_to_collapse.select = True
            bpy.ops.object.mode_set(mode = 'EDIT')
            bpy.ops.mesh.merge(type='COLLAPSE')

            bpy.ops.mesh.select_all(action='DESELECT')
            bpy.ops.mesh.select_all(action='SELECT')
            bpy.ops.mesh.vert_connect_concave()
            bpy.ops.mesh.select_all(action='DESELECT')
            bpy.ops.object.mode_set(mode = 'OBJECT')
            how_often_decimate -= 1
    else:
        # decimate edges where both vertices (of the edge) are part of the vertes group
        group_index = ob.vertex_groups[group_name].index
        while how_often_decimate != 0:
            vertices_which_are_assigned_to_group = set()
            edges_which_can_be_decimated = set()
            edge_length = dict()
            # I am unsure if I really need to do this every time. I know that the indexes could be altered by blender
            # but I need to know which vertices are inside the group. So yeah...
            for vert in mesh.vertices:
                for vert_group in vert.groups:
                    if vert_group.group == group_index:
                        vertices_which_are_assigned_to_group.add(vert.index)

            for edge in mesh.edges:
                vertex_0_index = edge.vertices[0]
                vertex_1_index = edge.vertices[1]
                if vertex_0_index in vertices_which_are_assigned_to_group and vertex_1_index in vertices_which_are_assigned_to_group:
                    edges_which_can_be_decimated.add(edge)

            if len(edges_which_can_be_decimated) == 0:
                print("No edges anymore!")
                break
            for edge in edges_which_can_be_decimated:
                edge_length[(mesh.vertices[edge.vertices[0]].co-mesh.vertices[edge.vertices[1]].co).length] = edge

            min_key = min(edge_length.keys())
            edge_to_collapse = edge_length[min_key]
            edge_to_collapse.select = True
            bpy.ops.object.mode_set(mode = 'EDIT')
            bpy.ops.mesh.merge(type='COLLAPSE')
            bpy.ops.mesh.select_all(action='DESELECT')
            bpy.ops.mesh.select_all(action='SELECT')
            bpy.ops.mesh.vert_connect_concave()
            bpy.ops.mesh.select_all(action='DESELECT')
            bpy.ops.object.mode_set(mode = 'OBJECT')
            how_often_decimate -= 1