Offset based on Vertex Colors
Unfortunately blender doesn't provide many options for doing much with vertex colours.

If you can convert the colours to greyscale texture, with values in a range of 0 to 1 (with your current green value as 0.5) instead, you can use this to control a displace modifier on the mesh. Set the midpoint to 0.5 and the strength to the difference between the maximum increase and maximum decrease that you want. This will require UV unwrapping your mesh. The conversion to texture could be achieved by creating the right shader and baking surface colour.

Alternatively, you could create 2 vertex groups for the positive and negative displacement and use these to control two displacement modifiers without textures, one using a positive strength, the other a negative. No UV unwrap is required here. A script would be required to convert the vertex colours to groups.