How can I set a custom projection matrix for rendering?
For orthographic rendering Blender can't take a custom projection matrix, however you can shear an orthographic projection to get the desired results.


Use an orthographic camera, with the camera set at an angle (45 degrees on the X and Z axis).
In the render settings, set the Aspect X to sqrt(2)This will give something almost exactly what you're looking for, except that its rotated 45 degrees.
To correct the rotation, use the compositors Distort -> Transform node.




Notes:


rotating the image will loose some accuracy, so you'll want to render larger, then scale down.
in the compositor you can scale down by 1/sqrt(2) to avoid loosing the corners.
Depth/Mist won't work as you might want, however its probably possible to use a gradient and offset the depth in the compositor.