Depth Pass to Point-cloud (with AN)
It is possible to deform a mesh grid with Depth Map or Point Position Map to create the scene geometry with the help of the Animation Nodes + Extra Nodes as well as the Shader Nodes.


Method for Depth Map with AN+EN:

Render the Color Map and Depth Map (must with OpenEXR) with OpenEXR format,


We need to create two grids, one grid (Mesh Grid) is scaled according to the aspect ratio of the depth map. The second to evaluate texture from the Texture Input node (In Texture Properties, set the mapping to clip or extend), and this is a square grid (2x2) because the texture is displayed with square aspect ratio in the world origin. So, I created them a group node so that we can cache them:

Then added resolution input for extra control:

This how these grids look (green for geometry, white for texture)


Now, the Mesh Grid points scaled texture aspect ratio but we have scaled these points using the camera distance factor and depth scale (from Texture Input node), that will give correct X, Y coordinates (as shown by @WhatAMesh in his answer).

The Z coordinate will be the negative of the depth map to deform toward the camera,

Now the points of Mesh Grid are deformed correctly. Now, to align them correctly with respect to the camera, we have to transfer them according to the camera,

Then combine points (from Step 3) with the Edge Indices and Polygon Indices to get a deformed Mesh Grid. I also added the UV map to the mesh Grid, so that we can use it for the camera projection of the color map.

The Mesh Grid is perfectly matching with the actual scene geometry.

Complete Node-Tree,


For the color map, use the UV Project modifier and set the scale according to the aspect ratio. Then in shader nodes use UVs to assign the color map to the Mesh Grid,



Blend File: 

Method for Point Position Map with AN+EN:
This is very simple as compared to the Depth Map Method and faster.

To render the point position map of a scene: In Shader Nodes, the Geometry node has a position pass that contains location (X, Y, Z) info for each point/pixel of the geometry. So, use this shader-node setup, and override the material of the whole scene to get the point position pass. Then render this as an OpenEXR image,


Now, we need pixels info of point position that we can't get with the Texture Input node because these values can be negative. So, use the Expression node to get info for dimensions and pixels from the texture. Then, we create Mesh Grid but with swap X, Y divisions to get the correct order for edge indices and polygon indices for pixels or location(X, Y, Z) of points from the pixels. The pixels from the Expression node in a single list so we have to use the Slice List to get R, G, B values or X, Y, Z:


Edit - Low-Resolution Grid:
We can generate a low-resolution mesh grid by resampling the high-resolution grid with the help of KDTree nodes. The Find Nearest Point node outputs the indices with that, we will get the points/vectors of the high-resolution mesh grid (in this case vectors from pixels) that is near to the low-resolution mesh grid points. Here is the group node,

Now, we have to replace the Mesh Grid with this group node and set the cache to One Time which is important for performance,


For the color map, do Step-5 of the Depth Map Method.


Blend File: 
Blend File For Low-Resolution Grid:


Method for Point Position Map with Shader Nodes:

First, render the Point Position Map as mentioned in Step-1 of Method for Depth Map with AN+EN

Change the render engine to Cycles and feature set Experimental. Add a square plane in the world center 

Then subdivide it for few times, and add the Subdivision modifier with Adaptive Subdivision option is enabled,


In the Shader Nodes, Options &gt; Settings &gt; Displacement to Displacement Only. Then use the Point Position map (In Image Input node, set the mapping to clip or extend) as a vector displacement as shown below, (In Render View)



Blend File: