   backface culling in vertex shader
  hi,  i have to manually perform backface culling in a shader.i have tried what most sources have suggested:if the dot product of the current vertex normal and the view vector is negative, the face is a backface.unfortunately i can't make this work at all. maybe you can see the error in my shader at a first glimpse:I'm just trying to make the backfaces blue for now, but that already gives really wrong results:float4x4 matViewProjection;float4x4 matWorld;float4x4 matViewInverse;struct VS_INPUT {   float4 Position : POSITION0;   float4 Normal   : NORMAL;};struct VS_OUTPUT {   float4 Position : POSITION0;   float4 Color : COLOR0;};VS_OUTPUT vs_main( VS_INPUT Input ){   VS_OUTPUT Output; float3 posWorld = Input.Position; float3 camPos = matViewInverse[3].xyz; float3 vAt = normalize(camPos - posWorld); float4 norm = mul(Input.Normal, matWorld); Output.Position = mul( Input.Position, matViewProjection );   float z = dot(norm, vAt);float4 col = 1; // make backfaces blue if(z < 0) { col = float4(0, 0, 1, 1); } Output.Color = col;   return( Output );  }help would be very appreciated   
  Define "wrong results", show us a screenshot. You can't perform face-level operations in a vertex shader, since it operates on a per-vertex basis. One invocation of the vertex shader cannot see the other vertices of a triangle - it doesn't even know whether the results are used for a triangle or some other geometry.If one vertex of a triangle fits your definition of "back-facedness" but the others do not, the blue will be mixed with the other vertex colors during the interpolation that the rasterizer does. This is very common on the view-space edges of most real-world models. It is possible to manually cull whole triangles by using a geometry shader, but it is not available for D3D9.Is there a reason why can't you use the built-in backface culling? It is very broadly supported. Also, you can switch between culling front -or backfacing triangles to draw both sides. And modern shader profiles (SM3 and up) give you the handedness of the current triangle in the pixel shader too.   ;  I have to do it myself because i have to draw the model in UV space but cull the faces like they normally would have been. It's a technique used in virtual texturing. Here's a better explanation:Quote:[...] Backface culling requires a special treatment: GPUs automatically handle back–facing triangles, but only by taking into account vertex position in rendering space. In our case – since we are rendering in texture space – the vertices positions are not their screen space position but their texture space coordinates: Automatic back face culling according to the screen space cannot be performed. Our approach is to use the far clipping plane to achieve back face culling. Recall we are rendering in a 2D texture space: The near and far clipping plane were not taking into account until now. We compute the z coordinate of the vertices during rendering in the TLM level as hpos.z = dot(nrm, vv) − 1.0 where hpos.z is the z coordinate of the resulting vertex (as computed in the TLM vertex program), nrm the normal associated to the vertex, and vv the normalized view vector. The far plane is positionned at z = −1. Vertices are thus culled if dot(nrm, vv) < 0.0 which corresponds to the operation required by backface culling. Note that if per-vertex normals are used, the triangle will be split at the iso-value dot(nrm, vv) = 0.0. (http://www-evasion.imag.fr/Publications/2004/LDN04/RR-5210.pdf)   ;  YO! Forget this! I didn't add the normals in Rendermonkey, and now that i did, it works good enough. I really spent hours on this, but i'm also new to Rendermonkey.;  Note the blending in the spoiler and the roof. That is the artifact caused by Gouraud interpolation of the vertex colors of the triangles. If you can live it, good for you, but this is not exactly the same as true backface culling.   ;  Definately right, it's very unprecise and seems to be absolutely useless for my purpose. It just culls away too much.Well it's what the article suggested so i had to give it a try, but i guess they were using per face normals then.Quote:Original post by Nik02And modern shader profiles (SM3 and up) give you the handedness of the current triangle in the pixel shader too.How is this done? Should i use the interpolated normals in the pixel shader for the backface test?   ;  No, you just disable the culling. Then, you can use the VFACE semantic in your pixel shader input parameters - the parameter will be either negative (back facing) or positive (front facing). The parameter should be of type float.Note that this does not skip the processing of the backfacing pixels unlike true backface culling, thus does not introduce a performance increase.Note that the paper is written in the first half of 2004; it was around April 2004 that the first SM3 cards became available.   ;  Also, Vista and 7 already have virtualized GPU memory built in. It can work in per-mip level.   ;  Thanks a lot Nik02. I will try it.It will not increase shading performance, but it will prevent texture tiles being loaded from hard disc for backfaces :-) so it would still be a performance win in this case.The technique is used for Virtual Texturing:You draw the scene in UV space and compute a LOD for each visible page of your atlas. Then you do CPU readback and load the tiles. So you try to draw only whats visible.   ;  This means virtual texturing ("megatexture") is obsolete? I didn't know that. Glad you mention that, because i've put months in this project already and getting grey hairs over it :)   