   (new topic again) Problems with Pretransformed Vertices [Solved]
  Hi there!I´ve come across an issue that surprised me. I´ve written a little engine-like thing that uses a deferred renderer.I do a Z-only pass, after that I fill 3 render targets with the data for diffuse color, depth and normal+specular (in one RT, specular as alpha value of normal). These 3 textures are then used in the final pass (for now) to calculate the lighting.Additionally I display wireframe spheres at the locations of the lights that are active.So, all this works fine on nVidia cards (GeForce 6800GT and 7800GT tested), but results in a black screen on every ATI card I had the chance to test (Radeon 9700, Radeon 9800 Pro, X800 GTO), except for the wireframe spheres.I will dig deeper into the problem over the next days, but I wanted to ask if somebody had some general things I should consider when writing HLSL shader in shader model 2.0 for those things. My assumption is that the shader that fills the render targets in the second pass doesn´t work as expected, as the guys that tested it for me on the ATI cards all reported they saw the wireframe spheres and nothing else, and the wireframe spheres disappeared behind some pipes I have in my scene like they should, so I assume the Z-only pass works fine.But even the shader for the ambient lighting returns a black screen on ATI cards, and that is why I assume the geometry pass behaves erratically.To sum it up I think:- the Z-only pass works fine (I would be surprised if it didn´t, close to nothing done here)- the geometry pass that fills the RTs behaves unexpectedly, therefore the RTs or at least the diffuse color one contains nothing.- the lighting stage should work fine I hope, as it´s just a bunch of shaders using 3 textures, nothing too weird done here, but I´ll see about that.I´ll post the source of the shader I use for filling the RTs:float4x4 mat_WV : WORLDVIEW;float4x4 mat_WVP : WORLDVIEWPROJECTION;texture2D ColorMap : COLORTEXTURE;sampler color_sampler = sampler_state{    Texture = (ColorMap);};texture2D NormalMap : NORMALTEXTURE;sampler normal_sampler = sampler_state{    Texture = (NormalMap);};texture2D SpecularMap : SPECULARTEXTURE;sampler spec_sampler = sampler_state{    Texture = (SpecularMap);};float4 floatToColor(float f){    float4 color;    f*=256;    color.r = floor(f);    f=(f-color.r)*256;    color.g = floor(f);    color.b = f-color.g;    color.rg *= 0.00390625;    color.a = 1.0;    return color;}struct VERTEX_IN{    float4 pos: POSITION;    float3 normal  : NORMAL;    float3 tangent : TANGENT;    float2 texcoord: TEXCOORD;};struct V2P{    float4 pos: POSITION;    float  depth   : TEXCOORD0;   //depth in proj-space    float2 texcoord: TEXCOORD1;   //uv-coordinates    float3 normal  : TEXCOORD2;   //normal in view-space    float3 tangent : TEXCOORD3;   //tangent in view-space};V2P simple_vs(in VERTEX_IN v){    V2P o;    o.pos = mul(v.pos, mat_WVP);    o.depth = o.pos.z / o.pos.w;    o.normal = mul(float4(v.normal,0),mat_WV);    o.tangent = mul(float4(v.tangent,0),mat_WV);    o.texcoord = v.texcoord;    return o;};float4 simple_ps(in V2P v,  out float4 viewNormal : COLOR1,    out float4 depth : COLOR2) : COLOR0{    float4 n = tex2D(normal_sampler, v.texcoord)*2.0 -1.0;    v.normal = normalize(v.normal);    v.tangent = normalize(v.tangent);    float3 bin = cross(v.tangent, v.normal);    viewNormal = float4(n.r*v.tangent + n.g*bin + n.b*v.normal, 1.0) * 0.5 + 0.5;// if no normalMap was set, use the interpolated vertex normal    if(n.r < 0.01 && n.g < 0.01 && n.b < 0.01)    {   viewNormal =float4(v.normal, 1.0);    }    viewNormal.a = tex2D(spec_sampler, v.texcoord).r;   depth = floatToColor(v.depth);// alpha value of normalMap contains ambient occlusion factor// as calculated by D3DXComputeNormalMap(...)    return float4(tex2D(color_sampler, v.texcoord).rgb, n.a);};technique tec0{    pass p0    {   VertexShader = compile vs_2_0 simple_vs();   PixelShader = compile ps_2_0 simple_ps();    }}Perhaps someone sees something I´m not supposed to do and ATI cards would refuse to do. In general I would really be interested if there´s some kind of guide to differences between ATI and nVidia cards when it comes to shaders, some list of commands that behave differently or something like that.Thx for reading and for your help![Edited by - matches81 on June 22, 2006 12:53:40 AM]
  I know null textures are sampled as black on ATI cards and white on NVidia (or was it the other way arround?). At any rate, make sure you're not assuming that.Are you using MRTs for the rendering to the render targets? That could start up a whole bunch of problems :). Looking at your shader, it looks like you are. Make sure the drivers on the test machines are not forcing FSAA. That doesn't play nice with MRTs, and could cause the whole scene not to render properly.That's all I can think of right now :).;  Thx for the tip with forced FSAA, I´ll ask my friends to check on that.Yes, I am using MRT to fill the render targets as my batching currently completely sucks (I´ll try to find something useful for that problem soon) and I like the idea of just having to render one pass to fill all the RTs.I´m not assuming anything about the value of tex2D for NULL textures, as I had already heard those were different for ATI and nVidia.Thx anyway![edit]okay, asked one of my friends with an ATI card and he is not forcing FSAA on the driver, he double-checked that for me. Anyway, was a good guess, too bad it doesn´t seem to be that "simple".[/edit]   ;  Okay, I nailed the problem down to setting the render targets.If I don´t change the render target in anyway (i.e. leave it to the backbuffer) the ATI card (Radeon 9800 Pro in that case) displays what it should.As soon as I do something like:   HRESULT hr = this->m_pDirect3DDevice->SetRenderTarget(0, this->m_pRTColorSurface);if(FAILED(hr)){MessageBox(NULL, "SetRenderTarget for ColorTexture failed!", "RenderGeometryPass error!", MB_OK);return -12;}//render stuffhr = this->m_pDirect3DDevice->SetRenderTarget(0, this->m_pRTBackBuffer);if(FAILED(hr)){MessageBox(NULL, "Reverting to backbuffer failed!", "Renderer::RenderGeometryPass() error", MB_OK);return -30;}and render a fullscreen quad using m_pRTColor (the LPDIRECT3DTEXTURE9 corresponding to the surface used above) the ATI card doesn´t display a thing.   ;  What does the reference rasterizer do? Whenever IHV's differ it pays to check it against the RefRast [smile]Are you sure the references for the back-buffers/targets are still valid? Are you getting any warnings from the debug runtimes (e.g. (silently) failed calls)?Can you get a PIX call-stream capture for both cards and step through them to see what data is actually in each resource, and whether they are all still valid?When using MRT's, have you made sure you're working to the restrictions listed in the "Multiple Render Targets" section of the SDK documentation. There are several caps-bits you should check and you might have to do differently on different hardware.Finally... have you got any other MRT code to cross-reference against? I dont think any of the SDK samples use MRT, but I'm pretty sure I've come across a few deferred shading samples in the past. Could be worth seeing what they're doing.hthJack   ;  Thx for the tip with the reference rasterizer. Should have remembered that thing by myself. The reference rasterizer behaves just like the ATI cards do and renders a black screen. However, currently I´m not using MRT at the moment for testing purposes, just a single render target.The debug runtime doesn´t spit out any warnings during the program running, only at startup time, where I receive some redundant sampler- and renderstates originating from the code that loads shaders and meshes.No warnings when creating or setting the render targets.I will now check whether the depth stencil format of D24S8 matches with the format of currently only render target format of A8R8G8B8.   ;  Okay, the depth buffer format of D24S8 seems to be fine with my render target format of A8R8G8B8 and the adapter format of X8R8G8B8.After that I captured the call stream for the reference rasterizer and for my GeForce 6800. On the reference rasterizer the RT I use stays black, while it contains the expected results when using D3DDEVTYPE_HAL...On both devices the status of the render targets is "Alive".Everything regarding the render target seems to be in order, except that they are empty when using the refrast.However I notices something odd:_All_ textures I use for my meshes have NULL surfaces (all surfaces, including top-level surface) are stated to be NULL (0x000...) in PIX. The only exceptions to this are the normal maps, which are generated using D3DXComputeNormalMaps when loading the corresponding height-maps.This is the case for the reference rasterizer and the HAL device, so it doesn´t seem to be related to my problem.Any help appreciated.   ;  Okay, I changed my shader so that it would just output a constant color different than the color specified for D3DDevice->Clear(...) just to make sure the shader definitely is not the problem. Furthermore I´m only using one render target now.Still, the render target I use stays black on the refrast and on ATI cards. I´ll sum up what I do with the render target, including its creation:this->m_pDirect3DDevice->CreateTexture(conf->GetResWidth(), conf->GetResHeight(), 1, D3DUSAGE_RENDERTARGET, D3DFMT_A8R8G8B8, D3DPOOL_DEFAULT, &this->m_pRTColor, NULL);this->m_pRTColor->GetSurfaceLevel(0, &this->m_pRTColorSurface);this->m_pDirect3DDevice->GetRenderTarget(0, &this->m_pRTBackBuffer);this->m_pDirect3DDevice->BeginScene();this->m_pDirect3DDevice->SetRenderTarget(0, this->m_pRTColorSurface);//clear only render-target, Z-buffer was filled in a Z-only passthis->m_pDirect3DDevice->Clear(0, NULL, D3DCLEAR_TARGET, D3DCOLOR_XRGB(255,255,255), 1.f, 0);  //rendering stuff using a single pass, single render-target effect herethis->m_pDirect3DDevice->SetRenderTarget(0, this->m_pRTBackBuffer);this->m_pDirect3DDevice->EndScene();Hopefully someone notices something odd about my code or has an idea what I should be looking for when using PIX, as I am totally new to that program.Thx for reading (and helping) again   ;  *push* sry I have to do that, but I really do need help with this.If I´m missing some info you think might help with the problem, please ask.   ;  Well, here are a couple ideas that might help you out, or maybe remind you of something else that might be the problem:Make sure any call to DrawIndexedPrimitive() supplies the correct parameters in the minVert and maxVert (forget the exact names), fields. IIRC, both the RefRast and ATI cards have a test to make sure they are correct, while NVidia completely ignores these parameters. It might be the other way around, but I think ATI have an "assert" for each vertex, so if these are incorrect, it might not draw. Since the RefRast is a software renderer, it likely actually uses these values to pretransform the vertices, so it really does need the correct values.Check for assumptions of default renderstates. One that really springs to mind is ZFunc. It might be set to LessEquals by default on NVidia cards, and to Less on ATI. This would completely kill any rendering on the latter. It isn't too likely, but worth a check.Try disabling anything that might stop rendering from reaching the target. Turn off z-buffering, turn off backface culling, turn off the StencilEnable. Anything that might cause something that's rendered to not get to the screen.[continued:]I'm not sure if you said this, but have you tried your exact code without the "SetRenderTarget" calls? I mean exactly the same. Does it render anything?Make sure all your vertex strides are correct, and recheck the vertex declaration. I've had cards recover from an incorrect vertex declaration by ignoring some data, but that can't be expected to work exactly the same on all cards.Try using the FFP to render to the render target. Just getting a pixel on the screen would indicate something might be wrong with the actual shaders.Okay, that's all I can think of right now :) If none of this helps, maybe you should try the DIRECTXDEV mailing list.Hope this helps.[Edited by - sirob on June 21, 2006 1:17:27 AM]