   Help me on ray picking i am getting crazy
  ray picking make me crazy. I try to do a ray picking that i move cursor on mesh it will jump out a message box say hit. My code doesn't work but i don't know why. I already deal with this problem for few weeks, but i still can't figure it out. Can anyone tell what wrong with my code?i create a function to transform the ray and do the hit detectionvoid calculate_ray_hit(LPD3DXMESH &mesh){POINT cursorpos;//LPRECT rect;//get cursor positionGetCursorPos(&cursorpos);ScreenToClient(get_hwndhandler(), &cursorpos);float dx=0.0f;float dy=0.0f;float dz=1.0f;D3DXVECTOR3 origin;D3DXVECTOR3 dir;D3DVIEWPORT9 viewport;D3DXMATRIX project,view,rview,world,rworld; // get projection matrixd3dd->GetTransform(D3DTS_PROJECTION, &project); // get viewportd3dd->GetViewport(&viewport); //calculate cursor from screen to projectiondx=(((2.0f*cursorpos.x)/viewport.Width)-1.0f)/project(1,1);dy=-(((2.0f*cursorpos.y)/viewport.Height)-1.0f)/project(2,2);   origin=D3DXVECTOR3(0.0f,0.0f,-5.0f);dir=D3DXVECTOR3(dx,dy,dz);d3dd->GetTransform(D3DTS_VIEW, &view);D3DXMatrixInverse(&rview, NULL, &view);   // transform ray in to world space   D3DXVec3TransformCoord(&origin, &origin, &rview);D3DXVec3TransformNormal(&dir, &dir, &rview);   //normalize directionD3DXVec3Normalize(&dir, &dir);DWORD fi;float u,v, dist;   //check if ray point on meshD3DXIntersect(mesh, &origin, &dir, &hit, NULL, NULL, NULL, &dist, NULL, NULL);   //if ray hit mesh then show the message boxif(hit)MessageBox(0,"hit","hit test", 0);}[Edited by - tomneo2004 on June 27, 2009 8:55:34 PM]
  From the code you posted it seems like your are transforming the vector dir with dx = 0.0,dy = 0.0 and dz = 1.0. You should be transforming the calculated dx,dy and dz = 1.0. As such, the line dir=D3DXVECTOR3(dx,dy,dz), should go after you calculate dx and dy. This vector then get transformed by the inverse view matrix to bring it into world space. You do not have to transform the direction vector into object space. All you have to do is normalize the vector (since its a direction vector) and the origin of this vector will always be the camera position (the last row of the inverse view matrix). I do my own intersection test so I don't remember the parameters to D3DXIntersect but I assume it all it needs is the ray direction and origin. Try the above with and without ScreenToClient I don't remember right now but I had issues with my code when I use/didn't use ScreenToClient. Good luck.;Quote:Original post by cgrantFrom the code you posted it seems like your are transforming the vector dir with dx = 0.0,dy = 0.0 and dz = 1.0. You should be transforming the calculated dx,dy and dz = 1.0. As such, the line dir=D3DXVECTOR3(dx,dy,dz), should go after you calculate dx and dy. This vector then get transformed by the inverse view matrix to bring it into world space. You do not have to transform the direction vector into object space. All you have to do is normalize the vector (since its a direction vector) and the origin of this vector will always be the camera position (the last row of the inverse view matrix). I do my own intersection test so I don't remember the parameters to D3DXIntersect but I assume it all it needs is the ray direction and origin. Try the above with and without ScreenToClient I don't remember right now but I had issues with my code when I use/didn't use ScreenToClient. Good luck.thaks but i do what you say but hit detection is always true whether my cursor is on the mesh or not and explain why i don't need to transform the ray into object's space?I draw a line to check but i got this picture. the ray is come from inside of mesh body and follow the mesh, but why it become like thisBuy the way, i use right system (RH) for my camera and projection[Edited by - tomneo2004 on June 27, 2009 9:59:18 PM];  Why not use one of the unproject functions?Simple version using xnamath:const XMVECTOR RayOrigin = mainCamera.CamPos;POINT mouse;GetCursorPos( &mouse );XMVECTOR CursorScreenSpace = XMVectorSet( mouse.x, mouse.y, 0.0f, 0.0f );XMVECTOR CursorObjectSpace = XMVector3Unproject( CursorScreenSpace, 0, 0, 1280, 1024, 0.0f, 1.0f, mainCamera.Projection, mainCamera.View, XMMatrixIdentity() );XMVECTOR RayDir = CursorObjectSpace - RayOrigin;RayDir = XMVector3Normalize( RayDir );// Then the testD3DXVECTOR3 vEye, vDir;XMStoreFloat3( (XMFLOAT3*)&vEye, RayOrigin );XMStoreFloat3( (XMFLOAT3*)&vDir, RayDir );AABB temp( Comp::BB[0] );temp.TransformAndResize( &g_World );BOOL hit = D3DXBoxBoundProbe( &temp.m_Min, &temp.m_Max, &vEye, &vDir );// Tested it, worksEdit:Used LH projection, view matrix is hand-built.   ;Quote:Original post by darkelf2k5Why not use one of the unproject functions?Simple version using xnamath:*** Source Snippet Removed ***Edit:Used LH projection, view matrix is hand-built.I did the same thing like ,but it result as same as what i do now.i still got the problem on ray picking, so i post my project. If some one can look my project and help me.My project can download here.The result i want is when i move my cursor on the mesh it should jump out a message say hit   ;  Works fine for me.D3DXVECTOR3 RayOrigin( view._41, view._42, view._43 );D3DXVECTOR3 CursorOS;D3DXVECTOR3 CursorSS( cursorpos.x, cursorpos.y, 0.0f );D3DXVec3Unproject( &CursorOS, &CursorSS, &viewport, &project, &view, NULL );D3DXVECTOR3 RayDir = CursorOS - RayOrigin;D3DXVec3Normalize( &RayDir, &RayDir );...D3DXIntersect(mesh, &RayOrigin, &RayDir, &hit, NULL, NULL, NULL, &dist, NULL, NULL);In renderpoint, use and identity transform instead of making the scene spin around. That way you can see it better.If you want to pick the spinning model, do the hit test after the spin.   ;Quote:Original post by darkelf2k5Works fine for me.D3DXVECTOR3 RayOrigin( view._41, view._42, view._43 );D3DXVECTOR3 CursorOS;D3DXVECTOR3 CursorSS( cursorpos.x, cursorpos.y, 0.0f );D3DXVec3Unproject( &CursorOS, &CursorSS, &viewport, &project, &view, NULL );D3DXVECTOR3 RayDir = CursorOS - RayOrigin;D3DXVec3Normalize( &RayDir, &RayDir );...D3DXIntersect(mesh, &RayOrigin, &RayDir, &hit, NULL, NULL, NULL, &dist, NULL, NULL);In renderpoint, use and identity transform instead of making the scene spin around. That way you can see it better.If you want to pick the spinning model, do the hit test after the spin.what do you mean "identity transform"? Do you know is there any thing wrong in my code? Is it possible because i set world transform to rotate model? Can you post complete code of ray transform?   ;  I just added what I posted before to your pick function. And took out the part where you spin the mesh around. If you want to be able to pick the spinning mesh, you have to do the pick AFTER all the transforms are applied. Picking and then making it spin will just look like it does not work. Put the pick() call after you set the final world transform for your mesh.   ;  I figure it out and now it is working well.thx   