   [Help] help me with ray picking problem
  I load a mesh and draw it up, also rotate it. Now i create a function that can create a ray to detect if cursor on the mesh every time, but it is not working(if it is working a message will show up) what is wrong with my code? can anyone help me?Buy the way, i refer to this websit click_here my project herebelow is my codevoid calculateray(LPD3DXMESH &mesh){POINT cursorpos;//get cursor posGetCursorPos(&cursorpos);   //get cursor position within windowScreenToClient(get_hwndhandler(), &cursorpos);float dx=0.0f;float dy=0.0f;D3DXVECTOR3 origin,objorigin;D3DXVECTOR3 dir,objdir;D3DXMATRIX project,view,rview,world,rworld;   // get the matrix of projectiond3dd->GetTransform(D3DTS_PROJECTION, &project); // transform cursor position to projectiondx=(((2.0f*cursorpos.x)/800)-1.0f)/project(1,1);dy=-(((2.0f*cursorpos.y)/600)-1.0f)/project(2,2);  //in dir's z axis i set it to 1.0forigin=D3DXVECTOR3(0.0f,0.0f,0.0f);dir=D3DXVECTOR3(dx,dy,1.0f); //get inverse matrix of viewd3dd->GetTransform(D3DTS_VIEW, &view);D3DXMatrixInverse(&rview, NULL, &view);   //after i get inverse matrix of view, i apply to dir and origin vector.   //mean of this is to translate dir and origin to world space dir.x=dir.x*rview._11 + dir.y*rview._21 + dir.z*rview._31;dir.y=dir.y*rview._12 + dir.y*rview._22 + dir.z*rview._32;dir.z=dir.z*rview._13 + dir.y*rview._23 + dir.z*rview._33;origin.x=rview._41;origin.y=rview._42;origin.z=rview._43;   //get inverse matrix of world d3dd->GetTransform(D3DTS_WORLD, &world);D3DXMatrixInverse(&rworld, NULL, &world);   //after i get inverse matrix of world, i translate origin and dir   // to model's space, also normalize dir   //buy the way, objorigin and objdir are new vectors, i use those   // to store the origin and dir in model's space D3DXVec3TransformCoord(&objorigin, &origin, &rworld);D3DXVec3TransformNormal(&objdir, &dir, &rworld);D3DXVec3Normalize(&objdir, &objdir);   //calculate intersection with mesh   BOOL hit;DWORD fi;float u,v, dist;D3DXIntersect(mesh, &origin, &dir, &hit, &fi, &u, &v, &dist, NULL, NULL);   // if hit is true than show message boxif(hit)MessageBox(0,"hit","hit test", 0);}here is the function setup and load the meshvoid setup(){//get the d3d device d3dd=getd3ddevice();HRESULT hr=0;   // load meshhr=D3DXLoadMeshFromX("tiger.x", D3DXMESH_VB_MANAGED, d3dd, &adjbuffer, &materialbuffer, NULL, &numofmaterial, &mesh);if(hr!=S_OK){MessageBox(0, "load mesh fail", "waring", 0);meshload=false;}if(meshload==true){ material=(D3DXMATERIAL*)materialbuffer->GetBufferPointer(); meshmaterial=new D3DMATERIAL9[numofmaterial]; texture=new LPDIRECT3DTEXTURE9[numofmaterial]; for(DWORD i=0; i < numofmaterial; i++) {meshmaterial.Ambient=meshmaterial.Diffuse;meshmaterial=material.MatD3D;D3DXCreateTextureFromFile(d3dd, material.pTextureFilename, &texture);  } adjbuffer->Release(); materialbuffer->Release();}//position cameraD3DXVECTOR3 eye(0.0f,0.0f,-5.0f);D3DXVECTOR3 lookat(0.0f,0.0f,0.0f);D3DXVECTOR3 up(0.0f,1.0f,0.0f);D3DXMATRIX v;D3DXMatrixLookAtRH(&v, &eye, &lookat, &up);d3dd->SetTransform(D3DTS_VIEW, &v);//projectionD3DXMATRIX proj;D3DXMatrixPerspectiveFovRH(&proj, -(float)D3DX_PI*0.5f,  (float)800/(float)600, 1.0f, 1000.0f);d3dd->SetTransform(D3DTS_PROJECTION, &proj);//setup rendering stated3dd->SetRenderState(D3DRS_LIGHTING,false);d3dd->SetRenderState(D3DRS_ZENABLE,true);d3dd->SetRenderState(D3DRS_CULLMODE, D3DCULL_NONE);}Buy the way, what are these mean:meshmaterial.Ambient=meshmaterial.Diffuse;meshmaterial=material.MatD3D;[Edited by - tomneo2004 on June 26, 2009 4:55:41 AM]
  is there anyone can help me with this?   ;  Try this code :void cCamera::GetRay( cRay & r, const cVector3d & v, const cMatrix4x4 & pos ){    cVector3d vec = v;    cVector3d vec1 = v;    vec1.z = 1;    D3DXVec3Unproject( vec, vec, &m_vp, m_projection, m_view, pos );    D3DXVec3Unproject( vec1, vec1, &m_vp, m_projection, m_view, pos );r.m_pos = vec;    r.m_length = vec1 - vec;}Replace cVector3d with D3DXVECTOR and set pos in identity matrixm_vp - it's viewport structurem_projection - it's projection matrixm_view - it's view matrixin const cVector3d & v must x and y equal you mouse screen coords and z set to 0Hope it helps.   ;Quote:Original post by Kostikov StasTry this code :*** Source Snippet Removed ***Replace cVector3d with D3DXVECTOR and set pos in identity matrixm_vp - it's viewport structurem_projection - it's projection matrixm_view - it's view matrixin const cVector3d & v must x and y equal you mouse screen coords and z set to 0Hope it helps.no it is not working but thanks   ;  http://www.mvps.org/directx/articles/rayproj.htm;  i got the question would like to ask. if i load a mesh do i need to translate the ray into model's space to detect it is hit or not? or into world space?I get the inverse projection, view and world matrix then i multiply it with my ray direction , however, the ray origin i set x,y and z to inverse world matrix _41,_42 and _43 and now i get message box without moving my mouse on mesh. wrong again, i am getting crazyhere is my project here;  I load a mesh (Tiger from media folder in directx sdk) and  draw on screen, i make mesh keep rotating and move circle. I create a ray depend on cursor position on screen and then transform it into world space and transform again into object's space do the hit detection(i use D3DXIntersect() function), but it is odd, when i move cursor on the object it say "not hit", but it say "hit" when my course is not on the object.is there anyone have same problem?Buy the way, how can i translate mesh to world space?   ;  ok now my problem has solved. it is working well now.thx   