   Unit Testing / TDD and real-time computer graphics
  Afternoon all,As I'd hope others here are, I'm a fan of good engineering practice. I've seen too much pain from cowboy coding in my time [grin]However I can't quite get into the mindset for TDD or unit-testing with real-time computer graphics applications (e.g. OpenGL or Direct3D stuff). From my online research it seems that for all the TDD or general testing guidelines out there graphics is really not covered in any sufficient detail.I've great respect for the regulars in this forum, so figured I'd come here on a fishing trip for any experience/comments on both the practical and conceptual side of testing graphics apps [wink]A few thoughts to get started:Real-world comparisonsIt's quite common in BRDF research papers for the authors to compare real-world measured results to computer simulated ones. For example Greg Ward's 1992 paper "Measuring and Modelling Anisotropic Reflection" details such a device. There is also the classic Cornell Box test-data as well.These tests work ok if you have the reference images (or the means to generate them) and are also striving for real-world realism (e.g. cel shading would be hard to test with this!).Standard unit-testing of an APIIt wouldn't be too hard to apply TDD or unit testing philosophies to normal D3D/OGL development. Testing HRESULT's and so on... but that'll only really test the correctness of your API usage and won't actually verify that the result is correct. You could quite easily write a completely correct D3D app that draws nothing or even utter garbage on the screen...You could probably write-up a test harness over an HLSL function that iterates over all inputs, but again you're probably stuck for reference test data. You could quite possibly go for known success/failure and fringe cases though.Testing against the reference rasterizerMore of a sanity check than anything as you'd really be testing the driver/hardware implementation rather than any code you'd written.Have I missed anything obvious in this list? Are there any other legitimate test strategies?If it wasn't clear already, I'm thinking more about the correctness of the image produced rather than the means by which it is created (e.g. legal/correct API usage). So if I were implementing a new BRDF, how could I write up some automated tests to verify that it was correct? How about a new post-processing effect - might be easier, but still doesn't seem to be covered in the test literature...All thoughts/comments welcome [smile]Jack   
  I'd like to use more automated testing in my graphics programming too. some vague suggestions, ideas rather than experience:tesing (image diff) against a reference image. This is useful when you already have the look correct, but need to optimise performance. You might be willing to live with say a maximum 25% overall pixel value difference. Obviously the allowable difference and the type of difference that's important to you depend on what you're doing. You may decide not to tolerate differences in shape (comparison of output images with an edge detect filter applied? comparison of simple white-on-black renders?), but to tolerate a certain amount of colour difference for example. IIRC you've posted images in your journal which show this kind of testing.An approach that can be useful in OpenGL, and I'd guess would apply to D3D to some extent, is to use glGet* to verify that the state machine is in the state you expect. This is something gDEBugger is very useful for, but I'd consider doing it at runtime in debug mode as a class or function invariant.I guess as graphics programming essentially boils down to numeric calculations, the main approach is to do testing on the numeric level rather than on the visual output.Sorry, not much to add.   ;Quote:I'd like to use more automated testing in my graphics programming tooGood to hear its not just me [grin]Quote:tesing (image diff) against a reference image. This is useful when you already have the look correct, but need to optimise performanceI hadn't considered this aspect of testing, its is a good one!Quote:IIRC you've posted images in your journal which show this kind of testing.Yup, I did exactly that. Although I didn't explicitly consider it to be a test at the time [lol]Quote:use glGet* to verify that the state machine is in the state you expectYup, that sort of thing can be a powerful test. I see quite a few threads in the DirectX forum where people think their state is correct but inspection via PIX reveals it isn't.However I'd argue that that approach is far more suited to the old-style fixed function pipeline where you could express everything via the state machine. Now that we've moved to shaders it becomes much harder as, within reason, you haven't a clue what that shader might do with the pipeline state...Quote:I guess as graphics programming essentially boils down to numeric calculations, the main approach is to do testing on the numeric level rather than on the visual output.True, so I wonder if launching HLSL code via some sort of GS+SO harness and effectively iterating over the range of inputs (e.g. for a BRDF you could have a 2D array of 'view' and 'light' vectors and the output can be compared against a set of results from MS Excel or something)...Cheers,Jack   