   [C++] How would you design a framework for per-application tests (no unit testing) ?
  Hello,I`m trying to gather as much opinions as possible for our current SW need, which is designing a framework that can easily handle automated checks/validations and new validations can be added easily. Majority of apps will use just command-line version, but there are few apps that will need to have an UI, so the data has to be easily accessible to GUI component. The output is a regular LOG file with passed/failed (with error status) info per each test.The tests in general revolve around file structure/naming/XML scheme. The tests itself are easy to do (and in fact, they have already been done for few apps).But this time we might have enough time to redesign it from scratch and that`s why I`d like to gather as much input as possible. I have my own views how to refactor the current code, but I`d like to use this opportunity to learn something different/some new design pattern maybe.Oh, and this time it has to run on non-MS platform - Solaris.So, how would you design it ?[Edited by - joe1024 on May 13, 2009 6:18:48 AM]
  I'd personally lean towards using an existing framework; there's quite a bit of information to be had via a quick Google search.   ;  Last time I did this I started with CppUnitLite, which provides a good core framework and heavily customised it. CppUnitLite is particularly nice for this as it avoid using the fancier C++ features so it's very portable, and it's quite simple at it's core so it's easy to customise.I also found it very helpful to write tests for the test framework (in the test framework). That's a little mind bending when you start but it was really helpful in the long term (and resulted in a better overall architecture).Oh, and accept that for good results you're going to have to delve into platform specific APIs somewhere along the line (for memory leak checking, or catching and processing access violations, or printing stack traces on errors, etc.), so plan for that up front so you can abstract them away in isolated modules rather than having #ifdefs scattered everywhere.   ;  Oh, I have used the wrong term probably.I don`t mean to use Unit Testing. By testing we mean the actual application tests, e.g. if the data file has correct fields (e.g. XML schema test), or if there are certain combinations of files with specific filename scheme, simply lots of specific application features.There are several end-user applications that are tailored to their specific needs, and each one has a set of specific test/validations that are needed to be run.We already have coded few of these applications, but I`m not looking to merely refactoring it, I`m looking for your idea how would you desing a framework that would allow an easy plug&play of new tests/validations.More specifically, what design patterns would you use.I have my own ideas, but I don`t want to let those fool you, so I`d rather hear your fresh ideas about the design.A simple example:App1 has 4 tests : Test0, Test1, Test2, Test3App2 has 7 tests : Test0 - Test6App3 has 3 tests : test0 - Test2Some of those test among the applications might overlap (basic file-existence tests, common filenaming scheme, XML compliance, ...), but some might have the app-specific falvour (currently implemented through virtual methods).Hope that explains it better.   ;  Still sounds like a candidate for using an existing unit testing framework to me [smile]   ;  Ditto. The only difference between unit tests and functional tests is how much code is actually in the test cases, in which case any existing unit test framework will be fine.If that's not what you're after then perhaps a specific example of what a current test looks like and is supposed to achieve would help?   ;  Well, I don`t understand how could you apply unit testing to this.The tests itself are the core of the application`s functionality that has to validate a set of files against a set of conditions (tests). These tests are tailored per each specific application/project, but they`d use a common framework for the adding/removing the tests depending on the needs of particular project.These apps will be run from command line, there`s no way the user would start Visual Studio.And I`m looking for an input how to design this framework ideally.   ;  There is no need to run Unit Tests from within visual studio. I can start my unit testing application from the Windows Explorer or from the commandline, when I'm in the mood ;)Basically stuff that is not hard coded needs to be verified, but the same thing that applies to "normal" unit tests can be applied here. You create a set of rules and catch the exceptions that are thrown when a criteria is not met. The only difference I see is that the test is actually not hard coded, but created from user input.To your actual problem. I think you will need to specify what exactly needs verification, but a general solution would be to create a structure that is able to verfiy the things that need verification. Those structures are then parsed by the framework and feed with the data. When something does not match an exception is thrown and reported to the user.You could use XML to store those structures (in the end you want to verify XML files, right?), probably something like this:<Verfiy expected="1" actual="count(body)" msg="There can only be one body-tag" />You would need to offer some functions (like count() in this example), but I think this can be expanded into a big Testing framework.   ;  Well, all tests are hardcoded, the only meaningful user input lies in specifying a set of directories or selecting which tests are meant to be run on those files.Haven`t tried unit-testing from outside of the VS,so I don`t know.Of course, it has to be a standalone app, I can`t call anything externally.I just want to redesign it completely. While the idea of using unit testing for this scenario is funny, that`s not what I`m looking for, which is the actual design patterns or class hierarchy outline of such a framework.   ;  Your vauge descriptions mean I'm struggling to understand what your current tests actually do. Are the tests actual pieces of C++ code which run against your application code? Are they in an external data file which is somehow read in and interpreted and sends button clicks to your app? Or does it run your app with a set of command line options and verifies the file output of the program against static, canned results?Similarly it's rather useless to say "I want to know how to refactor this" without saying what you're starting with. What's the current class structure and the relationship between them? What works well? What's a pain to maintain? What's a source of bugs?   