text: .
  Artima SuiteRunner is both an API and an application. You can use the API to create tests and the application
  to run them. The Artima SuiteRunner API consists of one package,
  org.suiterunner.
  The main concepts in this API are represented by these three types:
  Suite- a class whose instances represent test suites (one to many tests)
  Reporter- an interface implemented by objects that collect test results and present them to the user
  Runner- a Java application that runs test suites
  To create a test suite, you subclass
  org.suiterunner.Suite (
  Suite) and define test methods.
  A test method is public, returns
  void, and has a name that starts with
  "test".
  For example, class
  AccountSuite from the account example is a subclass of
  Suite.
  Its test methods include
  testDeposit and
  testWithdraw.
  A
  Suite can hold references to other
  Suites. I call the referenced
  Suites
  sub-
  Suites of the referencing
  Suite.
  You organize a large test by building a tree of
  Suites. The base
  Suite in the
  tree has sub-
  Suites, each of which may have
  sub-
  Suites, and so on. When you execute the base
  Suite, it makes sure all
  Suites
  in the tree are executed.
  Executing a suite of tests involves the three methods shown in Table 1, which are declared in class
  Suite:
  Table 1. Methods used to execute a suite of tests.
  Each of the three methods listed in Table 1 accept a
  org.suiterunner.Reporter (
  Reporter)
  as a parameter. Information about the executing suite of tests will be sent to the specified
  Reporter.
  Suite.execute simply invokes two other methods declared in
  Suite:
  executeTestMethods and
  executeSubSuites, passing along the specified
  Reporter.
  Suite.executeTestMethods
  discovers test methods with reflection, invokes them, and sends results
  to the specified
  Reporter.
  Suite.executeSubSuites invokes
  execute on each sub-
  Suite.
  The three methods shown in Table 1 can be overridden in
  Suite subclasses if different behavior is desired.
  The
  Reporter interface declares several methods that are used to report information
  about a running test. Classes that implement
  Reporter decide how to present reported information to the user.
  For example, reported information could be displayed in a graphical user interface, written to a file, printed to the standard
  output or error streams, inserted into a database, organized into web pages, and so on. Table 2 shows the methods declared
  in interface
  Reporter that are used to report information about a running suite of tests.
  Table 2. Methods used to report information about a running suite of tests.
  Most of the
  Reporter methods shown in Table 3 accept an instance of class
  org.suiterunner.Report. Class
  Report is a bundle of information including
  an
  Object source, a
  String name, a
  String message, a
  Date,
  a
  Thread, and an optional
  Throwable. You can subclass
  Report if you
  wish to send additional information to custom
  Reporter classes that know how to use the
  additional information.
  By simply invoking
  execute on the base
  Suite in a tree of
  Suites,
  you cause all test methods in all
  Suites in the tree to be invoked. (This default behavior
  can be modified by overriding the execute methods shown in Table 1.)
  Since the
  Reporter is passed along to all
  execute methods, the
  results of all the test methods will be sent to the
  Reporter.
  You can inspect the test results presented by the
  Reporter.
  To run a
  Suite, you specify the class name of the
  Suite to the
  org.suiterunner.Runner (
  Runner) application.
  Runner loads
  the class, instantiates it, and invokes
  execute on the resulting
  Suite object, passing in a
  Reporter.
  Runner decides
  which
  Suite or
  Suites to execute and which
  Reporter or
  Reporters to pass based on
  either command line parameters or
  a recipe file. A recipe file contains properties that define information
  that describes a single run of a particular suite of tests. If you specify multiple
  Suites,
  Runner executes them sequentially. If you specify multiple
  Reporters,
  Runner adds them to a single dispatch
  Reporter that forwards all method
  invocations to each specified
  Reporter.
  Runner then passes the dispatch
  Reporter
  to the specified
  Suite's execute method.
  Runner also provides a graphical user
  interface that gives you many more ways to run suites of tests and inspect their results.
  Suite
  To create a test suite, you simply subclass
  Suite and define test methods.
  For example, for the account exampleI CTK I created class
  AccountSuite
  and gave it three test methods.
  As shown in Figure 1,
  AccountSuite is a subclass of
  Suite.
  Its three test methods are
  testConstructor,
  testDeposit, and
  testWithdraw.
  Figure 1. AccountSuite extends Suite and defines test methods.
  If your are familiar with JUnit, you can think of class
  Suite as the
  JUnit types
  TestCase,
  TestSuite,
  Assert,
  and
  Test all collapsed into one type. Whereas JUnit uses the composite
  pattern to combine test cases into test suites, Artima SuiteRunner uses plain old composition.
  Any Artima SuiteRunner
  Suite can hold references to other
  Suites.
  A fixture consists of objects and anything else needed to perform a test.
  In general, all test methods in a
  Suite
  share the same fixture, which is usually composed of objects
  stored in private instance variables of the
  Suite. You can create
  fixtures in either of two ways, via the constructor of your
  Suite
  subclass or via
  Suite's
  setupFixture method.
  Suite.executeTestMethods calls
  setupFixture before
  invoking each test method, and
  cleanupFixture after
  each test method completes.
  These methods can be used to create a fresh fixture
  before each test method, and destroy it afterwards.
  The
  setupFixture and
  cleanupFixture methods are useful
  when your test methods destroy or change the fixture.
  If you are certain your test methods won't destroy the fixture, and you don't need to perform
  any fixture cleanup, you can simply initialize the private variables that represent your fixture in
  a constructor of your
  Suite subclass.
  Class
  Suite's
  setupFixture and
  cleanupFixture methods are similar to
  setup and
  teardown methods of JUnit's class
  TestCase.
  Unlike JUnit, however, Artima SuiteRunner's
  Suite.executeTestMethods invokes all test methods on
  the same
  Suite instance.
  (JUnit generally creates a different
  TestCase instance for each test method.) If any of
  your test methods destroy its fixture such that the fixture can't be used
  by sibling test methods invoked later on the same
  Suite object, you should
  use
  setupFixture to create the fixture.
  In
  setupFixture, you create the necessary objects and perform any other
  tasks to ready the fixture, such as opening a file or socket. In
  cleanupFixture, you
  can release the objects to the whims of the garbage collector and perform
  any other necessary cleanup, such as closing the file or socket.
  Test methods may have one of two signatures:
  public void test...()
  public void test...(Reporter reporter)
  The
  "..." in
  "test..." represents
  any non-zero length string. Some example test method names are
  testFest,
  testimonial, and
  testOfCharacter.
  Test methods indicate success by returning, failure by throwing an exception. The
  Artima SuiteRunner API includes one exception,
  TestFailedException, whose
  purpose is to indicate a failed test.
  Suite.executeTestMethods interprets any exception thrown from a test method, not
  just
  TestFailedException, as an indication of failure.
  In the body of test methods, you can take advantage of these six methods declared in
  superclass
  Suite:
  public static void verify(boolean condition)
  public static void verify(boolean condition, String message)
  public static void fail()
  public static void fail(String message)
  public static void fail(Throwable cause)
  public static void fail(String message, Throwable cause)
  The
  verify methods check the specified Boolean condition.
  If the condition is
  true,
  verify returns
  quietly. Else,
  verify throws
  TestFailedException.
  The
  verify method that takes a
  String
  message parameter uses that
  String for the
  detail message of the thrown exception.
  The
  fail methods always throw
  TestFailedException.
  If a
  String
  message or
  Throwable
  cause is provided, the
  fail method uses these as
  the detail message and cause for the thrown
  TestFailedException.
  Test methods generally do not catch
  TestFailedException. Instead,
  they complete abruptly with the exception, thereby indicating failure.
  The calling method, usually
  Suite.executeTestMethods, catches
  the exception and reports the failure to the
  Reporter.
  The
  Reporter in turn passes the information in some manner along
  to the user.
  As an example, here's the
  testDeposit method of class
  AccountSuite from the
  account example:
  public void testDeposit() { Account account = new Account(); account.deposit(20); long bal = account.getBalance(); verify(bal == 20, "Account.deposit() didn't deposit 20 correctly. " + "Resulting balance should have been 20, but was " + bal + "."); account.deposit(20); bal = account.getBalance(); verify(bal == 40, "Account.deposit() didn't deposit 20 twice correctly. " + "Resulting balance should have been 40, but was " + bal + "."); try { account.deposit(-1); fail("Account.deposit() didn't throw IllegalArgumentException when " + "negative value passed"); } catch (IllegalArgumentException e) { // This is supposed to happen, so just keep going } account = new Account(); account.deposit(Long.MAX_VALUE); verify(account.getBalance() == Long.MAX_VALUE, "account.deposit() " + "couldn't handle Long.MAX_VALUE"); account = new Account(); account.deposit(1); try { account.deposit(Long.MAX_VALUE); fail("Account.deposit() didn't throw ArithmenticException when a " + "value passed that would cause overflow"); } catch (ArithmeticException e) { // This is supposed to happen, so just keep going } }
  Inside the
  testDeposit method, I call
  deposit (the target of this test method) several
  times on various
  Account objects, making sure it either performs the correct action or throws the
  expected exception.
  For example, after you deposit 20 into an
  Account, that
  Account's
  getBalance
  method should return 20.
  The first
  verify statement in the method checks to make sure
  that
  getBalance returns the expected value of 20:
  account.deposit(20); long bal = account.getBalance(); verify(bal == 20, "Account.deposit() didn't deposit 20 correctly. " + "Resulting balance should have been 20, but was " + bal + ".");
  In addition, the contract of
  Account.deposit states that the method should
  throw
  IllegalArgumentException if the requested deposit is less than
  or equal to zero. The following
  code ensures this behavior works according to the contract:
  try { account.deposit(-1); fail("Account.deposit() didn't throw IllegalArgumentException when " + "negative value passed"); } catch (IllegalArgumentException e) { // This is supposed to happen, so just keep going }
  The previous code snippet shows one use of the
  fail method. If
  account.deposit(-1)
  throws
  IllegalArgumentException as expected, it will be caught by the empty catch clause and the test method
  will continue.
  But if
  account.deposit(-1) returns normally, instead of throwing a
  IllegalArgumentException
  as required, the
  fail method will be invoked resulting in a
  TestFailedException. If
  account.deposit(-1)
  throws a different
  exception besides
  IllegalArgumentException, then the entire
  testDeposit method
  will complete abruptly with that exception. The calling method (normally
  executeTestMethods)
  will catch any exception and report the failure to the
  Reporter.
  Artima SuiteRunner's
  TestFailedException corresponds to JUnit's
  AssertionFailedError. Artima SuiteRunner's two
  verify methods correspond to the
  JUnit's multitudinous
  assert methods declared in class
  Assert.
  Unlike JUnit, Artima SuiteRunner does not differentiate between "failures" and "errors." JUnit
  calls any thrown
  AssertionFailedError a failure, any other
  thrown exception an error. In Artima SuiteRunner,
  a test either succeeds or fails. If the test fails, the user can inspect information
  about the failure to better understand how to correct the problem.
  Suitesto Build a Tree of
  Suites
  A
  Suite consists of zero to many test methods and zero to many sub-
  Suites.
  As mentioned previously, a sub-
  Suite is merely a
  Suite that is held in a composition relationship by another
  Suite. The referenced
  Suite is called a sub-
  Suite of the referencing
  Suite.
  The customary manner to create larger tests is to define focused
  Suite classes, then aggregate them
  together as sub-
  Suites of other
  Suite classes. An entire test suite is then represented in memory
  by a tree of
  Suite objects, starting at a base
  Suite whose
  execute method
  kicks off a run of the entire suite of tests.
  The account example
  has a main
  Suite, called
  AccountTestKit, that has no test methods, only sub-
  Suites.
  Its constructor
  instantiates two
  Suite objects and adds them as sub-
  Suites to itself via
  the
  addSubSuite method.
  AccountTestKit represents the full conformance
  test kit for the
  com.artima.examples.account.ex6 package.
  Here is the entire
  AccountTestKit class:
  package com.artima.examples.account.ex6test; import org.suiterunner.Suite; public class AccountTestKit extends Suite { public AccountTestKit() { addSubSuite(new AccountSuite()); addSubSuite(new InsufficientFundsExceptionSuite()); } }
  Figure 2 shows the structure of the
  AccountTestKit in memory.
  AccountTestKit, the
  base
  Suite in the tree, holds references to two sub-
  Suites.
  These two sub-
  Suites,
  AccountSuite and
  InsufficientFundsExceptionSuite,
  declare test methods but do not themselves contain sub-
  Suites.
  Figure 2. The
  AccountTestKit is a tree of
  Suite objects.
  When the
  Runner invokes
  execute on the
  AccountTestKit object,
  execute first invokes
  executeTestMethods on itself.
  executeTestMethods uses reflection to discover that
  AccountTestKit has no test methods,
  and returns.
  execute then invokes
  executeSubSuites, which invokes
  execute
  on each of its two sub-
  Suites. The
  execute methods of each of those
  Suites ensures
  that all test methods executed.
  To run a suite of tests, you use the Artima SuiteRunner application.
  The Artima SuiteRunner application is distributed as an executable JAR file with the name
  suiterunner-[release].jar, where the
  [release] is the release name. For
  example, the name of the executable JAR file for the 1.0beta2 release of Artima SuiteRunner is:
  suiterunner-1.0beta2.jar.
  In this section, I'll refer to the executable JAR file without a release name:
  suiterunner.jar.
  The Artima SuiteRunner application is embodied in the
  main method of
  org.suiterunner.Runner (
  Runner).
  The
  Runner application accepts command line arguments that specify an optional recipe file, runpath,
  zero to many
  Reporters, and zero to many
  Suite classes.
  Runner can be started in either of two ways. If
  suiterunner.jar is available on the classpath,
  then the command line takes the following form:
  java [-cp <classpath>] org.suiterunner.Runner [<recipefile>] [-p <runpath>] [reporter [...]] [-s <suiteclass> [...]]
  If
  Runner is started using the executable jar file, the command line takes the following form:
  java -jar suiterunner.jar [<recipefile>] [-p <runpath>] [reporter [...]] [-s <suiteclass> [...]]
  A recipe file contains properties that define runpath,
  Reporters, and
  Suites.
  If a recipe file is specified, it must be the first argument. Any other arguments following the
  recipe file are ignored. The standard file extension for Artima SuiteRunner
  recipe files is
  ".srj".
  A runpath is the list of filenames, directory paths, and/or URLs that Artima SuiteRunner
  uses to load classes for the running test. If runpath is specified, Artima SuiteRunner
  creates a
  java.net.URLClassLoader to load classes available on the runpath.
  The graphical user interface can optionally reload the test classes for each run
  by creating a new
  URLClassLoader for each run.
  If the executable JAR file is available on
  the Java classpath, then the classes that comprise the test may also be made available on
  the classpath and no runpath need be specified.
  The runpath is specified with the -p option. The -p must be followed by a space,
  a double quote (
  "), a white-space-separated list of
  paths and URLs, and a double quote. For example:
  -p "serviceuitest-1.1beta4.jar myjini"
  Reporters can be specified on the command line in any of the following ways:
  -g[configs...]- causes display of a graphical user interface that allows tests to be run and results to be investigated
  -f[configs...] <filename>- causes test results to be written to the named file
  
  -r option causes the
  Reporter specified in
  <reporterclass> to be
  instantiated.
  Each
  Reporter class specified with a -r option must be public, implement
  org.suiterunner.Reporter, and have a public no-arg constructor.
  Reporter classes must be specified with fully qualified names. If the Artima SuiteRunner JAR file is
  available on the classpath, not run directly as an executable JAR file, the specified
  Reporter classes can also be
  deployed on the classpath. If a runpath is specified with the
  -p option, specified
  Reporter classes may also be loaded from the runpath.
  All specified
  Reporter classes will be loaded and instantiated via their no-arg constructor.
  For example, to run a
  Suite using two
  Reporters, the graphical
  Reporter and a print
  Reporter
  writing to a file named
  "testresults.txt", you would type:
  -g -f testresults.txt
  The
  -g,
  -o, or
  -e options can
  appear at most once each in any single command line.
  Multiple appearances of
  -f and
   on itself to each contained reporter.
  Runner will pass
  the dispatch reporter to executed
  Suites. As a result, every
  specified reporter will receive every report generated by the running suite of tests.
  If no reporters are specified, a graphical
  Runner will be displayed that provides a graphical report of
  executed
  Suites.
  Each reporter specification on the command line can include configuration parameters. Configuration parameters
  are specified immediately following the
  -g,
  -o,
  -e,
  -f, or
  -r. Valid configuration parameters are:
  Y- report
  runStartingmethod invocations
  Z- report
  testStartingmethod invocations
  T- report
  testSucceededmethod invocations
  F- report
  testFailedmethod invocations
  U- report
  suiteStartingmethod invocations
  P- report
  suiteCompletedmethod invocations
  B- report
  suiteAbortedmethod invocations
  I- report
  infoProvidedmethod invocations
  S- report
  runStoppedmethod invocations
  A- report
  runAbortedmethod invocations
  R- report
  runCompletedmethod invocations
  Each reporter has a default configuration. If no configuration
  is specified on the command line for a particular reporter, that
  reporter uses its default configuration.
  Runner will configure each reporter for
  which configuration parameters are specified
  via the reporter's
  setConfiguration method.
  For example, to run a
  Suite using two reporters, the graphical reporter (using its default
  configuration) and a print reporter configured to print only test failures, suite aborts, and
  run aborts, you would type:
  -g -eFBA
  Note that no white space is allowed between the reporter option and the initial configuration
  parameters. So
  "-e FBA" will not work, and must be
  changed to
  "-eFBA".
  Suites are specified on the command line with a -s followed by the fully qualified
  name of a
  Suite subclass, as in:
  -s com.artima.serviceuitest.ServiceUITestkit
  Each specified
  Suite class must be public, a subclass of
  org.suiterunner.Suite, and contain a public no-arg constructor.
  Suite classes must be specified with fully qualified names. If the Artima SuiteRunner JAR file is available
  on the classpath, not run directly as an executable JAR file, the specified
  Suite classes can be
  loaded from the classpath. If a runpath is specified with the
  -p option, specified
  Suite classes may also be loaded from the runpath.
  All specified
  Suite classes will be loaded and instantiated via their no-arg constructor.
  The
  Runner will invoke
  execute on each instantiated
  org.suiterunner.Suite,
  passing in the dispatch reporter to each
  execute method.
  You can also use the -s parameter to specify JUnit test cases. To run JUnit tests from
  Artima SuiteRunner,
  junit.jar and the test cases you wish to run must be available via the
  runpath. Or, if you run Artima SuiteRunner from the class path, you may put
  junit.jar
  and your test cases on the class path. Each JUnit test case specified must be a subclass of
  junit.framework.TestCase, and contain a public constructor that takes a single
  String parameter. JUnit is not included in the Artima SuiteRunner distribution. If you
  do not already have
  junit.jar, you can download it from junit.org
  The Artima SuiteRunner API is designed to give client programmers much room to customize its behavior.
  For example, the contracts of the classes and interfaces in the Artima SuiteRunner API use the words "test" and "suite"
  generically to allow subclasses and implementation classes maximum flexibility. A
  org.suiterunner.Suite
  is one kind of suite. A test method in a
  org.suiterunner.Suite subclass is one kind of test. But you
  can define other kinds of suites
  and tests if you wish. For example,
  JUnitSuite, a package-access subclass of
  Suite
  in the
  org.suiterunner package, defines a "test" as a test method in a JUnit test case.
  Aside from just declaring test methods in your
  Suite subclasses, you can
  override many methods declared in
  Suite. Of particularly usefulness is overriding
  executeTestMethods,
  executeSubSuites, or
  execute itself.
  For example,
  JUnitSuite
  overrides
  execute so that it runs a suite of JUnit test cases.
  JUnitSuite
  overrides
  getTestCount
  so that it returns the expected number of JUnit test cases to be run.
  Artima SuiteRunner comes with several
  Reporters built in:
  a graphical
  Reporter that presents test results via a graphical user interface; a print
  Reporter that writes
  results to the standard output, standard error,
  or a file; and a dispatch
  Reporter that forwards results to multiple
  Reporters.
  You can make your own
  Reporters to customize the way test results
  are presented simply by creating classes that implement
  Reporter.
  Some ideas for custom
  Reporters are:
  Reporterthat writes results to XML
  Reporterthat puts results into a database
  Reporterthat posts results to a log using the Java SDK 1.4 logging mechanism
  Reporterthat writes results to one or more web pages
  Reporterthat send a summary email at the end of a run
  Reporterthat blows a fog horn when a test fails and fires a cannon when a test succeeds
  For example, if as part of your nightly build process you automatically run a full test suite on your
  software, you could define
  Reporters that email a summary
  of the results each night to team members and place detailed results on a set of web pages.
  Most
  Reporter methods take a single parameter of type
  Report includes many pieces of information, such as a
  Thread that
  can help you understand multi-threaded tests.
  If you wish to pass more information than you can with a standard
  Report, you
  can create a subclass of
  Report that has extra
  information. You can then override any of the
  execute,
  executeTestMethods,
  or
  executeSubSuites methods, and create test methods in your
  Suite
  subclasses. These methods can fire instances of your
  Report subclass
  which include the extra information you desire to have reported.
  You can then
  create custom
  Reporters that downcast the
  Report to your new
  Report subclass, extract the extra information, and report it to the user.
  Artima SuiteRunner is an API and application that can help you build and run unit and conformance
  tests. The main concepts of the API are represented by the types
  Suite,
  Reporter,
  and
  Runner. You define test methods in subclasses of
  Suite. You use
  Reporters to present test results. And you use
  Runner to run the
  Suites.
  Runner uses recipe files, which contain
  settings that describe how to perform a particular run.
  You can create recipe files to reuse later, to share among the members of a team, or to distribute
  to the public as part of a conformance test kit.
  Artima SuiteRunner is to a great extent JUnit refactored.
  If you have existing JUnit tests, you can use Artima SuiteRunner to run them.
  In JUnit, runners are a separate concept from the framework API. The JUnit distribution
  includes three runners: a text runner, which write test results to the standard output;
  an AWT runner, which presents test results via an AWT graphical user interface; and a Swing runner,
  which presents test results via a Swing graphical user interface.
  One way to think of Artima SuiteRunner, therefore, is simply as another JUnit runner. Using Artima SuiteRunner to
  run your JUnit tests allows you to enjoy features not
  present in the three standard JUnit runners, such as recipes,
  Reporters, runpaths, and full stack traces in reported results.
  For help with Artima SuiteRunner, please post to the SuiteRunner Forum.
  Resources
  JUnit is available at:
  Why We Refactored JUnit: