Functional testing issue - Feels like one long test I'm using Selenium to test a PHP site that goes through the following process:


Register user
Log in
Fill out Form 1 and submit
Fill out Form 2 and submit
Fill out Form 3 and submit
(tear down) Delete user


A real user would go through these steps in this order.

I've read in places that tests should be bite-sized, and to avoid asserting a lot of things in each test. So I've written my tests like this:

function testRegister() {
    //...
}

/**
 * @depends testRegister
 */
function testLogin() {
    //...
}

/**
 * @depends testLogin
 */
function testFillForm1() {
    //...
}

/**
 * @depends testFillForm1
 */
function testFillForm2() {
    //...
}

/**
 * @depends testFillForm2
 */
function testFillForm3() {
    //...
}


This way my tests execute in the correct order.

However, splitting up my tests is merely cosmetic. Since each test is dependent on the success of the previous one, it may as well be one long test.

It also causes another problem. If testFillForm2 fails, I have no way of re-running the test on its own, I have to wait for all the other tests to complete each time because of the dependencies.

Is there a better approach I can take?
@depends does not define order


  [@depends] do not define the order in which the test methods are to be executed


Using @depends and not returning/passing arguments is an indicator that @depends isn't appropriate for what you're currently doing. You need to have a consistent application state before running each individual test method so that they do not depend on execution order. Otherwise tests easily become quite unwieldy and ultimately of little value.

If you want to do that via @depends, you need to return  from a test method and the following method should expect it as the first argument (and literally depend on whatever it receives).

Reduce dependencies

In tests you should aim to reduce, not add dependencies, otherwise you can be in quite a fragile position. If for example your register function breaks, it wouldn't affect existing users, or therefore all other tests, at all - but using cascading tests as shown you would be unable to test/demonstrate that. 

A broken register function would certainly be a problem - but is it a "100% of the site is broken" problem? It makes test results un-representative, and potentially harder to identify the root cause. If you have one failing test - it's very obvious where to focus your attention just from the test results. If all tests fail, it is not.

Of course anything more than of 0% fails warrants developer attention.

Achieve consistent application state

So, for example, each test should:


Establish a consistent application state (setUp functionality)
Use the concept of fixtures to e.g. populate the session, cookies, db - whatever is required
test what it needs to test
cleanup after itself (tearDown functionality)


In Context, I would say that means:

Create a test user to be able to test logging in without first being forced in the test case to "test" your register function:

public funtion testLogin() {
    // Insert a user directly into the users table - simulate anything else from testRegister
    ...
    $this-&gt;assertSame("UserName", $_SESSION['user']['name']);
}


You setup the session/cookie/whatever state for each subsequent test so that you don't need to "test" your login function to be able to test other authorized functions:

public funtion testFillForm1() {
    $this-&gt;_login();
    ...
}

public funtion testFillForm2() {
    $this-&gt;_login();
    ...
}

public funtion testFillForm3() {
    $this-&gt;_login();
    ...
}

protected function _login($user = array()) {
    if (!$user) {
        $user = array('name' =&gt; 'Default');
    }
    $_SESSION['user'] = $user;
}


If any of your setup steps are generic enough to apply to all tests, put them in the setUp function. E.g. loading 10 test users, with various permissions - put that in setUp and therefore testing login, registration-"that username already exists" would need no in-test setup at all.

I'm not exactly sure how that applies to using selenium, but hopefully this gives you enough of a hint to remove some of your inter-test dependencies.