What open source software should I use to write scripts to test for no errors on a site?

Could I / we write something better ourselves if there a limited number of goals outlined?
 - yet flexible enough to take on new rules etc.

The only consistent response we want is no errors, period.

I know Java, ASP and scripting languages if that helps.

Thanks!