Pete Becker wrote:
Ian Collins wrote:
Pete Becker wrote:
Ian Collins wrote:
That is why it is important to fix every bug by adding a failing
test or
tests to the test suite that reveal the bug and then getting these to
pass. The tests remain in the suite, so the risk of regressing an old
bug is eliminated.
No, the risk is not eliminated. That eliminates the risk that whatever
test you wrote in response to the defect report hasn't failed. That
doesn't mean that the customer's code will work right, because the
previous fix might not have gotten at the entire problem, and some other
untested thing might now cause the customer's same code to fail. There's
no substitute for real-world code.
True, it may fail somewhere else, but the conditions described in the
original defect report will not cause problems.
That's not true. The added test tests what the developer or the tester
thinks caused the problem, after distilling the original defect report.
You still need to run the original code now and then, in case the
problem was more complex than the developer realized.
I think we are arguing at cross purposes here, what are tests run
against if not the original code?
I don't know where "There's no substitute for real-world code" came
from, tests are always run against real code.
Real code, yes. Real-world code, no. Most tests are fairly short,
intended to isolate a particular part of a specification. That's
important, but it's also important to incorporate real users in the
testing cycle, for example, through a beta test. They typically don't
find anywhere near as many problems as the internal testers do (back
when I managed Borland's compiler testing, over 90% of the defects were
found by internal testers), but the ones they find are often killers.
I see, you are talking about real world test cases. I agree these are a
must, but I tend to disagree with your description of tests. While
individual tests may cover individual user stories, specification
clauses or whatever, the test suite as a whole covers way more. If the
tests are designed by or with the customer, the application should be
being tested to ensure it does what it says on the tin in the way the
users expect.
The ability to achieve a very high test coverage with minimal labour
overhead is the biggest improvement offered with fully automated test
frameworks.
more tests in a given time). Test coverage is the result of design and
Roundhouse Consulting, Ltd. (www.versatilecoding.com)