Re: Testing in C++
Ian Collins wrote:
Pete Becker wrote:
Ian Collins wrote:
That is why it is important to fix every bug by adding a failing test or
tests to the test suite that reveal the bug and then getting these to
pass. The tests remain in the suite, so the risk of regressing an old
bug is eliminated.
No, the risk is not eliminated. That eliminates the risk that whatever
test you wrote in response to the defect report hasn't failed. That
doesn't mean that the customer's code will work right, because the
previous fix might not have gotten at the entire problem, and some other
untested thing might now cause the customer's same code to fail. There's
no substitute for real-world code.
True, it may fail somewhere else, but the conditions described in the
original defect report will not cause problems.
That's not true. The added test tests what the developer or the tester
thinks caused the problem, after distilling the original defect report.
You still need to run the original code now and then, in case the
problem was more complex than the developer realized.
A test suite is seldom,
if ever, perfect, but adding tests for field failure cases is an
excellent means of improving it. Not adding tests for field failure
cases is nothing short of negligent.
I don't know where "There's no substitute for real-world code" came
from, tests are always run against real code.
Real code, yes. Real-world code, no. Most tests are fairly short,
intended to isolate a particular part of a specification. That's
important, but it's also important to incorporate real users in the
testing cycle, for example, through a beta test. They typically don't
find anywhere near as many problems as the internal testers do (back
when I managed Borland's compiler testing, over 90% of the defects were
found by internal testers), but the ones they find are often killers.
--
-- Pete
Roundhouse Consulting, Ltd. (www.versatilecoding.com)
Author of "The Standard C++ Library Extensions: a Tutorial and
Reference." (www.petebecker.com/tr1book)