I started writing this as a comment, but I felt it’s own post was deserved.
Ricky Clarkson left me a link in a comment to one of his posts that ties in quite nicely to my recent Getting with it: Test Driven Development post.
Ricky makes the point that TDD can be dangerous, and can lull you into a false sense of security. I agree.
As with any technology, when used incorrectly it can cause more damage than not using it at all.
Ricky’s example should serve as a double-barreled warning. You can’t be test-driven only when it’s convenient, and you need to have good test coverage.
Being Test-Driven isn’t being Convenience-Driven
Just by being in a car with seat-belts doesn’t mean they’ll save you when you crash, you need to actually wear them.
You can’t pick and choose which parts of TDD you want to use, then be surprised when it lets you down.
If your boss comes storming in demanding a copy of the system, tell him to wait. Remember, it’s your hide on the line, if you give him a broken system you’ll be the one to pay. If he doesn’t understand testing, lie, tell him it’s building or something similar, it doesn’t exist until it’s finished.
Test coverage
There’s very little more important than good test coverage, without it you’re leaving yourself wide open for problems.
In Ricky’s example, he’s fallen into a common problem with test design. He’s only testing the expected outcome of his method. We’ve all fallen into this trap at some point, but it’s a dangerous place to be. Without full coverage, just as Ricky said, you end up with a false sense of security.
For full test coverage of a method, you really need to test it’s expected outcome, how it handles edge-cases, and how it handles bad data.
(assert (equal (point+ (point 999 987) (point 789 998)) (point 1788 1985)))
Had Ricky’s test suite included the above edge-case test, it would have caught the flaw in his method’s design, and been able to correct it. A few more like the above, to cover minus numbers and very low values, and his situation would have been greatly different.
If there’s a way you can improve testing, it’s to write more unit tests. Only leave those bits you really can’t test to integration. Everything else should be adequately covered.
In parting
Remember, using TDD isn’t an excuse to leave your common sense at the door. If you’re writing a method which you know is going to be used in several places, test those several places, give it very good coverage. If you’re writing a sum function like Ricky, you know how it’s going to be used, don’t just write if statements to cover all your input data, that’s using coding to make a test pass an excuse to write bad code.
Comments...
So what you’re saying is that if you have only one case being tested, then it’s ok to hard-code the answer into the test case, but if you have 10 cases, then it’s not ok and you should then write the method properly.
I disagree – you should write the method properly in the first place. If you don’t, then even the first test case is useless.
In terms of coverage, if that refers to whether lines are executed or not, consider the identity function:
(lambda (x) x) (or, in Java, Object id(Object x){return x;}).
If I write my test for that to only test for 5, it makes zero sense to write the identity function as (lambda (x) 5) (or, return 5;).
The identity function is so simple that it really doesn’t need (runtime) tests anyway, of course, but what is plain is that you get complete code coverage of it from one test. Writing ten tests for it doesn’t improve matters. As soon as your test code is bigger (conceptually, not necessarily in lines of code) than the code it’s testing, you’re probably trying to prove through testing, which is usually impossible.
I’d suggest you use testing to give yourself confidence, not to prove things. Proofs can be constructed with other tools, such as type systems (or with your brain).
By Ricky Clarkson — 7 Jul, 2007 @ 3:32 pm
Well, I think proof is part of confidence. I’m not a big believer, faith doesn’t come into programming for me. If I write a method, I want proof that it works how it’s supposed to. Maybe when it was written it was obvious how things would work, but leave it a year and have several other programmers work on it, and how sure are you now?
Using your example, it’s obviously very simple and doesn’t require a wealth of tests, but I still think you need to at least test if it works under valid and invalid conditions. Yes, we both know how it works now, but what happens when somebody replaces { return x; } with { if (x == null) return new NullObject() else return x; }.
These examples are a bit contrived though, testing something as tiny and abstract probably wont fall into the usual realm of testing. As long as these methods are covered in your other more substantial tests then I probably wouldn’t worry about them.
By James Gregory — 7 Jul, 2007 @ 4:25 pm
Testing can only prove that your code works for the cases that you test. Exhaustive testing is impractical for many functions.
Maybe you should look at a real system for constructing proofs, such as Haskell, or Epigram. My experience with Haskell is that the effort required to construct the proof is greater than the need for the proof. All I really need is some confidence in the code, usually. There may be cases where I need real proof, and that is when a type checker may be useful.
I’m not saying that automated tests are useless, merely that applying a rule that states that they should always be done before writing the code that they test, is likely to be harmful. Also, that the intermediate stage, where you write naive code to make a single test pass, is also harmful.
By Ricky Clarkson — 7 Jul, 2007 @ 3:45 am
I am happy I found your site on bing. Thank you for the sensible critique. Me and my brother were just preparing to do some research about this. I am very happy to see such trusted info being shared for free out there.
Regards,
Amadeus from Eugene city
By Brychan — 2 Feb, 2010 @ 8:32 am
Post a comment...