From 1189691eaa22f5d37ffba23efb3b65bac8c4191a Mon Sep 17 00:00:00 2001 From: zooko <> Date: Fri, 30 Mar 2012 23:06:25 +0000 Subject: [PATCH] instructions on the Why of code coverage [Imported from Trac: page HowToWriteTests, version 3] --- HowToWriteTests.md | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/HowToWriteTests.md b/HowToWriteTests.md index 7d76665..8d5e491 100644 --- a/HowToWriteTests.md +++ b/HowToWriteTests.md @@ -13,13 +13,17 @@ touch src/allmydata/test/test_fname.py Okay, so it didn't work, because there is no file by that name. Create such a file, with these contents: ``` -rom twisted.trial import unittest +from twisted.trial import unittest class T(unittest.TestCase): def test_a(self): pass ``` +Now run it! + +# code coverage + Now install Ned Batchelder's "[coverage"](http://nedbatchelder.com/code/coverage/) tool and run your with code coverage, like this: ``` @@ -38,4 +42,16 @@ This does the same as running the tests without coverage -- print a list of what ./bin/tahoe @coverage html ``` -That will product a directory named `htmlcov`. View its contents with a web browser. \ No newline at end of file +That will product a directory named `htmlcov`. View its contents with a web browser. + +# using code coverage results + +This is important: we do not treat code coverage numbers as a litmus test (like "aim to have 90% of lines covered"). We hardly even treat it as a scalar measurement of goodness -- 91% code coverage is not necessarily better than 90% code coverage. Maybe the alternative would have been to remove some (covered) lines of code that were not necessary, which would have resulted in a worse "code coverage" metric but a better codebase. Finally, note that even if you have 100% branch-level coverage of a codebase, that doesn't mean that your tests are exercising all possible ways that the codebase could be run! There could be data-dependent bugs, such as a divide-by-zero error, or a path which sets one variable to a setting which is inconsistent with a different variable. These sorts of bugs might not be getting exercised by the test code even though every line and every branch of the code is getting tested. + +So what do we use it for? It is a lens through which to view your code and your test code. You should look at the code coverage results and think about what it says about your tests. Think about "what could go wrong" in this function -- where bugs could be in this function or a future version of it -- and whether the current tests would catch those bugs. Both authors of patches and reviewers of patches should look at the code coverage results, and see if they indicate important holes in the tests. + +Likewise, even if the code coverage shows maximal coverage, you should *still* think "Are there any kinds of bugs that could exist in this or a future version of this that *wouldn't* be caught by these tests?". + +# further reading + + \ No newline at end of file