Reading Assignment: All of Programming Ch6 Fixing Your Code: Testing and Debugging
Testing and Debugging are two closely and distinct steps.
- Testing is the process of trying to find problems(“bugs”) in your code.
- Debugging is the process of fixing those bugs.
Step 6: Testing
Testing is all about finding bugs in your code. A good test case is one that identifies a problem.
BAD CODE KILLS PEOPLE.
One of the keys to testing your code well-both thoroughly and in a manageable fashion-is to test incrementally. Write one function, then test it well before moving on.
Black Box Testing
The testing methodology that most people think of first is black box testing. The lack of access to the implementation details is where this testing methodology gets its name-the function’s implementation is treated as a “black box” which the tester cannot look inside.
White Box Testing
White box testing involves examining the code to devise test cases. One consideration in white box tests is test coverage–a description of how well your test cases cover the different behaviors of your code.
We will discuss three levels of test coverage: statement coverage, decision coverage, and path coverage.
- Statement coverage means that every statement in the function is executed.
- Decision coverage means that all possible outcomes of decisions are exercised. In order to visualize decision coverage, we need to understand the concept of a control flow graph, whose nodes are basic blocks, and whose edges represent the possible ways the control flow arrow can move.
- Path coverage is the strongest type among these three. Our test cases must span all possible valid paths through the control flow graph.
So how to pick the right level of test coverage? The answer is “it depends”. The first aspect of this decision is “how confident do you need to be in your code?” If you are doing preliminary testing of your algorithm by hand in Step 4, then statement coverage is a reasonable choice. If you are testing a piece of critical software which will be deployed when you finish testing, you would want to aim for more than just the minimum to achieve path coverage. If you require absolute certainty that the code is correct, you must prove it.
Generating Tests
Asserts
In almost all cases, giving the wrong answer due to an undetected error is significantly worse than the program detecting the situation and crashing.
Regression Testing
Regression Testing is the process of running a set of test cases–your regression test suite–which have worked in the past to ensure that they still work on your current code. On large software projects with many developer, a common practice is to run “nightly regressions”–run the regression test suite on the code each night.
Code Review
In a code review, another skilled programmer reviews the code you have written, looking for mistakes you might have made. Code reviews have one nice advantage over other forms of testing: often when your reviewer identifies a problem, she can propose steps towards fixing it.
One form of code review is a code walk-through in which the programmer explains the algorithm and code to the reviewer.
Step 7: Debugging
Once you have found a problem in your code, you need to fix it–this process is called debugging.
Observe a Phenomenon
In programming, our observation of phenomena relates to the behavior of our programs on certain inputs.
Ask a Question
Discovering what is wrong in a iterative fashion is perfectly fine, and in fact a great way to proceed. Eventually, your chain of questions and answers leads you to the discovery of the problem, even if it somewhat far removed from the visible symptom.
Gather Information, Apply Expert Knowledge
The next step of the scientific method is actually to gather information and combine it with your expert knowledge.
The simplest way to gather information is to insert print statements (in C, calls to printf) to display the values of various variables at various points in the program.` Although it is simple, it has several disadvantages. One is that you need to recompiling and re-running your program. Another one is that the output may be overwhelming if your program executes for even a modest time before experiencing the problem. The third is that it cannot replicate or replace many features that debuggers offer.
Another approach is use a debugger–a tool specifically designed to aid programmers in the debugging process. One widely used debugger is gdb, which we cover in detail in Section 37.2.
Form a Hypothesis
The characteristics of a good hypothesis:
- It is testable.
- It is actionable.
Test the Hypothesis
Testing our hypothesis may proceed in a variety of ways:
- Constructing test cases.
- Inspecting the internal state of the program.
- Adding asserts.
- Code inspection.
The decision to throw away large portions of code and redo them from scratch is not one to be taken lightly, nor an easy one to make. In making such a decision, the programmer should be wary of The Poker Player’s Fallacy–the temptation to make a decision based on prior investments rather than future outcomes.
If instead of accepting your hypothesis, you find that you must reject it, do not despair.