cwash into software » development practices

In response to Stackoverflow #38/”Quality Doesn’t Matter That Much” — Jeff and Joel

Chris Wash — Sun, 01 Feb 2009 01:16:05 +0000

Update: Robert Martin is scheduled to appear on the Feb 10 episode of the SO Podcast. Should be interesting to see where things go. Also, Jay Fields has weighed in on the topic.

Update #2: Listened to the new SO podcast and am working on a short followup post. I just met another local boy and kindred spirit in Justin Etheredge who has also had a few things to say about this whole debacle.

I wanted to add my two cents to this philosophical, in my view very important, but not very pragmatic debate. For the uninitiated, the argument begins with the Stackoverflow Podcast Episode #38 which is a discussion between Joel Spolsky and Jeff Atwood. They discussed, among many other topics, some of “UncleBob” Martin’s recent material found in his book Clean Code, (actually, Spolsky cited Martin’s appearance on Hanselminutes as the spark to his comments), which Martin responded to with a number of tweets and a blog post.

Jared Richardson introduced me to the idea of technical debt, (term was coined, like so many other terms, by Ward Cunningham) which I think is a pragmatic way to cut through the philosophical differences in this argument. But in the interest of carrying on the debate, and because it’s the weekend, I’ve decided to (perhaps) add some fuel to the fire. Please don’t read this at work.

The gist of the argument is about how much emphasis you should focus on robustness/code hygiene/maintainability in your code, even if it is unclear there is much business value in doing so. Essentially, if we consider impacts to budgets, deadlines, etc., and even in terms of its impact to your code (see Antipattern: “Yet Another Useless Layer”) — is it worth it? It’s very much a discussion of what software quality is and why is it important, which raises questions like:

Are robustness and business value mutually exclusive?
Does quality require rigor and conceptual cleanliness?
Is code hygiene a purely academic pursuit, or is there intrinsic value in it?

These are interesting questions. Atwood and Spolsky take what they feel is a more pragmatic position; Atwood explaining quality in terms of a Frank Zappa quote [paraphrasing] “Nobody gives a crap if we’re great musicians,” meaning that what matters is ultimately what is delivered to the customer. He explains further that “Quality is just another axis” — another set of competing concerns for your development time and interest. Spolsky’s comments constitute a self-professed rant that there is a thread of zealotry in the OO community around TDD. It reminded me a lot of what Cedric Beust has argued in reference to what Jim Coplien has debated with Martin himself. While Spolksy could be playing devil’s advocate, or could have been looking to start a “blog fight” (already popping up in comment threads) I think he could have gone about it in a more pragmatic way. While Beust does paint Martin as a zealot as well, at least his main point didn’t portray unit testing as just short of completely useless; rather, he is one of the creators of the popular TestNG framework:

“Tests first” or “tests last” is unimportant as long as there are tests.

I might be wrong, but I even remember reading a section of Clean Code in Borders last week and essentially reading Beust’s point in Martin’s book. It stuck out in my mind after watching his debate with Coplian. But I digress.

Before going further, I wanted to point out how much this debate reminds me of From pirate dwarves to ninja elves…, which I believe is a great way to classify personality types — especially those of people writing code. We know for sure that UncleBob falls into the Ninja-Elf quadrant. He thinks practicing TDD is a professional responsibilty and lays out his own definition for TDD in Clean Code. I think this is the very definition of a Ninja-Elf. My personal leanings are toward that quadrant as well. When you are innundated with Dijsktra, the ulitmate Ninja-Elf, in school, the following quote really sticks with you:

I mean, if 10 years from now, when you are doing something quick and dirty, you suddenly visualize that I am looking over your shoulders and say to yourself “Dijkstra would not have liked this”, well, that would be enough immortality for me.

But there are people whose opinions I respect with Dwarf-Pirate leanings, too. Obviously Atwood and Spolsky lean in this direction (though they may not fall directly into that quadrant). I’ve heard some others that I respect espouse the same opinion as Spolsky; Gavin King, for example, I’ve heard him say, “The types of bugs that I introduce usually aren’t caught by a unit test.” He wasn’t as over the top in his argument, he simply explained that he prefers integration tests. And I don’t think you’d hear any of the people mentioned in this post say that they never write any tests. For a great argument from the ultimate Pirate-Dwarf see Cal Henderson’s normalized data is for sissies (slide #27); it is pretty entertaining and compelling when scalability is of utmost concern.

I think Atwood’s argument ultimately came down to championing the idea of continuous production; that being able to respond to change or bugs found and having a quick no-brainer deployment process is the Zen-like state you’d like to get to. But the idea of getting there is that it comes out of some notion of continuous integration, which is predicated on the idea of testing. Henderson himself makes this very argument (I believe in this talk, but perhaps in another). You need to trust your team enough to get changes and fixes out to address problems quickly, and be smart about what you spend time on. Testing is great and if you can find people that can test their code well and be productive, that’s awesome. But Henderson’s point is those people are extremely hard to find and retain.

All this goes back to the old question, is producing code an artistic or engineering practice? (Interesting that Atwood supported his comment with a Frank Zappa quote about art.) Obviously you need a talented and cohesive team to be able to make it work either way. I think the answer is it has to be a little bit of both. You’ll need some measure of creativity, some degree of engineering aptitude, also strong analytical skills, and to succeed in a business environment, some acumen in decision making and managing (at the very least your own time).

The fact is, though, that we’ve come leaps and bounds in terms of being able to efficiently produce these tests, and balance their production and maintenance out with constraints, business expectations, budgets and deadlines. And it can fuel your creative side, too.

As for professional responsibilities, I definitely wouldn’t recommend trying to write code without at least some tests if you expect anyone else to have to maintain your code. If you’ve got a close knit team and an overwhelmingly urgent need to deliver untested code (and are of the Pirate-Dwarf persuasion) then go for it. But that being said, I’d say I agree with Martin and Jared Richardson that argue it’s a professional responsibility to not take on the technical debt of untested code. Of course, if tests mean you can get maintainability (and other -ilities) along the way then I think that you’re well on your way to a quality product.

One final way to look at this: would you say it’s better for your team to take the view that code is innocent until proven guilty, or guilty until proven innocent? In the interest of skepticism, considering the teams that most developers “in the trenches” have to work on, and how far we’ve come in making testing easier and more efficient, I’d need a very strong team to make the assumption that all they code they write is innocent. We all make mistakes. I believe this is also brought up in Clean Code.

Ultimately the message Atwood and Spolsky send in their Stackoverflow podcast is that unit testing and efforts to support it are not a silver bullet. Personal slights aside (“I don’t think these people write very much code.”) they have a point — and Martin agrees in his blog post. But I think it took them a long time to make that point which is certainly no great revelation. In the end, I did not feel they explained a whole lot about what the right approach to quality is, but communicated their distaste for an attempt (especially if accepted dogmatically) that aims to do so. I can relate to this, and agree you must treat what you read, hear and are taught with a healthy dose of skepticism. Martin’s ideas may seem dogmatic on the surface but I don’t think it’s fair to paint him in that light. If you really read what he writes, I think he realizes that he is a voice many people are listening to, and works hard to say what he thinks is the right. Martin defends a style of development that favors more rigor and discipline than many others feel is necessary, but in an industry that can be categorized by its lack of discipline, I can’t do much but respect him for that.

Continuous Integration Dissected

Chris Wash — Thu, 13 Mar 2008 16:29:06 +0000

Setting the Record Straight

A lot gets written about Continuous Integration, particularly on which is the best visual cue to let you know your build is broken or that a test is failing – lava lamps, Beta Brights, Ambient Orbs, and some even suggest traffic lights. But aside from this extraneous (at least to business) nerd-banter, a lot of what I find written about the actual topic of CI is fluffy, ivory tower, or pie-in-the-sky jibber-jabber that leaves out important parts of the big picture or confuses people more than it helps. In hopes of clearing up confusion on what exactly CI is and how it’s supposed to work, I’m ripping out a description that I wrote for a client proposal recently (so my apologizes for the dry-tone). I hope sheds some light on the true nature of CI, why it’s important and how to implement it from a birds-eye point of view.

Continuous Integration Dissected

Any large scale development project needs an automated, repeatable build process. Following best practices while developing a build process properly separates environment-specific configuration concerns from the codebase. This allows new environments to be created quickly and easily by simply overriding any environment-specific configuration values when first executing the build process. Whatever build tool is being used, builds should share a common, consistent process and interface. A consistent, repeatable build will know all of its dependencies and the goal is to be able to build any given module anywhere, independently, at any time.

Automated, repeatable build processes typically begin by obtaining dependencies (which can be specified using a dependency management tool) and a specific working-copy of the codebase (“checking out”) from a SCM system like CVS or Subversion. It is important to note that this codebase includes any code that is responsible for performing automated testing in addition to source code and configuration (and possibly other source-like artifacts).

Once the checkout has completed, the process will compile code and run automated unit test suites for each module in the system. At this point, all automated unit tests should pass, and custom development can begin. Any changes to code must be adequately covered by unit tests (either by changing existing tests or creating new ones), must fully compile without any errors and pass all automated unit test suites before being committed to the repository. The practice of always keeping the code committed to the SCM repository in this state (no compilation or unit test errors) is known as Continuous Integration and ensures that new development is safe to proceed at any point without fear of integration errors.

Subversion (and CVS) support concurrent development by following a Copy-Edit-Merge paradigm; any contention over files is usually caught when a developer tries to commit their changes and notices the underlying files have changed since they obtained their copy. In many cases, Subversion is capable of performing a merge automatically, if there was no contention over the same piece of a file. Sometimes, however, a manual merge will be required. Merging becomes more painful as the number of differences in the conflicting files increase. A good rule of thumb is that every developer should commit their changes at least daily.

Designating a single machine as a Continuous Integration (CI) environment provides many added benefits to a large scale development project. There are many operations which are good candidates to have run “continuously” but quite often are too expensive for developers to perform before every commit. Examples include executing automated in-browser system tests (which, if maintained over multiple releases, can serve as a “mini” regression test suite), performance tests/profiling, producing test metrics, generating documentation, etc. CI servers are an ideal place to schedule these processes to occur in an automated fashion.

Bugs, Detectives, and Test Automation

Chris Wash — Tue, 11 Mar 2008 21:08:33 +0000

You’ve done a thorough code review with your team and your code has come out with a few nice improvements. Good. You know that many a trustworthy source have speculated code reviews find as many as two times the number of defects that testing does. So you might suppose going to the trouble of creating automated tests, simply for your own devices, might be a bit overkill. As it turns out, it is worth it, and it’s going to save you a lot of time in the long run. Why is it so valuable?

Indulge me in a brief aside… I saw Full Metal Jacket recently on cable for the up-teenth time; the first act of the film, the part that is set on Paris Island, is hard for me to flip away from. I usually end up catching at least the first quarter of the movie. If you’ve seen it, you’ll probably remember the part where the main characters recite, as new recruits, the Marine Corps prayer – it ends with “Without me, my rifle is useless; without my rifle, I am useless.” I think the prayer metaphorically makes sense if developers were reciting it and replaced the rifles with their automated tests.

To figure out why, let’s answer the burning questions that first come to mind:

Why should a developer care about doing their own testing?

It alerts you of problems (duh).
You’ll have a check in place in case a bug ever comes back to life.
It gives you the opportunity to fix things before anyone else finds out about them.
It’s the best way to demonstrate that your code meets its requirements!

Why don’t you want testers (or worse, end users) to find your bugs?

First of all, let’s be honest, this is an ideal to strive toward, even though actually achieving 100% test coverage and catching every possible bug is a definite pipe-dream. But we have to keep in mind that bugs discovered by people other than a developer cause a lot of speculation, confusion, and in general, overhead work for everyone – time that could be better spent doing more productive things. We’d like nothing better than to hand testers (and ultimately users) squeaky-clean code. What happens if we don’t? (and you won’t!)

Imagine taking a class of third graders outside to find an insect in the “wild” and learn about (not burn) it with a magnifying glass. You’re even able to get everyone to concentrate long enough to take a peek at a would-be critter. The first candidate you find turns out to be a piece of bark that looks like a beetle. As you’re searching for another, two of your students get in a fight over claim to a partially decomposed boot they find. Another gets a nose bleed. When you finally manage to find another potential insect, you pass it over, mistaking the critter for a leaf. Little do you know it is actually a very well camouflaged member of the family Phyllidae. By this time a child has fallen face first into a mud puddle and another has peed in his pants (which, one would think, should start to attract some insects).

Imagine that you do manage to find an insect in the wild; how would you get the whole class to observe it properly? There’s just one magnifying glass. The moment you get it under the glass, it darts away. It’s a struggle to keep track of. Not everyone is going to have a good viewing angle. If it disappears, then it might just as well be dead to you (did you hear that tree fall in the woods?).

Enough already with the horrible analogy? Okay, okay. The comparison of users to third graders in our thought experiment was purely coincidental – but you get the point. Having a group of people focus on a moving bug in “the wild” is tough! It’d be much easier to bring a captured critter into a controlled class environment to observe – our analog of a bug that we have caught with nifty our automated test net.

Bug Detective?

Given the fact that a tester’s job is to find defects, odds are sometimes you’ll get a defect report filed that looks like it’s written in cuneiform and requires a wizard’s hat, cauldron and lizard eyeballs to reproduce it. Sometimes a tester might not be giving you a legit bug – sometimes. There are a few scenarios that come to mind:

A requirement is incomplete, misunderstood, miscommunicated or changed.
An external system was down.
An admin fat fingered something.

This comes with the territory and you’re just going to have to suck it up. You’re ultimately the gumshoe on the case. A good way to think of it is that testers and users are like the police of our system – they’re responsible for being the first on the scene, drawing chalk outlines, “booking” the suspects we arrest, etc. What the tester is explaining is probably going to confuse you at least a few times before it makes sense. If you have good testing support, this might not be that painful. But you could have the Super Troopers on the case, and each clue that comes in is going to waste more and more of your time. And heaven forbid an end user finds this problem, and then tries to communicate this to a support rep – your job then becomes ten times harder because you have to decode 2-3 levels of miscommunication that’s occurred. Sometimes no good comes of all this, sometimes you find and squash a bug. In cases where you don’t know what to do, write some tests – ask what the outcome should be, let the answers create more questions and more tests. And be sure to hold onto them!

Whatever the outcome, because good developers are lazy, we want more than anything else to prevent this from ever happening again. Worse case, you’ve spent a lot of time addressing something and should have at least a documented set of work you’ve done and decisions made. Make sure others know about this. Best case, you are able to add an automated test so you don’t have to think about this ever again (or tell anyone about it – you’ll know the moment when something goes wrong).

Conclusion

So let’s recap –

Bugs that slip to production are costly!
Automated tests help us exercise code before and after it goes into production.
If we work on a defect, our automated test net can help us catch a bug, or examine something that looks like a bug more closely.
Whenever we work on a bug (even if it’s something that isn’t truly a bug, but is being tracked as one) we should do our best to produce an automated test so we don’t have to reinvestigate in the future. A test beats an email- or source control commit-trail any day!

cwash into software » development practices

In response to Stackoverflow #38/”Quality Doesn’t Matter That Much” — Jeff and Joel

More Related Content

Continuous Integration Dissected

Setting the Record Straight

Continuous Integration Dissected

More Related Content

Bugs, Detectives, and Test Automation

Why should a developer care about doing their own testing?

Why don’t you want testers (or worse, end users) to find your bugs?

Bug Detective?

Conclusion

More Related Content