cwash into software » Software Engineering

Transactions, Part 1

Chris Wash — Thu, 08 Jul 2010 14:46:54 +0000

Preface

This work came out of an lack of consistent, cohesive documentation for beginners on transactions. Similar material does exist, but much of it suffers from either being hard to find, scattered and spotty, or not written at a basic enough level where prerequisite knowledge is not required in order to understand the material covered. Over the course of this series of articles, we’ll talk about what concepts you need to know in order to effectively and correctly make use of transactions in your applications. This series of articles is written as a set of core concepts to understanding the basics of transactions, and is intended for junior to intermediate level developers, but my hope is that others will benefit from this series as well. As always, feedback is greatly appreciated.

What is a transaction?

Etymology

The word transaction comes from the Latin word transactionem meaning “an agreement, accomplishment,” which itself comes from the past participle of the verb transigere, transactus, meaning “drove or carried through.”

Transactions defined

The word’s definition is typically related to business or economics, meaning a single business deal, or an exchange of goods between two parties. Most introductions to transactions use an example of debiting one bank account and crediting another. When we remove funds from one account, we need to make sure that they are deposited into the other. If something fails (sufficient funds do not exist in the account being debited, or the account being credited has been closed) then both accounts need to be returned to their original state before the exchange began. This is actually a very good example, but how does that apply to software?

Let’s back up for a second. If we focus on what’s going on, we begin to see there are two distinct activities in play:

Grouping a set of multiple changes together that all have been applied successfully, or, in the event of a single failure, that all need to be undone.
Managing (or coordinating) the execution of these changes, when their interactions might interfere with one another.

Examples of Transactional Systems

Like many other terms in the software lexicon such as “class”, “object”, “system” and “code”, the definition of the word “transaction” suffers a bit from the word’s inherent over-abstraction; it is widely applicable to a great many circumstances and situations. But once understood, is easy to spot scenarios where the pattern is used and applied. Transactions are important in software when updating shared resources, like a cache, database, or message queue, but also come into play when reading – especially when access must be coordinated across many separate but concurrent workers reading shared data. We’ll talk more about this in depth later. Some examples of systems that heavily utilize software transactions are:

eCommerce (Shopping-Cart, Billing, Banking) Applications
Relational Databases
Caching Systems
Messaging Systems (Message-Oriented Middleware)
Source Code Management Systems

As you can see, not only are these fundamental building blocks of business software, but also modern distributed systems. Transactions are prevalent in the frameworks and tools we use to build, tune, and scale the software we write.

Specifying Transactions

ACID properties

A technological definition of the word transaction involves specification of qualities (or “properties”) that must be enforced in order for the transaction to be processed successfully. These properties, the ACID properties as they are known, grow out of our business concept of a transaction (an exchange between two parties). They exist as conventions that specify or constrain the software that’s responsible for processing transactions. Enforcing all four of these properties guarantees the transaction is reliable and therefore viable candidate for successful completion/processing.

Atomic – When we say transactions need to be atomic, we don’t mean that they need to be small, though often they are. What we mean is that you can’t subdivide a transaction any further – the group of changes must be viewed as a single set. When applying that set of changes, it’s “all or nothing” – that is, each and every individual change processes successfully, or else the entire group is undone completely. There are no “partial” transactions; we can’t subdivide them further.
Consistent – If a system successfully processes a transaction, then all resources involved in the transaction must be consistent afterwords – whether the transaction processed successfully or not. For example, if one account is debited and another is credited, the balance of the debited account must be equal to the difference of the original balance and the price of the transaction (and vice-versa for the credited account). Transactions that do not complete successfully should not have any side-effects to either party involved in the transaction.
Independent – Transactions should not interfere with one another, even though they may involve shared resources. For example, one transaction should not be able to see the work another transaction is currently performing until that transaction has completed. They should be independent or isolated. Typically, transaction isolation becomes an issue as transactions being processed become larger: that is, the longer they live or the more resources that they enlist. Isolation also has performance implications as the amount of load or concurrency increases. Isolation mechanisms expose either locking or multi-version concurrency control (MVCC, “diff-ing” or delta-management). We will discuss these later when we talk about concurrency control and isolation.
Durable – Systems processing transactions must record successful transactions so that a history of the transactions processed is not lost. Relevant actions taken are logged for auditing purposes. This audit trail allows the state of the transactional system to be recreated at any point in time if there is a system failure. If the information cannot be recorded in this way, we can not say we have a durable transaction, because we won’t be able to recreate the state of the system if a failure occurs.

In a real world business transaction, these conventions are likely enforced by human beings involved in the transaction. If one of these properties were violated during your checkout at a store in the mall, then either you or the clerk would be pretty upset and likely to call in relevant authorities to sort out the matter. For example, if you bought a shirt but were refused a receipt for the transaction, you would most likely be upset and demand a refund, as the receipt is confirmation of the transaction’s durability.

Why Should I Bother Using Transactions?

As you can see, transactions also provide us with some guarantees about what will happen in the event that something goes wrong during the interactions between shared resources. These guarantees are very useful to us as software designers and developers, but their benefits may not be intuitively obvious until we consider what our jobs would be like without them. Transactions give us with a safety net of assumptions; when something goes wrong, we can lean on the transaction’s guarantee to rollback, or automatically revert all changes from the start of the unit of work. Imagine if we didn’t have this guarantee: we would be forced to write code to cleanup the mess we’ve created, and not only is this code hard to maintain, it’s hard to write and get correct the first time. With transactions, we don’t have to worry about writing extra code to “cleanup” the mess we’ve created, and can instead focus on what the code we’re writing is actually supposed to do: the business logic. When something goes wrong, we can safely assume certain scenarios are out of the question by using transactions. In many circumstances, transactions are key to writing code that is functionally correct, and most of the time make the code cleaner, more robust and maintainable (especially when considering the alternative of coding cleanup logic by hand).

Transactions are fundamentally all about coordinating access to shared resources. A term that can be used interchangeably with the word “transaction” is unit of work. Transactions are important to software systems because they’re how we manage interactions with the different systems involved that have access or expose shared resources to clients.

Typically when we work with transactional systems, as application developers we:

Write code that explicitly utilizes a transactional API, like JDBC, that will contain the semantics for how to handle the processing of the transaction, or…
Configure a transaction management mechanism, like an EJB container, that can do the processing and coordination work on our behalf based on conventions.

The next article in the series will provide examples of both approaches. Either way, the end result provides a way to keep each unit of work’s interactions separate from each other, and guarantee that all changes in the set are applied together or undone completely.

Up Next: Writing Data Reliably

Programming With Transactional Systems
Declarative Transaction Management

Eliminate Branching (IF Statements) to Produce Better Code

Chris Wash — Thu, 07 Jan 2010 21:05:46 +0000

In a recent tech talk, I watched Miško Hevery propose an interesting challenge to his audience: start a toy project and try to write the code with no if-else or switch blocks at all. None at all? Before thinking about how to do this, why would you want to do it in the first place? On the surface, it may seem to the unassuming a bit counter-intuitive. Comparison-based branching is at the heart of programming, and the concept of an if-else or switch block is almost universal, existing in most every programming language ever devised. What’s Hevery’s agenda with this exercise – and what can possibly be gained from trying to write a program without branching entirely?

Branching Considered Harmful?

In 1968, E.W. Diskjstra, famed programmer and author of the shortest-path algorithm, wrote an article to be published in the Communications of the ACM entitled “A Case Against the GOTO Statement” that was later renamed “Goto Statement Considered Harmful” by the editor. The phrase has grown into a meme of its own in the CS community, denoting a practice that has fallen from grace and become widely discouraged.

I bring this up not to point out its cultural signficance, but the core of his argument:

The go to statement as it stands is just too primitive[;] it is too much an invitation to make a mess of one’s program.

Essentially, Hevery is making this very point for the if-else statement, 40 years later, but for very similar reasons. If-else and switch statements easily account for some of the ugliest code that I have seen (and written). Overuse of if-else blocks can lead to the arrow antipattern, a scenario where conditional branches are so deeply nested they become unmaintainable. Too often developers fall into this trap, described aptly by another Dijsktra quote:

The prisoner falls in love with his chains…

Hevery’s point is that code without if (or switch) statements is:

Easier to read and understand
Easier to test
Easier to maintain, extend, etc.

But how do we write code without branching constructs? If we are talking about fixing the arrow antipattern, you may know a time when you avoided this antipattern by using separation of concerns, for example, the MVC pattern instead of a monolithic servlet.

Hevery’s Suggestions

Hevery goes on to explain how good OO design can eliminate the need to explicitly code the most egregious uses of branching, by instead favoring dependency injection and polymorphism. But this technique is only really useful where we can create methods and objects that are cohesive and loosely coupled.

With polymorphism, you dispatch into a method whose binding is not known at compile-time but determined at runtime.

This is the weapon of choice when we are working on code that is highly procedural and there is a bad OO-design smell. But it is far from a one-size-fits-all solution. If we were to transform every branch in our program by instead using polymorphism, it would be a tangle of incohesive objects that would likely be harder to maintain than the original procedural code. Luckily, there are other techniques that are more appropriate in certain circumstances.

Hevery continues: certainly, primitive comparisons are going to be the most difficult to eliminate, but shouldn’t happen very often. So we reluctantly allow these to stick around. Another class of if statements exist to check against nulls. Hevery explains how null pointer dereference checks quickly become pandemic, occurring every time a member of an object is referenced. This should largely be a non-issue, and is only a problem because code is allowing nulls to be returned. The semantics/method contract of most methods can easily be changed to prohibit the return of nulls. (This is only really difficult when recursion is used.) Developers, as it turns out, don’t have to return null values. This is particularly true with collections, and warranted a section in Joshua Bloch’s Effective Java. Instead return an empty collection can be returned, or you can implement the null object pattern (e.g., a no-op logger). Hevery finally rallies for ditching using return codes in favor of throwing relevant exceptions.

Note: I encourage you to watch the talk, as he continues talking about how specifically you can use inheritance and polymorphism to remove branching. The rest of this article will talk about other strategies to remove unnecessary branches.

Remove Copy-Pasted Loops And Conditionals: Introduce Enumerable Operations

Many languages contain an enumerable construct that includes “higher order” looping operations that can eliminate the need for many specialized cases that are usually solved by a foreach loop with an embedded conditional check of some sort. This type of construct can be traced back to Smalltalk, and exists in most modern dynamic languages such as Ruby, Groovy, Python, and provided by a number of Javascript libraries. Getting this to work in Java is a bit of a challenge because there is no mechanism for closures (yet) but some partial solutions do exist (they lack the full power of true closures, but overusing true closures can break down OO-style constructs). For further information, take a look at lambdaj, Adrian Kuhn’s Pimp My Foreach and Google Collections’ Iterators.

Here is an example of the types of things this style of programming can fix.

List numbers = Arrays.asList(0, 1, 2, 3, 4, 5, 6, 7, 8, 9);
List oddNumbers = new ArrayList();

for (Integer check : numbers) {
    if (check % 2 != 0) {
      oddNumbers.add(check);
    }
}

In the case of the Google Collection, they call their enumberable construct a “Iterable” and the closure construct a “Predicates,” we’ll be able to replace the above code with this:

Predicate oddIntegerPredicate = new Predicate() {
    public boolean apply(Integer input) {
        return input % 2 != 0;
    }
};

Iterable oddNumbers = Iterables.filter(numbers, oddIntegerPredicate);

LambdaJ, and Kuhn’s solution work in a similar fashion. While syntactically not as pretty as the dynamic languages, these constructs are functionally equivalent and arguably no more painful than the original code. In many cases of deeply nested for-if-for-ifs, it significantly reduces the cyclomatic complexity). This style of programming, once everyone on the team has become familiar with it, can reduce often copy-and-pasted looping code to one liners, removing some really ugly duplication and the potential for those hard-to-spot copy and paste errors.

Lookout for Train Wrecks: Obey the Law (of Demeter – and other SOLID Principles).

Steve Freeman and Nat Pryce describe “train wreck” code in their new book Growing Object Oriented Software Guided By Tests as chained “getter” method calls, linked together like train cars. The authors call them “wrecks” because the code is essentially an egregious violation of the Law of Demeter. One of Nat’s examples:

employee.getJobDescription().getResponsibilities().add(new ManagementResponsibility(employee.getDepartment()));

The problem is that this style of programming destroys all encapsulation and information hiding concepts in code. Train wrecks can become pandemic as well because instead of using an object-oriented style of programming that passes messages between objects to perform work, we essentially allow objects to know too much about their collaborators and details. Another way this has been said is “talk only to your friends” or “ask, don’t tell.” A client object asks other objects what their state is, and uses an if statement to perform an action, instead of telling the collaborating object what it wants done, and letting that object figure it out. Instead of the train wreck, we should extract the behavior to a method that allows us to tell our collaborator what we want done:

employee.promoteToManager();

Complex conditional “train wrecks” can often be extracted in the same way. Quite often these types of objects are really just procedural code masquerading as a class, and will be in violation of other SOLID-principles such as the Single Responsibility principle. Making this code more “SOLID” will make it easier to test and maintain.

The Bigger Picture: Well Designed, Testable Code

Remember, this exercise in design (by removing ugly branching) is not just for aesthetics. There is a very legit reason to try these techniques: writing testable code. Hevery’s talk (and his other talks) are essentially about how to write code that is testable. Freeman and Pryce’s book is about the very close relationship between testability and OO-design.

If you like these ideas, you might be interested to try the TDD As If You Meant It exercise as well. This is an exercise that will show you how to listen to test code to drive adding features that are driven out by tests into your production codebase by implementing them first in the test class. This is the ultimate way to ensure that your production code contains only what it needs and nothing more, and that code is adequately covered by tests! Remember, the more code you write into production code, the more you’re going to need to maintain it, and the more skeptical you should be about how readable and well designed it is. These techniques and exercises may seem silly on the surface, but if you explore them I am confident they will help you write better code!

The Elements of Reusable Code

Chris Wash — Fri, 24 Jul 2009 19:35:49 +0000

Back in April, there was an interesting article quoting Ron Jeffries et al on InfoQ looking at code reuse from an agile perspective. The conversation steered toward explaining reuse as a concern that is very expensive, looking at it from a top-down, “enterprise”-wide lens.

But is code reuse a black and white issue? My contention is there are varying degrees of reuse that are often neglected on a microscale within an even moderately-sized project. If the code you produce isn’t going to be reused, is it even going to be looked at? These are similar problems with similar solutions.

As far as producing reusable code, I believe there is a medium of full-fledged, framework or API degree of reuse and also a “just enough” degree of reuse. The article touched on a similar concept:

Adam Sroka called this ‘emergent reuse’ and noted that this seemed more efficient than the ‘design for reuse’ approach. A poster named Tim explains it to his business people this way: “The first time you have to pay for it to be coded, the second time you have to pay for it to be reusable, the third time it’s free.”

A higher degree of reuse requires more of an investment in these principles and practices. What’s more, I think the article fails to draw a distinction between the roles (or competing concerns) involved in reuse: not only should we be worried about reusing code we produce, but often we are presented with a very real problem that wasn’t talked about in the article which is how we consume code that we choose to reuse.

A project is itself a consumer of its own code, and many times you’ll find a codebase is not even internally consistent in how it reuses code within itself. Quite often consumer-reuse happens in the form of depending on third party libraries, open source frameworks or APIs. In this sense, reuse becomes an issue of code quality, or more aptly, code cleanliness. It’s hard to come up with solutions to big problems by just thinking about an ideal solution, and even if you are able to do it, odds are your solution will probably be rather academic. It’s easier to make progress on large-scale problems by breaking them down, and making a habit of looking for patterns and ways to combine, compose and chain smaller solutions. Like we do with Agile problem solving approaches. Can’t we do that for the hairy reuse problem?

Decomposing the Problem

Code reuse is a tough problem to nail down, because it’s a high-level view composed of a whole host of smaller concerns. With a nod to separation of concerns, I’ve tried to decompose these competing factors into their own categories and make some suggestions of things to look at if you’d like to have a higher degree of reuse within your team or code base.

Communication Concerns

Communication Saturation

Last night I listened to an Agile Richmond talk by Jeff Sutherland espousing the ideal of communication saturation. Jeff was talking about Scrum, and wanted to draw a strong correlation between this saturation and “hyper-productive” teams. The key, Jeff said, was the ability of the group to recognize what is the next most important thing to work on and come to a collective decision about it. This resonated with my experience with agile. Our group was able to defer stories where we saw reuse was going to be apparent in the future, and this not only did this save us time in the long run, but it made the problem of “discovery” and consumption a no brainer. Everyone on the team knew a component we delivered one release was to be reused in future releases, even if we weren’t involved in developing it. This, coulped with the idea of collective code ownership, opens all the right communication channels for a software development project in my opinion.

Value cross training

Don’t be afraid to ask for help or give it if you can provide help. Educate yourself and teach others. You’ll find that successful teams do this naturally through many different ways. If it seems tough to approach individuals, a better course of action may be to schedule a weekly development meeting to discuss hard problems and solutions to those problems. Record topics that come up regularly. An informal retrospective after a release or milestone can go a long way to correcting the course of the ship.

Design Concerns

Cross cutting concerns

A whole class of reuse problems can be boiled down to reinventing the wheel instead of solving core business problems. If you’re solving a technical problem, is it a cross-cutting concern? If you can identify a problem that has been solved in more than one way, it’s a good idea to pull the team together and talk about a good way to solve it moving forward.

Refactor mercilessly to remove duplication

Refactoring to remove duplication is a skill that takes practice and diligence to do well. But it’s often a quick win to spot and remove duplication; Don’t Repeat Yourself will go a long way to improving the quality of the code your team produces by coercing the design of the code into something more fundamentally sound and amenable to larger scale refactoring later.

Reuse problems are often a manifestation of not having “everything in its right place.” If you always know where to go to get a hammer, then the problem is reduced to that hammer being there when you go to get it. The bigger problem is (rightfully so, now) knowing when you need that a hammer. This is not necessarily going to solve the problem of discoverability or consumption for you (communication is needed for that), but it will help with organization tremendously. Whether or not you achieve an enterprise degree of reuse may be a horse of a different color and have a lot to do with communication, but in the microscale refactoring is the weapon of choice.

Names Matter

If you are able to refactor using composed methods, naming becomes a lot easier. Each method will aim to solve one single “atomic” concern, and larger methods will become more readable and logical as they get shorter and more maintainable.

Technical Concerns

Know the language and learn standard APIs

Unit testing can help you learn a new API fairly quickly and document some hard-earned pieces of knowledge going forward. A handy construct that is available in many languages is a shell interpreter like IRB or BeanShell, which can be invaluable in helping to pick up the syntax and semantics of a language. Wikipedia has a list of interactive programming language shells.

Modularize your code

Modularization is important. Few languages support full modularization OOTB. At the base level, you need to be able to encapsulate your code and also package or namespace it. This is all you really get OOTB in most languages. Ideally, full modularization would provide you with the ability to bundle together code and provide a mechanism to have the code installed or uninstalled dynamically at runtime – in order to achieve this you often have to specify dependencies between modules and declare a version for your bundle. (OSGi aims to do this in Java, and is definitely worth a look).

Modularization is important to code reuse because it presents a barrier to someone else using your code. If I want to use a particular component or method that exists in your library, that means I am fundamentally absorbing the lowest-level dependencies that you have. If your project is not properly modularized, I may have to eat all of your dependencies. The larger your component is, the more complexity any consumer must absorb to reuse your code. Modularization should help with this problem.

SOA was supposed to have solved this problem, right? At least a subset of reuse problems – iteroperability. We’ve found it’s a lot harder in practice than it is in theory. You have to version services, and use some registry to look up and discover a version. Not to mention the governance issues involved. This is why I’m so excited about OSGi. This might be the way to take your microscale reuse to the enterprise, and the model (in practice, and arguably theory) is a good deal simpler than SOA, too.

Documentation Matters?

If you are writing an API or framework, you can’t argue that documentation will be incredibly important to the success of your project. Note, I’m not talking necessarily here about Java-doc style-documentation; rather, I’m talking about providing a would be reuser of your code with enough support to make them successful. Sometimes that means concise, well written and up-to-date API documentation. Quite often this will come in the form of a user or developer’s guide with a more narrative style or structure. FAQs or troubleshooting guides are often very important tools.

Reuse is something that is far easier to say than to do. Doing it requires both good design and very good documentation. Even when we see good design, which is still infrequently, we don’t see the components reused without good documentation.

-D.L. Parnas, Software Aging. Proceedings of the 16th International Conference on Software Engineering, 1994

Going Forward

We all know waste is a thief. Rework is wasteful. Solving the problem of reuse on a grand scale may seem intractible, but I think it’s just another problem, and arguably not even the hardest one we have to contend with in software development. If we can break the reuse problem down from its mighty enterprise-wide incarnation into smaller, more digestable reuse problems, it becomes a lot easier to make progress. And maybe one day we’ll find a way to get to that enterprise-wide reuse that may seem too good to be true at the moment.

In response to Stackoverflow #38/”Quality Doesn’t Matter That Much” — Jeff and Joel

Chris Wash — Sun, 01 Feb 2009 01:16:05 +0000

Update: Robert Martin is scheduled to appear on the Feb 10 episode of the SO Podcast. Should be interesting to see where things go. Also, Jay Fields has weighed in on the topic.

Update #2: Listened to the new SO podcast and am working on a short followup post. I just met another local boy and kindred spirit in Justin Etheredge who has also had a few things to say about this whole debacle.

I wanted to add my two cents to this philosophical, in my view very important, but not very pragmatic debate. For the uninitiated, the argument begins with the Stackoverflow Podcast Episode #38 which is a discussion between Joel Spolsky and Jeff Atwood. They discussed, among many other topics, some of “UncleBob” Martin’s recent material found in his book Clean Code, (actually, Spolsky cited Martin’s appearance on Hanselminutes as the spark to his comments), which Martin responded to with a number of tweets and a blog post.

Jared Richardson introduced me to the idea of technical debt, (term was coined, like so many other terms, by Ward Cunningham) which I think is a pragmatic way to cut through the philosophical differences in this argument. But in the interest of carrying on the debate, and because it’s the weekend, I’ve decided to (perhaps) add some fuel to the fire. Please don’t read this at work.

The gist of the argument is about how much emphasis you should focus on robustness/code hygiene/maintainability in your code, even if it is unclear there is much business value in doing so. Essentially, if we consider impacts to budgets, deadlines, etc., and even in terms of its impact to your code (see Antipattern: “Yet Another Useless Layer”) — is it worth it? It’s very much a discussion of what software quality is and why is it important, which raises questions like:

Are robustness and business value mutually exclusive?
Does quality require rigor and conceptual cleanliness?
Is code hygiene a purely academic pursuit, or is there intrinsic value in it?

These are interesting questions. Atwood and Spolsky take what they feel is a more pragmatic position; Atwood explaining quality in terms of a Frank Zappa quote [paraphrasing] “Nobody gives a crap if we’re great musicians,” meaning that what matters is ultimately what is delivered to the customer. He explains further that “Quality is just another axis” — another set of competing concerns for your development time and interest. Spolsky’s comments constitute a self-professed rant that there is a thread of zealotry in the OO community around TDD. It reminded me a lot of what Cedric Beust has argued in reference to what Jim Coplien has debated with Martin himself. While Spolksy could be playing devil’s advocate, or could have been looking to start a “blog fight” (already popping up in comment threads) I think he could have gone about it in a more pragmatic way. While Beust does paint Martin as a zealot as well, at least his main point didn’t portray unit testing as just short of completely useless; rather, he is one of the creators of the popular TestNG framework:

“Tests first” or “tests last” is unimportant as long as there are tests.

I might be wrong, but I even remember reading a section of Clean Code in Borders last week and essentially reading Beust’s point in Martin’s book. It stuck out in my mind after watching his debate with Coplian. But I digress.

Before going further, I wanted to point out how much this debate reminds me of From pirate dwarves to ninja elves…, which I believe is a great way to classify personality types — especially those of people writing code. We know for sure that UncleBob falls into the Ninja-Elf quadrant. He thinks practicing TDD is a professional responsibilty and lays out his own definition for TDD in Clean Code. I think this is the very definition of a Ninja-Elf. My personal leanings are toward that quadrant as well. When you are innundated with Dijsktra, the ulitmate Ninja-Elf, in school, the following quote really sticks with you:

I mean, if 10 years from now, when you are doing something quick and dirty, you suddenly visualize that I am looking over your shoulders and say to yourself “Dijkstra would not have liked this”, well, that would be enough immortality for me.

But there are people whose opinions I respect with Dwarf-Pirate leanings, too. Obviously Atwood and Spolsky lean in this direction (though they may not fall directly into that quadrant). I’ve heard some others that I respect espouse the same opinion as Spolsky; Gavin King, for example, I’ve heard him say, “The types of bugs that I introduce usually aren’t caught by a unit test.” He wasn’t as over the top in his argument, he simply explained that he prefers integration tests. And I don’t think you’d hear any of the people mentioned in this post say that they never write any tests. For a great argument from the ultimate Pirate-Dwarf see Cal Henderson’s normalized data is for sissies (slide #27); it is pretty entertaining and compelling when scalability is of utmost concern.

I think Atwood’s argument ultimately came down to championing the idea of continuous production; that being able to respond to change or bugs found and having a quick no-brainer deployment process is the Zen-like state you’d like to get to. But the idea of getting there is that it comes out of some notion of continuous integration, which is predicated on the idea of testing. Henderson himself makes this very argument (I believe in this talk, but perhaps in another). You need to trust your team enough to get changes and fixes out to address problems quickly, and be smart about what you spend time on. Testing is great and if you can find people that can test their code well and be productive, that’s awesome. But Henderson’s point is those people are extremely hard to find and retain.

All this goes back to the old question, is producing code an artistic or engineering practice? (Interesting that Atwood supported his comment with a Frank Zappa quote about art.) Obviously you need a talented and cohesive team to be able to make it work either way. I think the answer is it has to be a little bit of both. You’ll need some measure of creativity, some degree of engineering aptitude, also strong analytical skills, and to succeed in a business environment, some acumen in decision making and managing (at the very least your own time).

The fact is, though, that we’ve come leaps and bounds in terms of being able to efficiently produce these tests, and balance their production and maintenance out with constraints, business expectations, budgets and deadlines. And it can fuel your creative side, too.

As for professional responsibilities, I definitely wouldn’t recommend trying to write code without at least some tests if you expect anyone else to have to maintain your code. If you’ve got a close knit team and an overwhelmingly urgent need to deliver untested code (and are of the Pirate-Dwarf persuasion) then go for it. But that being said, I’d say I agree with Martin and Jared Richardson that argue it’s a professional responsibility to not take on the technical debt of untested code. Of course, if tests mean you can get maintainability (and other -ilities) along the way then I think that you’re well on your way to a quality product.

One final way to look at this: would you say it’s better for your team to take the view that code is innocent until proven guilty, or guilty until proven innocent? In the interest of skepticism, considering the teams that most developers “in the trenches” have to work on, and how far we’ve come in making testing easier and more efficient, I’d need a very strong team to make the assumption that all they code they write is innocent. We all make mistakes. I believe this is also brought up in Clean Code.

Ultimately the message Atwood and Spolsky send in their Stackoverflow podcast is that unit testing and efforts to support it are not a silver bullet. Personal slights aside (“I don’t think these people write very much code.”) they have a point — and Martin agrees in his blog post. But I think it took them a long time to make that point which is certainly no great revelation. In the end, I did not feel they explained a whole lot about what the right approach to quality is, but communicated their distaste for an attempt (especially if accepted dogmatically) that aims to do so. I can relate to this, and agree you must treat what you read, hear and are taught with a healthy dose of skepticism. Martin’s ideas may seem dogmatic on the surface but I don’t think it’s fair to paint him in that light. If you really read what he writes, I think he realizes that he is a voice many people are listening to, and works hard to say what he thinks is the right. Martin defends a style of development that favors more rigor and discipline than many others feel is necessary, but in an industry that can be categorized by its lack of discipline, I can’t do much but respect him for that.

On Software Quality

Chris Wash — Wed, 14 Jan 2009 02:27:04 +0000

Dwight D. Eisenhower said “In preparing for battle I have always found that plans are useless, but planning is indispensable.” After reading Jason Yip’s No matter how many times you say it, we still don’t need a QA on the team, I started thinking about how similar this is to the correlation between (software) quality and exploring what software quality is. In the spirit of Jason’s article, thinking about how you “get quality right” will have more positive results for the sake of quality than searching out defects. So the question stands, how do you get quality right? I’m not sure there’s a cut and dry answer, but I have a feeling it’s in the details. I know it sounds anecdotal, but quality isn’t easy. It requires a lot of effort applied constantly. It requires reward and incentive, not punishment and fear. It’s caring about small things, and shifting how your team thinks, communicates and works together.

Jason wisely points out, “Everyone has a responsibility for quality.” But how are you ever to translate this to your team that plays the blame game, avoids at all real issues of quality, or blindly assumes that the “QA” (sic) will find all the product defects? How do you start this discussion, much less keep it going constantly?

Do you simply say, “Quality is your responsibility”? Or hang a sign on the wall to the same effect? I wouldn’t. The very idea of it undermines the ideal you’re trying to espouse: taking personal responsibility. It does nothing to encourage people to care about quality, outside of fear of losing your respect, maybe.

It’s leading by example. If someone finds a defect in my work, or takes issue with something I’m responsible for, I congratulate them with a “Good catch – thanks!” It may sound silly. It may even sound sissy. That’s natural. After all, we’re brought up, educated, cultivated with instincts to defend our own work. I wouldn’t be the least bit surprised to learn the average adult brain is predisposed to think this way due to the way its been “wired” through learning, and even, more philosophically, through our own sense of ego and intellectual self-worth. But think about psychologically what “good catch” does instead:

It acknowledges your teammate’s effort, and encourages that person to continue to put forth the same effort. Or more.
It encourages me to be more diligent in the future, even if I don’t agree 100%.
Most importantly, it tells my teammate that I care about quality, too.

Yet, all of this assumes people on your team desire to produce quality software. Somehow I had to tie this into Programming: Love It or Leave It, right? More rambling to come on this topic later.

Must Haves/References For Modern Java EE Developers

Chris Wash — Sat, 29 Nov 2008 01:10:54 +0000

I’ve been doing a lot of reading lately and have been meaning to plug some of my favorite reads, and one of the things that I’ve been trying to read with an eye toward is for converting those that have been stuck on “behind-the-curve” projects to the new way of thinking and doing things. As such, and just in time for the weekend, I’ve compiled a list of my favorites with an eye to the “movers and shakers” that have driven innovation in the Enterprise Java world recently. Note that even though some of these links are specific to certain frameworks/technologies, there is a common thread throughout most of these that have a bigger focus on the best way to solve problems using Java 5/EE 5 constructs and concepts.

A book that should be on any Java developer’s shelf is Effective Java (2nd Edition). Make sure you get the second edition, for all of the Java 5 changes. This talk by Joshua should whet your appetite. (His talk on good API design is also very good.) You should also take a look at Bob Lee’s talk on Guice. Guice’s philosophy and focus on typesafety has spilled over into Web Beans (and newer versions of Seam) and its annotation based approach has heavily influenced newer Spring features. I especially like his points on type safety around the 11-minute mark. The walkthrough at the end of the talk from “old way” to “new way” is great for people who haven’t really gotten away from Factories, Delegates, and Service Locators yet.

The Guice video mentions that a common thread in modern Java frameworks is a focus on testability. Testing is a very important concept that nearly all new frameworks have embraced, but you still need know-how to be successful with testing. I’ve come across some great books recently in the automated (developer) testing arena. I’ve found the most thorough book on the topic out there is Lasse Koskela’s Test Driven. The ideas and topics discussed in the book are really language independent, and work equally well with other languages. To cut your teeth on TestNG, which the Seam documentation turned me onto, I recommend the TestNG developers’ book Next Generation Java Testing: TestNG and Advanced Concepts. Cedric Beust, also the author of EJBGen, and Hani Suleiman, author of the Bile Blog, produced a book that was a suprisingly good read. It’s packed with a lot of knowledge and some interesting takes on EE concepts. Newer versions of JUnit also include many of these features and concepts, so you’ll get the underpinnings of these concepts in this book. This video gives you a basis to TestNG’s approach.

As one of the growing number of developers who have an application in the Seam In Production page, there are a few Seam resources I have to share. Even if you don’t get into Seam, it should be on your radar because a lot of the core concepts around conversations have evolved into Web Beans, and are found other frameworks like Apache Orchestra and Spring WebFlow. Pete Muir did a Webcast that’s a great introduction to Seam 2.

Author of one of my favorite Seam blogs, Jacob Orshalick, has created a DZone RefCard for Seam 2.1 Core. It’s right off the presses, being published this week, but is a great quick-capsule review of what Seam can do for you, and nice to have on your desk if you’re writing Seam. If you need more in-depth descriptions of what’s on the RefCard, Jacob also coauthored the latest edition of the great Seam reference Seam Framework: Experience the Evolution of Java EE.
Dan Allen’s Seam in Action is another excellent resource. Of course, the awesome Seam Docs and Forums always come in handy. You should also RSS in.relation.to – the Hibernate/Seam developers’ group blog.

I know I’m leaving some great things out. What resources do you think someone who is used to J2EE should take a look at to get them “up to speed” with the state of the art?

How I Escape the “Reuse Trap”

Chris Wash — Sat, 02 Aug 2008 16:25:57 +0000

If you’ve checked out my blogroll, you’ll notice I have a link to Basil Vandegriend’s “Professional Software Development” blog. I like a lot of the articles he has written and to a large extent the subject matter I envision for this blog frequently crisscrosses with Basil’s.

While I was reading “The Reuse Trap In Software Design” I found myself thinking,”Me too! Me too!” like a giddy kid on the playground that’s found a new pal with a mutual interest in “pet snakes and/or tarantulas.” As with many of the problems Basil writes about in his blog, I experienced the same problem, investigated it and found the same root causes, and came to many of the same conclusions as those outlined. He describes the “reuse trap” as:

…[A] term I coined to describe the situation when one becomes stuck trying to design new functionality while simultaneously attempting to reuse existing code that needs some modifications.

He then goes on to describe why it’s a tough problem for n00bs and outlines a strategy he uses to get out of the trap–one that’s best described as a two-step process: copy-paste to reuse/refactoring to remove duplication. I encourage you to read his article if you’re not familiar with the terms, or how this strategy can solve the problem described above.

My preferred technique for mitigating the problems involved isn’t copy-paste reuse/refactoring; it’s stubbing the calls to the code-to-be-reused (C2BR) that needs modification. If I can fake out the behavior I need, it lets me focus on the nature of the dependency that exists between the two objects/components. When I go to modify the C2BR, my inputs and outputs (or other consequences that occur as a result of a call to the C2BR) translate directly into a test that I can use to drive the changes I make to that code. It also ensures that the coupling that exists between the objects is loose and legit.

The only thing to watch out for is that care must be taken to then remove the (small) duplication of the stubbed out inputs/outputs and actually wire the two pieces together when finished with the reused code. I usually do that by throwing a TODO:WIRE marker in when I stub something.

I’ve found this is the approach I take more often, though I do take the copy-paste reuse/refactoring route from time to time. This stub/rewire approach is a bit more advanced, but it’s how I envisioned solving the “reuse trap” when, as Basil puts it, “I became aware of it” and all that it entailed.

Do you ever come across this problem? How do you get out of this “trap” or avoid being caught up in it in the first place?

Continuous Integration Dissected

Chris Wash — Thu, 13 Mar 2008 16:29:06 +0000

Setting the Record Straight

A lot gets written about Continuous Integration, particularly on which is the best visual cue to let you know your build is broken or that a test is failing – lava lamps, Beta Brights, Ambient Orbs, and some even suggest traffic lights. But aside from this extraneous (at least to business) nerd-banter, a lot of what I find written about the actual topic of CI is fluffy, ivory tower, or pie-in-the-sky jibber-jabber that leaves out important parts of the big picture or confuses people more than it helps. In hopes of clearing up confusion on what exactly CI is and how it’s supposed to work, I’m ripping out a description that I wrote for a client proposal recently (so my apologizes for the dry-tone). I hope sheds some light on the true nature of CI, why it’s important and how to implement it from a birds-eye point of view.

Continuous Integration Dissected

Any large scale development project needs an automated, repeatable build process. Following best practices while developing a build process properly separates environment-specific configuration concerns from the codebase. This allows new environments to be created quickly and easily by simply overriding any environment-specific configuration values when first executing the build process. Whatever build tool is being used, builds should share a common, consistent process and interface. A consistent, repeatable build will know all of its dependencies and the goal is to be able to build any given module anywhere, independently, at any time.

Automated, repeatable build processes typically begin by obtaining dependencies (which can be specified using a dependency management tool) and a specific working-copy of the codebase (“checking out”) from a SCM system like CVS or Subversion. It is important to note that this codebase includes any code that is responsible for performing automated testing in addition to source code and configuration (and possibly other source-like artifacts).

Once the checkout has completed, the process will compile code and run automated unit test suites for each module in the system. At this point, all automated unit tests should pass, and custom development can begin. Any changes to code must be adequately covered by unit tests (either by changing existing tests or creating new ones), must fully compile without any errors and pass all automated unit test suites before being committed to the repository. The practice of always keeping the code committed to the SCM repository in this state (no compilation or unit test errors) is known as Continuous Integration and ensures that new development is safe to proceed at any point without fear of integration errors.

Subversion (and CVS) support concurrent development by following a Copy-Edit-Merge paradigm; any contention over files is usually caught when a developer tries to commit their changes and notices the underlying files have changed since they obtained their copy. In many cases, Subversion is capable of performing a merge automatically, if there was no contention over the same piece of a file. Sometimes, however, a manual merge will be required. Merging becomes more painful as the number of differences in the conflicting files increase. A good rule of thumb is that every developer should commit their changes at least daily.

Designating a single machine as a Continuous Integration (CI) environment provides many added benefits to a large scale development project. There are many operations which are good candidates to have run “continuously” but quite often are too expensive for developers to perform before every commit. Examples include executing automated in-browser system tests (which, if maintained over multiple releases, can serve as a “mini” regression test suite), performance tests/profiling, producing test metrics, generating documentation, etc. CI servers are an ideal place to schedule these processes to occur in an automated fashion.

Bugs, Detectives, and Test Automation

Chris Wash — Tue, 11 Mar 2008 21:08:33 +0000

You’ve done a thorough code review with your team and your code has come out with a few nice improvements. Good. You know that many a trustworthy source have speculated code reviews find as many as two times the number of defects that testing does. So you might suppose going to the trouble of creating automated tests, simply for your own devices, might be a bit overkill. As it turns out, it is worth it, and it’s going to save you a lot of time in the long run. Why is it so valuable?

Indulge me in a brief aside… I saw Full Metal Jacket recently on cable for the up-teenth time; the first act of the film, the part that is set on Paris Island, is hard for me to flip away from. I usually end up catching at least the first quarter of the movie. If you’ve seen it, you’ll probably remember the part where the main characters recite, as new recruits, the Marine Corps prayer – it ends with “Without me, my rifle is useless; without my rifle, I am useless.” I think the prayer metaphorically makes sense if developers were reciting it and replaced the rifles with their automated tests.

To figure out why, let’s answer the burning questions that first come to mind:

Why should a developer care about doing their own testing?

It alerts you of problems (duh).
You’ll have a check in place in case a bug ever comes back to life.
It gives you the opportunity to fix things before anyone else finds out about them.
It’s the best way to demonstrate that your code meets its requirements!

Why don’t you want testers (or worse, end users) to find your bugs?

First of all, let’s be honest, this is an ideal to strive toward, even though actually achieving 100% test coverage and catching every possible bug is a definite pipe-dream. But we have to keep in mind that bugs discovered by people other than a developer cause a lot of speculation, confusion, and in general, overhead work for everyone – time that could be better spent doing more productive things. We’d like nothing better than to hand testers (and ultimately users) squeaky-clean code. What happens if we don’t? (and you won’t!)

Imagine taking a class of third graders outside to find an insect in the “wild” and learn about (not burn) it with a magnifying glass. You’re even able to get everyone to concentrate long enough to take a peek at a would-be critter. The first candidate you find turns out to be a piece of bark that looks like a beetle. As you’re searching for another, two of your students get in a fight over claim to a partially decomposed boot they find. Another gets a nose bleed. When you finally manage to find another potential insect, you pass it over, mistaking the critter for a leaf. Little do you know it is actually a very well camouflaged member of the family Phyllidae. By this time a child has fallen face first into a mud puddle and another has peed in his pants (which, one would think, should start to attract some insects).

Imagine that you do manage to find an insect in the wild; how would you get the whole class to observe it properly? There’s just one magnifying glass. The moment you get it under the glass, it darts away. It’s a struggle to keep track of. Not everyone is going to have a good viewing angle. If it disappears, then it might just as well be dead to you (did you hear that tree fall in the woods?).

Enough already with the horrible analogy? Okay, okay. The comparison of users to third graders in our thought experiment was purely coincidental – but you get the point. Having a group of people focus on a moving bug in “the wild” is tough! It’d be much easier to bring a captured critter into a controlled class environment to observe – our analog of a bug that we have caught with nifty our automated test net.

Bug Detective?

Given the fact that a tester’s job is to find defects, odds are sometimes you’ll get a defect report filed that looks like it’s written in cuneiform and requires a wizard’s hat, cauldron and lizard eyeballs to reproduce it. Sometimes a tester might not be giving you a legit bug – sometimes. There are a few scenarios that come to mind:

A requirement is incomplete, misunderstood, miscommunicated or changed.
An external system was down.
An admin fat fingered something.

This comes with the territory and you’re just going to have to suck it up. You’re ultimately the gumshoe on the case. A good way to think of it is that testers and users are like the police of our system – they’re responsible for being the first on the scene, drawing chalk outlines, “booking” the suspects we arrest, etc. What the tester is explaining is probably going to confuse you at least a few times before it makes sense. If you have good testing support, this might not be that painful. But you could have the Super Troopers on the case, and each clue that comes in is going to waste more and more of your time. And heaven forbid an end user finds this problem, and then tries to communicate this to a support rep – your job then becomes ten times harder because you have to decode 2-3 levels of miscommunication that’s occurred. Sometimes no good comes of all this, sometimes you find and squash a bug. In cases where you don’t know what to do, write some tests – ask what the outcome should be, let the answers create more questions and more tests. And be sure to hold onto them!

Whatever the outcome, because good developers are lazy, we want more than anything else to prevent this from ever happening again. Worse case, you’ve spent a lot of time addressing something and should have at least a documented set of work you’ve done and decisions made. Make sure others know about this. Best case, you are able to add an automated test so you don’t have to think about this ever again (or tell anyone about it – you’ll know the moment when something goes wrong).

Conclusion

So let’s recap –

Bugs that slip to production are costly!
Automated tests help us exercise code before and after it goes into production.
If we work on a defect, our automated test net can help us catch a bug, or examine something that looks like a bug more closely.
Whenever we work on a bug (even if it’s something that isn’t truly a bug, but is being tracked as one) we should do our best to produce an automated test so we don’t have to reinvestigate in the future. A test beats an email- or source control commit-trail any day!

cwash into software » Software Engineering

Transactions, Part 1

Preface

What is a transaction?

Etymology

Transactions defined

Examples of Transactional Systems

Specifying Transactions

ACID properties

Why Should I Bother Using Transactions?

Up Next: Writing Data Reliably

Further Reading

More Related Content

Eliminate Branching (IF Statements) to Produce Better Code

Branching Considered Harmful?

Hevery’s Suggestions

Remove Copy-Pasted Loops And Conditionals: Introduce Enumerable Operations

Lookout for Train Wrecks: Obey the Law (of Demeter – and other SOLID Principles).

The Bigger Picture: Well Designed, Testable Code

More Related Content

The Elements of Reusable Code

Decomposing the Problem

Communication Concerns

Communication Saturation

Value cross training

Design Concerns

Cross cutting concerns

Refactor mercilessly to remove duplication

Names Matter

Technical Concerns

Know the language and learn standard APIs

Modularize your code

Documentation Matters?

Going Forward

More Related Content

In response to Stackoverflow #38/”Quality Doesn’t Matter That Much” — Jeff and Joel

More Related Content

On Software Quality

More Related Content

Must Haves/References For Modern Java EE Developers

More Related Content

How I Escape the “Reuse Trap”

More Related Content

Continuous Integration Dissected

Setting the Record Straight

Continuous Integration Dissected

More Related Content

Bugs, Detectives, and Test Automation

Why should a developer care about doing their own testing?

Why don’t you want testers (or worse, end users) to find your bugs?

Bug Detective?

Conclusion

More Related Content