effective software development

Manager's Introduction to TDD video is online

Tue, 2 Dec 2008 15:15:56 -0500

InfoQ made a video of the presentation Karl Scotland and I gave of the Manager's Introduction to Test-Driven Development sesion at Agile2008 in Toronto last August. They have put the video online on their site at this URL: http://www.infoq.com/presentations/TDD-Managers-Nicolette-Scotland. You can download the accompanying materials from here: http://davenicolette.wikispaces.com/TDD+for+Managers. Karl and I hope you will find it useful.

Whose line item is it, anyway?

Tue, 2 Dec 2008 10:59:16 -0500

Not long ago, I wrote that value has to mean business value, and business value has to mean return on investment. I also asserted that Function Points would not be a meaningful way to measure agile software projects, since with adaptive development methods we don't know in advance how many of each type of "function" there will be in the finished product. Since then, I've had cause to reconsider these opinions.

I still think that business value primarily means return on investment, although a business enterprise may value other considerations besides return, such as market share, employee retention, environmental stewardship, community relations, and (in some cases) ethics. I still think that Throughput Accounting represents a simple and meaningful approach to calculating return on investment (even if we have to pretend the project budget is equivalent to the "sales price" of the software). I still think that Earned Business Value (even if it is based on subjective, customer-defined relative weighting of features or stories) offers a realistic and practical measure of the business value delivered by a software project.

What I have learned, however, is that not everyone thinks like a business person. There is another concept of "value" we have to take into consideration in order to satisfy certain stakeholders. Some stakeholders value productivity.

Productivity is a dangerous word. A single-minded focus on productivity can easily lead to the problem of local optimization. If a team or an organization works in specialized silos with hand-offs of interim artifacts via work-in-process queues (the most typical model today), then when each silo concentrates on maximizing its own productivity the inevitable result will be local optimization, with all its detrimental effects on the organization's ability to deliver value to its customers. So, you may well question the wisdom of valuing productivity for its own sake. As is true of so many issues, it's all a question of context.

Given present economic circumstances, a business person will look at the IT portfolio of his/her company and ask, "Which of these initiatives can we discard?" This is because the business person is responsible for revenue generation, cash flow, and profitability. The person in charge of the IT department asks a very different question. He/she asks, "How can I carry out all these initiatives with a smaller staff, lower budget, and shorter timelines?" Why does he/she not consider trimming the portfolio? Because IT management is not responsible for revenue generation, cash flow, and profitability. IT management is responsible for managing a fixed budget and carrying out the tasks that have been assigned. That is how their job performance is measured.

Therefore, the approach we take to quantifying and demonstrating value must depend on the audience who will consume the information. If we are dealing with business management, then the tools I wrote about before will be sound choices. If we are dealing with IT management, then we need something different. We need to be able to show differences in productivity based on which software development methodology is applied. The key to that sort of comparison turns out to be our old friend, Function Points.

A couple of colleagues set me straight about a few things. Most of the people we interact with when trying to describe the value of adaptive methods such as "agile" and "lean" software development are in IT management roles, not business managment roles. They have a cost center mentality, not a profit center mentality. They are measured on how well they execute against the IT portfolio, not on their judgment of which initiatives are worthwhile. Typically, they know exactly where they stand with respect to their budget burn-down for the fiscal year, but they have no idea of the true costs of software development and delivery. The last thing they would relate to is an analysis of return on investment; after all, they live in the world the Standish Group found delivers about $0.59 for every $1.00 spent, on average. It would be too painful for them to know their own numbers, let alone to do a hard-nosed business value calculation based on those numbers. What they do care about is how effectively they can deliver the projects that have been assigned to them. That is, they care about productivity — how to deliver more projects in less time with fewer people and less money.

A colleague of mine, Mark Smith, suggests using Function Points as a common yardstick for measuring productivity. While it is true that agile development projects don't use Function Points during development, it is also true that any finished software product can be measured in terms of Function Points. It's possible to count the Function Points in a product regardless of the methodology that was used to create the software. Furthermore, Function Point counting is an ISO standard, it has an official governing body that certifies Function Point counters, and it has a long history of widespread use in the IT industry. In other words, IT managers already understand Function Points. We can talk to them about "value" in familiar terms; they needn't learn about Throughput Accounting or Earned Business Value just to have a conversation about value.

Depending on your audience, you may want to express the value of agile and lean methods in terms of real business value, or you may want to express it in terms of relative effectiveness for delivering IT projects. For the latter purpose, Function Points are a practical and realistic basis for measurement and comparison.

You could say (hypothetically) that a team using a waterfall process can deliver 10,000 FP in 12 months with a team of 25 at a cost of $2,000,000, while a team using an agile process can deliver 10,000 FP in 6 months with a team of 8 at a cost of $300,000. That sort of comparison would be meaningful to an IT manager.

Refactorings in großen Softwareprojekten

Tue, 2 Dec 2008 09:28:34 -0500

Refactorings in großen Softwareprojekten by Stefan Roock and Martin Lippert is a welcome addition to the literature in the area of scaling agile in large enterprises. It's not a new book — it was published in 2004 — but I've only just discovered it recently, and I wanted to be sure people are aware of it.

Other books on the subject of agile-in-the-large focus on either management and governance issues or on the challenges of agile adoption. This book addresses an important aspect of the application of agile thinking and principles to enterprise IT: Refactoring for incremental improvement of enterprise architecture. The title may be a bit misleading, in that problems of architectural refactoring often affect an organization's technical infrastructure as a whole. As such, they are not really project-related problems; not even for relatively large projects. The authors introduce specific architectural smells and suggest specific refactorings to deal with each smell. They cover issues that are usually (but not always) beyond the scope of individual software development projects.

The authors cover topics not usually discussed in books about agile software development, including database refactoring and refactoring of published APIs. While they make an effort to keep the material at a general level, the authors' deep experience with Java colors the material. For example, they mention cyclic dependencies between packages as a general architectural smell, while it is a problem that almost always occurs in products written in Java. In my view, the Java focus takes nothing away from the general value of the authors' advice. Most of the issues they raise are not language-specific.

The specific refactoring advice is based on fundamental principles of object orientation. This is a strong and proven basis for building robust, reliable, and performant technical infrastructures and complex solutions. The authors describe several key principles such as separation of concerns, don't repeat yourself, and the open-closed principle. I was surprised that they do not mention one of the most important fundamental OO principles: the single responsibility principle. I call SRP "fundamental" because several other commonly-cited principles can be derived from SRP in obvious ways. Thus, including SRP would have kept the list shorter. SRP applies very naturally at the level of enterprise and solution architecture. Just as a class should have a single responsibility, so should a component and so should an architectural layer. Even so, the authors do not overlook anything because SRP is omitted from their list of principles.

I recommend this book for anyone who has an interest in large-scale agile in an enterprise IT context.

Fertig ist fertig

Wed, 3 Dec 2008 14:06:36 -0500

Sometimes, when a concept is described using different words than you usually use, it causes you to look at the concept with a fresh eye. In the Scrum workshop at XP Days Germany, the last column on the task board was labeled fertig, which translates into English as "ready." When I read it, I wondered in which sense the presenters meant it: Ready for testing? Ready for deployment? Ready for actual use? One of the participants asked about it, and the answer was that "ready" means ready for use by the end user.

Of course, that is exactly what it should mean. Unfortunately, a lot of agile teams have fallen into the routine of treating a story as "ready" or "done" as soon as they have taken the work as far as they can before encountering a dependency on something or someone outside the project team. Many development teams in the US treat stories as "ready" when all they really mean is "ready for acceptance testing" or "ready to wait in a work-in-process queue indefinitely before deployment." We've been creating more and more definitions of "done" so that we can ignore organizational constraints that prevent us from taking a story all the way to fertig, and so that we can claim credit for some number of story points greater than zero. We talk about done, done-done, and (yes, I've seen this in writing) done-done-done as if there were really multiple definitions of "done."

What we're doing is surrendering to present-day organizational dysfunction, and pretending done is okay, because it's all the development team can control directly. The effort and time required to correct the underlying organizational dysfunction is too great; the development team must focus on delivering the product. Done-done sure would be nice, but since it's not achievable we'll just redefine done so we can collect our story points and show a non-zero velocity for the iteration. Instead of re-shaping the organization to accommodate agile work, we re-shape ourselves to conform with whatever the present-day realities happen to be. While that may be the course of least resistance in many cases, it doesn't help advance the state of the art of IT.

Hey, no one said it was going to be easy. As Super Chicken said to his sidekick, Fred: "You knew the job was dangerous when you took it, Fred."

Fertig ist fertig: Ready is ready. Anything short of that is...well, it's nicht fertig. When we compromise on this point we're not fooling anyone but ourselves.

On edit: Normally, I leave minor errors alone. After all, it's only a blog. But writing vertig instead of fertig was just too stupid. I had to go back and correct it!

Ich bin ein Hamburger

Tue, 2 Dec 2008 09:22:22 -0500

Just back from Hamburg, where I participated in XP Days Germany for the third consecutive year. As always, it was a highly interactive and enriching experience for all. My day started with an introductory Scrum workshop conducted by Jens Coldewey and Bernd Schiffer. There were a couple of experienced people, but mostly the group consisted of people who had heard positive things about agile development and wanted to get a taste of it. One of the User Stories for the workshop read, "Spaß haben." Personally, I thought that should have been a non-functional requirement; otherwise, once the story was deemed "done" we would have to stop having fun. Fortunately, that didn't happen. We had a lot of fun building a shuttle docking station for a Mars base out of Legos®. I'm not sure participants got a realistic sense of how Scrum works, but at least there was ample Spaß to go around. My perspective may be a bit biased, but I think our team's base was the best.

Alexander Kriegisch's presentation, Agiles Multi-Projektmanagement - weniger ist mehr, dealt with the problem of keeping multiple interrelated projects in sync, whether using a single overarching backlog or separate backlogs per project. He presented techniques based on Critical Chain Project Management and Theory of Constraints with the theme, "less is more."

Joseph Pelrine presented a condensed version of his one-day workshop, Coaching Self-Organizing Teams. I've had the pleasure of attending a couple of longer presentations of this material, and I learn something new every time. This is probably one of the most informative and unique sessions making the rounds of conferences in the past few years. If you get a chance to attend one of these, do yourself a favor and make time for it.

The organizers made sure first-time presenters had an opportunity to run sessions. I think this is a great idea that helps build competence in the community. It would be even better if some of these presenters received a bit of coaching about effective presentation techniques. Some of the presentations, although well-researched and thoroughly prepared, consisted of sleep-inducing "slide shows" in which the speaker merely read bullet points from the presentation slides. The only thing that kept me awake was the sheer effort it took for me to follow what was being said in German. Coming directly from XP Days Benelux, with its strong emphasis on interactive and highly engaging session formats, the contrast was all the more dramatic. This is not a major criticism; it's just a question of experience. These new presenters will improve with practice.

Lasse Koskela and I were scheduled to present Overcoming Resistance to Change, a session we co-developed last year and that we have presented a number of times since. Unfortunately, Lasse was ill and could not attend. I gave the session on my own, and it seemed to go fairly well. As usual, the clock ran out before the material did. There were a lot of questions and discussion afterwards, and the organizers had to shoo us out of the way to make room for the next presenters. It seems change agents in German companies are facing many of the same challenges as those in US companies. Exchanging information and experiences between these two communities could bring benefits to both.

Congratulations to the organizers on preparing another great conference.

XP Days Benelux 2008: One more thing...

Tue, 2 Dec 2008 09:21:15 -0500

Another example of agile thinking in action: The feedback forms for the conference were published last Tuesday; the second working day after the conference ended. What other conference do you know of that turns feedback around that quickly? Kudos to the organizers, once again.

XP Days Benelux 2008: Personal highlights

Mon, 24 Nov 2008 16:11:57 -0500

Agile Metrics

My session on Agile Metrics was well attended and well received. How do I know it was well received? First, no one got up and left in the middle of it. Second, we had a retrospective at the end of each day, remember?. I've never gotten through all the material for this session, and this time was no exception. It isn't only because of time constraints. It's mainly because the presentation generates a lot of questions and discussion. I like it that way, because I'm more of a down-to-earth person than a theorist. Given some basic information, participants always have interesting questions about how to apply metrics in their own situations. In this case, there were over 30 participants and quite a bit of good discussion. Many people approached me after the session to ask where they could download the slides and sample spreadsheets. In case you're interested, the answer to that question is here.

Overcoming Resistance to Change

This is a session Lasse Koskela and I developed last year, and that we've presented several times since then. The session explores the types of resistance a change agent may encounter, the forms it may take, the various groups in an organization that may resist certain changes, their reasons and how to discover them (because they don't tell you their reasons, and sometimes they don't understand the real reasons themselves), and specific techniques to alleviate their concerns and encourage their support for the change. We were expecting about 20 people, and we set up the room with tables for five groups. We actually had around 33 participants. The room was a bit crowded, but everyone stayed and everyone participated actively. It seemed to go very well. Several participants told us they got value from the session and they learned some things they could apply. That's just what I always hope for. If you want the materials for this one, they're here.

Interesting conversations

Agile vs. waterfall?

In an earlier post, I mentioned that the participants in the workshop in the session, Seeking to Perceive More Than to Be Perceived, were asked to identify an improvement that could be made to next year's conference and to come up with a plan to achieve it, but I didn't say what improvement they decided on. They thought it would be a good idea to invite a prominent anti-agilist to present the case against agile development.

After the session, several of us talked about the implications of that. I asked pointedly whether they really believed there was a "community" of like-minded individuals out there who self-identified as "waterfallists" and who would think this sort of debate would make any sense. It seemed to me some people were building walls to shut out people who don't think the same way; there was a lot of "we vs. they" thinking and word choices.

I think this is a counterproductive approach because it encourages the myth that there is some sort of "agile religion" that people follow because they "believe" in a rigidly-defined "dogma." If you follow the link in the last sentence, you'll notice a number of rather odd misconceptions about what the blogger thinks agile means. His opinions are not unique. It isn't helpful to encourage or reinforce misconceptions like those. There is no real division between software professionals along this imaginary agile fault line. I'm sure most professionals know our industry has suffered from a number of endemic problems: Long lead times, cost overruns, schedule slippage, features delivered that are never used, and many more. We're all interested in discovering ways to improve things. There's no one who "believes" in agile. There's no one who "believes" in waterfall. A confrontational "agile vs. waterfall" debate would only create needless problems.

Apart from being totally meaningless to begin with, the notion of a conference track to explore weaknesses in the agile approach is not new. One of the "stages" at Agile2008 this year was Questioning Agile. It was far more constructive in spirit and implementation than the idea of inviting a pro-waterfall speaker to an anti-waterfall conference.

Is the profession forking?

After the session, The Invariant Game, presented by Matteo Vaccari, Matteo and Laurent noted that the participants in the workshop found the material very advanced and challenging, while the two of them felt it was rather basic computer science. Some of the participants had commented that going through the exercise of defining invariants didn't help them craft meaningful unit tests. Instead, it just felt like doing the same work twice — once to create an abstract representation of the invariant, and again to write the unit test; and defining the invariant was actually much harder than writing the unit test. So, why not just write the unit test? That's the useful artifact we're trying to get to, isn't it?

All this led Laurent to speculate that the profession may be forking into two species of software developers — one that just assembles building blocks into routine solutions, and one that understands how to apply logic and computer science to complex problems. He wonders whether programmers are losing some of the intellectual rigor that used to be mandatory for success in the field, since solutions to most of the basic programming problems are available in libraries and solution developers can simply call them without understanding the logic behind them.

I commented that it sounded like a classic science fiction theme: A future society whose members have forgotten how to maintain the technology on which they have come to depend for their survival. Laurent said, "Like the Morlocks. Yes." As an American of a certain generation, I was actually thinking of Star Trek. Specifically, the 1968 episode from the original series entitled, Spock's Brain. "Brain and brain! What is brain?" Or: "Invariant and invariant! What is invariant?"

Aikido fun

Consultant Olivier Costa is a student of aikido. He and his sensei, Frank Vanhoeck, gave us a quick and dirty session of aikido exercises after the day's events on Wednesday. It was great. I felt more relaxed and yet more energetic and clear-minded at the same time. And happy. Those are the effects of this sort of practice. And it's absolutely relevant to software development. Maybe it should become a standard expectation of software professionals to begin each day with aikido practice, or tai chi, or something along those lines.

XP Days Benelux 2008: Seeking to Perceive More Than to Be Perceived

Mon, 24 Nov 2008 14:12:02 -0500

The session description">session title comes from Jim and Michele McCarthy's Core Commitments, and so does one of the three techniques Emmanuel Gaillot and Bernard Notarianni presented in this experiential workshop.

Without any explanation of what to expect, the facilitators asked for volunteers to form a circle in the center of the room. Five people did so. The rest of us formed an outer circle around them. We were the observers. Part of the value came from having the different perspectives of participants and observers. The participants were tasked with achieving a goal together: To think of an improvement to make in the conference for next year and to create a plan to achieve it. They worked in four timeboxed "iterations" with a discussion after each one in which the participants spoke first and the observers second. In each iteration, the participants used a different technique to improve the ability of each member to listen intently to his/her teammates.

In the first iteration, the participants were on their own. Working without any predefined process or ground rules, they used most of the available time feeling each other out and figuring out how to interact. In the retrospective, one observer commented that the group had wasted its time, since they came up with no concrete results. I countered, saying they had learned how to talk to each other, and that in itself was a useful outcome.

For the second iteration, the facilitators briefly explained the Investigate Protocol, which is part of the Core Protocol developed by Jim and Michele McCarthy. At first, I didn't see much value in it. You just ask questions that are meant to elicit the other party's ideas. There didn't appear to be much more to it than that. Before long, it became apparent that everyone was asking leading questions or loaded questions. Leading questions are designed to fish for an answer the investigator is looking for. Loaded questions convey the investigator's bias toward a particular answer. Both of these are problems, since they create obstacles to clear communication. It was quite surprising to note how many of our questions are leading or loaded. Many of us tend to ask questions that way because we already have an answer in mind that we think is "correct," and we aren't asking because we want to know the other party's opinion, but only so we can pass judgment. It's very helpful to know a technique that helps us become conscious of this and correct it.

The third iteration involved a technique from the world of theater, Tadashi Suzuki's "Soft Focus." The idea is to make yourself aware of all the people in the room by trying to see all of them at once. You focus on a point in space that causes everyone to be somewhere in your field of view. You can't actually focus sharply on any one person; hence the name. The idea is that you will be more receptive to the other people if you are paying attention to them, so you pay attention to everyone. Simultaneously.

The exercise turned out to be rather funny. One participant, Jamey, did his best to do the technique throughout the iteration. He stared at a point in space while talking and listening to the other participants. He commented that the technique forced him to pay close attention to verbal communication, since he was unable to depend on non-verbal cues as he normally does. But the way he did it had an unusual effect on one of the other participants, Ellen. Jamey did not just focus on a point in space. He bored a hole through the point with his gaze. He was utterly rigid from head to toe. I'm not sure, but I think he didn't blink even once. Ellen grew progressively more fidgety and nervous as she tried to carry on a conversation with Jamey. She asked him several times to stop doing the soft focus thing, but he continued, saying he wanted to see the experiment through to find out what would happen. Ultimately, Ellen said the experience gave her an unnerving sensation of "being ignored and stared at at the same time."

My personal take on this is that "soft focus" might be useful as a way to prepare for a meeting or discussion. It may help clear your mind and prepare your spirit to be receptive and attentive to others. But I don't see using it during a discussion. People prefer you to make direct eye contact when you're speaking to them. When you are listening or speaking to one individual in a group, it's best to make eye contact with that individual. I suspect if we took this technique to an extreme, as Jamey did during the exercise, we may well generate reactions like Ellen's. That would impair clear communication rather than enhancing it.

To go along with being aware of others, the Investigate Protocol helps us become aware of ourselves and how the way we frame questions might interfere with communication. The two ideas are complementary.

The fourth iteration explored the emic approach to learning, elaborated by cultural anthropologist Marvin Harris. Emic is defined in contrast to etic, and both terms are neologisms coined by the linguistic anthropologist Kenneth Pike. If you follow the link on Harris' name, don't expect a simple and concise description. Instead, expect things like "the debate over emics and etics raises a number of fundamental ontological and epistemological issues." It's a good read for an insomniac. Apparently, Pike and Harris spent years debating the finer distinctions between the two terms and the philosophical nuances of the ideas behind them. They are made-up words, and the guy who made them up is free to make them mean whatever he wants them to mean. So what? Fortunately for us, Emmanuel had a simpler explanation.

The idea for the exercise was that we can learn more effectively if we ask someone to teach us to do something they know how to do, rather than just asking them to explain the thing to us. This certainly makes sense in many work-related situations. In the context of the workshop, it didn't work quite so well. Since the group had been tasked with collaborating to identify an improvement to the conference, there wasn't any particular skill that one group member could teach another. But something like this can be very useful in a team coaching situation. If we ask team members to teach us how they work, we can learn a great deal more about the reasons they work that way than we could ever do just by asking them to describe how they work, or even by observing them at work. That can give us a good basis to recommend improvements.

XP Days Benelux 2008: SimProject

Mon, 24 Nov 2008 12:00:05 -0500

SimProject is a prototype application Laurent Bossavit is working on that supports theSystems Thinking tools, Diagram of Effects and Causal Loop Diagrams.

Based on the session description, I was expecting this session to consist of a demonstration of the SimProject software and a discussion of which agile practices can safely be set aside in the beginning of an organizational change. In the event, it turned out to be something much more interesting and useful: A hands-on exploration of DOEs as an aid to understanding complex problems and how causal loops can emerge from them. It really has nothing to do with which agile practices can be set aside initially. The technique is useful for identifying which problem to solve first, or solve next, when the overall performance of a work flow is not as good as we would like it to be. We worked in small groups to diagram a problematic situation; each group was free to choose whatever problem the group members wished.

In complex systems such as teams of humans, a change to any variable may have many effects, some of which are quite subtle and some of which may lead to new emergent behaviors. DOEs are useful to get a sense of what the effects of a change are likely to be. More fundamentally, they are useful to gain an understanding of the current state. Causal loops occur when the cause and effect relationships of elements in the diagram eventually lead back around to the starting point. The loop is balancing when the effects created in one part of the loop are mitigated or reversed by effects created in another part of the loop. When effects generated in the path of a loop build on one another to create a magnified effect, the loop is called reinforcing. These two types of causal loops are useful for identifying process steps, practices, or organizational constraints that tend to lock an organization into a pattern of behavior that may not be desirable. They can point to the part of a process that is likely to produce an improvement if changed.

Personally, I expect these tools to be most useful when exploring a client's or prospect's problems. I can see how they would help everyone understand what the most significant issues really are, and therefore what to address first and how to approach the problem. Very often, people assume they know what their problems are, when they may only be distracted by visible effects or by assumptions about the changes they should make.

For example, our group explored a problem one of the members was having at his company. We began by drawing a circle to represent the most important problem as he saw it. In the circle we wrote, "Testing the product enough." As we identified and discussed the causes, effects, and external factors relevant to the problem, we eventually discovered the core problem was not testing, but rather the alignment of development results with business needs. The "distraction" was the assumption that if we tested the product more, we could improve the alignment of results with needs. The DOE helped us learn the issue with testing was not how much testing was done, but whether we were testing the right things. Testing more wouldn't help if we were only testing more of the wrong things. We changed the label in our first circle to "Testing the product effectively." The discovery led us to explore what was causing us not to know what the right things to test were. In turn, that led to the discovery of a lot of ideas that ranged far afield of testing as such. Here's a photo of the diagram we ended up with. It isn't a finished product; the clock ran out on us. (The photo is from Yves' flickr set.)

XP Days Benelux 2008: The best way to introduce agile?

Mon, 24 Nov 2008 11:50:58 -0500

Nicole Belilos facilitated a session to explore the strengths and weaknesses of various patterns of agile adoption. It's no accident that the session title ends with a question mark. Her intent was to have participants share their own experiences and ideas with each other rather than to deliver a lecture. After a brief introduction, Nicole organized everyone into small groups. Each group discussed their own experiences and thoughts about the upsides and downsides of a particular approach to introducing agile into an organization.

Each group dealt with one approach — top-down, bottom-up, technical practices first, or iterative process first. Each group included some people who had been involved with an agile initiative of the type the group was exploring and some people who had not. Nicole used a "debate" format for the debrief portion of the workshop. Personally, I didn't think that was the best possible format, since we weren't trying to determine a single optimal way to introduce agile; we were trying to understand the circumstances in which each approach might be effective, and what potential pitfalls may lie in wait for the unwary. Even so, I think participants came away with a better understanding of how agile might be introduced, when to take a particular approach, and what problems to be aware of.

The photo is from Yves Hanoulle's XP Days set on Flickr.