effective software development

Spoiler

Sat, 21 Nov 2009 07:49:17 -0500

A friend of mine tweeted the other day:

So now that Java 7 is getting closures at the end of 2010, it'll only be behind C# by what, 5 years? Better late than never.

To which I half-jokingly replied:

So now that C# can run on more than one platform thanks to mono, it's only behind Java by what, 5 years?

In 1986, I bought a Toyota MR-2, a small, lithe, mid-engine two-seater. It could take curves like a slot car. More fun to drive than any other car I've owned. It was red, and it had a rear spoiler. I liked the spoiler. It made me feel cool.

The purpose of a rear spoiler is to hold the back end of the car down on the road when you're driving really fast. The 1986 Toyota MR-2 could briefly flirt with 95 MPH, going downhill, with a tail-wind, and given five minutes to build up speed gradually, provided you had no passenger aboard and you removed everything heavy from the car, and while driving you wished with all your heart that the speedometer pointer would move just one more millimeter, oh please please please. In other words, the spoiler could never have any aerodynamic effect on the vehicle. It was only a decoration. Even so, I liked it. It made me feel cool.

Support for closures in a programming language makes for clean, concise, readable code. All good.

Consider two hypothetical programming languages such that:

Language A

Supports closures, and
Can only run on one platform, so that customers who use the language for mission-critical apps are locked into a single operating system vendor.

Language B

Doesn't support closures, and
Runs on all mainstream platforms, so that customers who use the language have flexibility to change their technical environment as their business needs change or as the relative cost of ownership of different platforms changes.

Which of the two programming languages is "better?"

My friend might say that Language A is better than Language B. Language B is "behind" Language A because it doesn't (yet?) support closures.

I might say that closures are a great feature, or at least an interesting feature, but since Language B offers greater flexibility to customers, it yields better business value overall than Language A.

I've been playing with C# on Mono on a Linux system recently. C# is a cool language, if you consider programming languages outside of any business context, as if you were playing with toys in a toy shop. If a toy will only work in one section of the toy shop, who cares? It's only a toy, after all. It isn't as if the success of your business depended on it. Plus, it's fun to argue with the other children about which toys are the most fun. I suspect a lot of people who choose software engineering as their profession choose it mainly for the opportunity to argue, and only secondarily for the opportunity to write code. The relative amount of time software developers spend arguing with one another about pleonastic niceties of programming style or compiler features seems to support that suspicion.

Mono doesn't (yet?) support .NET completely seamlessly; a C# application has to be written specifically for Mono if you intend to run it on any platform other than Microsoft Windows. Your staff won't be able to run their familiar Visual Studio tools on anything other than Microsoft Windows, even if your target platform is something else. So, we don't quite have cross-platform support for C# that is ready for prime time. We will. Novell is behind Mono, and the product is maturing rapidly. If I had to make a prediction, it would be that Java will have support for closures long before C# becomes a truly viable cross-platform tool. Or maybe it will be the other way around. Eventually, both things will happen and the argument will settle itself. If you enjoy that sort of argument, take heart: There will be some other cool language feature to argue about. Before there were rear spoilers, I hear that people used to like fins on the back of their cars.

As a developer, I like closures. They make me feel cool. And, like any faithful Monty Python fan, I like a good argument. As an IT consultant with the responsibility to make well-considered recommendations to clients, I have to look at programming languages from another angle. The cost of change can be prohibitive when your organization is nailed to a single hardware or software vendor. Those of us who have earned a few gray hairs have been there and done that, probably more than once. Be careful about selecting programming languages or other software development tools that will lock you in, even if they have nifty features that make you feel cool. It's more important to manage the total cost of ownership of your IT assets than to have the coolest programming language in town.

P.S.

I'm talking about conventional business applications, BTW. Given the literal-mindedness of some of the people who have posted comments here, I guess I have to spell that out. Embedded apps, like flight control systems and toll plaza systems, and software that requires low-level access to devices, like video games, have to be closely tied to the hardware. That is not the context of this post. Nearly all conventional business apps are just CRUD apps. There is no valid reason to tie a CRUD app to any particular OS, or any particular desktop, or any particular back-end data store. Building a solution in that way only imposes a vendor-lock on your customers while providing no value-add in exchange.

When success = failure

Wed, 18 Nov 2009 10:38:14 -0500

There's a lot of buzz in the community about how to drive high productivity at the grassroots level, and how great it is when we can help develop a high-performing team. There are also those in the community who call for a heavy-handed approach to change; sometimes they use the term "shock therapy" to describe a process of instituting radical changes in team structure and organizational workflow in a short time. Some consultants speak of improvements in productivity in the range of 4x to 10x or greater. Although there has been recent movement away from the term, some consultants still seem to like the idea of hyperproductive teams. People still talk about "spinning up" high-performing teams.

In thinking back over a couple of experiences in team coaching and organizational change, I've begun to realize that there is more to success than just spinning up one or two "hyperproductive" teams. One experience was at the first company where I learned agile methods in 2002, and the other was earlier this year, 2009. In both cases, we achieved astonishing success in creating one team or a small group of teams that performed far outside the norm for their respective organizations.

In each case, the teams in question pushed the envelope of state of the art agile practice. In 2002, that meant two-week iterations, relative story sizing in terms of abstract points, task decomposition, and a daily burndown of tasks. In 2009, it meant a one-week development cadence with an independent release cadence, stories split vertically into small enough chunks that no task decomposition was necessary, no daily burn tracking, "naked planning," fully generalized skill sets, and self-management to the extent they screened and auditioned job candidates and handled their own personnel issues. In both cases, it meant a disciplined approach to Extreme Programming practices and an approach to development that today we could call software craftsmanship. The point is, these teams rode the bleeding edge of whatever the state of the art was at the time. Indeed, one could justifiably say they sharpened that edge. They definitely would have qualified for the label, hyperproductive.

So, these were roaring successes, right?

I used to think so. But I have to wonder: If they were successes, then why did "agile" peter out at the first company after an amazing three-year run of successful projects? Why are the people at the second company unhappy with the new way of working, despite the fact they deliver more business value than before and they can pump far more work through their process than they previously imagined possible?

On reflection, I think the basic reasons are these:

Local process optimization
Insensitivity to emotional factors

Case 1

In 2001 I joined a mid-sized financial services company with about 33,000 employees, operating in 6 US states, and with an IT department of about 1,300 people. It was a nightmare of waterfallish processes, crushing bureaucracy, cartoonish fake smiles, dehumanizing toadying, and walled-and-moated functional silos. Less than a year later I was fed up and ready to leave both the company and the IT field. At the urging of a colleague, I joined a small group of employees who wanted to analyze the business value the IT department added to the enterprise, and determine how to improve things. Along the way we discovered the Agile Manifesto. We enlisted help from ThoughtWorks to teach us agile methods. Long story short, in a few months we had built up an internal agile practice that could handle several projects at once, and we were delivering value regularly. The "business side of the house" loved our group. Line of business managers started to demand that the IT department carry out their initiatives using our methods. Things just got better and better...

...for a while. There are many war stories to tell from this time period. For purposes of this post, let's skip to the end of that three-year run. As soon as they could, IT management "took control" of the agile group. Within a week, they had (a) cancelled the practice of locating technical teams on-site with their internal customers; (b) scattered the agile practitioners around in the traditional organization, to dilute their influence; (c) conjured negative performance reviews of key agile proponents in the organization as a way to encourage them to leave the company; some were even put on probationary status, to pressure them to leave; (d) assigned some of the top performers to dead-end jobs babysitting relatively unimportant legacy apps; another tactic to encourage them to leave; and (e) re-established the waterfall process (decorated with a few agile buzzwords) as the sole way of delivering projects for internal customers. One year later, there was nothing left of "agile" but the word, much as the Cheshire Cat faded away and left only his smile behind.

Why would people do this, to the detriment of their own company, the organization whose success fuelled their own bonus programs and retirement plans? At the time, the reasons were incomprehensible to me. I chalked it up to incompetence or, in some cases, maliciousness. In hindsight, the reasons are obvious: We had embarrassed IT management. Where they insisted something would be too costly, too time-consuming, or technically infeasible, we went ahead and did it at reasonable cost, in a short time, and with technical excellence. We demonstrated that everything the traditional IT managers believed to be true of software development was wrong. We went around them, and worked directly with the lines of business. We ignored their standards and procedures. We openly disrespected the chain of command, the org chart. We went around the HR department, as well, and recruited our own people to join the agile teams. This made HR feel disrespected, too, since our behavior demonstrated that we did not trust them to screen job candidates adequately. To add insult to injury, we did all that while wearing blue jeans and working four-day weeks...at a traditional financial institution with a very conservative organizational culture.

It was all amazingly productive and added tremendous value to the enterprise. It was also absolutely intolerable. One year after IT management "took control" of agile, only 4 out of 60 agile practitioners were still employed there. Those four had settled back into quiet, secondary roles in the organization where they could accumulate long-term retirement benefits as long as they kept their mouths shut and didn't make any waves. Everybody (who was anybody) was happy again.

Local process optimization. Our group was physically and philosophically separate from the rest of the IT department. It grew up as a skunkworks operation and only survived as long as it did because our customers wouldn't allow IT management to crush it openly, since they were realizing value from our work. We did not improve the company's IT processes across the board. We only optimized our own work, in isolation from the other 1,300 IT professionals in the company. We were so fully isolated that most of those people probably never even realized anything unusual was going on. If you asked them about it today, I'll bet they wouldn't even know anything like this had happened in 2002-2006. The business managers who learned about the value of agile methods during that time moved on to other jobs, or retired. The organization itself has not gained or retained any deep tacit knowledge of agile practices or their value, and has not fundamentally changed its culture.

Insensitivity to emotional factors. We simply ignored most of the individuals whose support we needed in order to achieve long-term process improvement. They responded in a perfectly normal human way: They bided their time until they could crush the agile initiative, and then they did so quickly and firmly. They "punished the innocent" and made sure anyone who had been part of the agile group either sat down and shut up, or left the company. It's possible that those managers left the company eventually, too, and that a new crop of managers has experimented with agile or lean methods again since 2006. If so, they have not benefited from anything we did in 2002-2006, because internal history was rewritten to expunge all memory of it. This huge waste, this huge loss, is a direct result of our insensitivity to the emotional needs and politically-motivated fears of key management personnel. We succeeded and we failed with the same stroke.

Case 2

Early in 2009 I was engaged, along with another agile coach, to help a certain company improve its business processes and to help a couple of technical teams get up to speed with Extreme Programming practices. This is a smaller company than that in Case 1, with under 1,000 employees working in only two geographical locations. We were working with teams and stakeholders at one location, and most of the technical teams worked at the second location. There, they had been practicing Extreme Programming for five years already, under the guidance of an internal agile coach and with the support of their own management team.

The two of us quickly determined that the main problem in the organization was not the technical teams' use of Extreme Programming, but rather backlog grooming upstream from the technical teams. We focused mainly on that, and did not pay any attention to the teams at the second location. In the meantime, I coached the technical teams at the first location in agile methods, which were new to them.

Long story short, in about three months the teams that were new to agile methods were applying more-advanced techniques and delivering value better than the teams with five years of XP experience behind them. They were using methods I mentioned earlier, which represented state of the art practices circa 2009, and were functioning at a hyperproductive level as compared with the organization's general IT performance. They were applying agile values and principles throughtfully and crafting their own practices accordingly, rather than following "rules" from a book by rote. After five or six months, however, they were floundering. The team was far out of sync with the pace of work the rest of the organization could sustain.

Within their own workstream, they were able to pull work much faster than the stakeholders could keep the backlog up to date. This caused a lot of dissonance in customer collaboration. It also forced the team to revert to story sizing. Whereas relative sizing was a state of the art practice in 2002, as described in Case 1, in 2009 it is no longer a state of the art practice. Here, a team that was already skilled at commitment-based short-term planning without fine-grained estimation had to drop back a step and start sizing user stories, simply because the stories were not in appropriate shape for them to pull; they did not have any idea how big the stories might be or any inkling of potential technical risks until the very day they were expected to start working on them. This was a source of frustration for the team, and also a source of stress for stakeholders, who felt as if they were "failing" in some way although they really didn't have time to do any more than they were already doing.

Local process optimization. We had a single team functioning at a level of performance that was out of sync with the rest of the organization. We introduced change too rapidly and in too localized a way, and caused friction in the organization. This particular team was capable of absorbing that much change that quickly, but because they were out of sync with the rest of the organization their very success created stress and frustration for them and their customers. When they had to back off from leading-edge practices, it made them feel as if they had "failed" with agile, in some sense. That is especially unfortunate in view of the fact that even after they slowed down they were operating at a level of agility beyond that of most agile teams in industry.

Insensitivity to emotional factors. In hindsight, I think this was the area where we failed most spectacularly. We failed on two levels: First, we drove change forward as rapidly as possible and ignored the emotional impact of change on the team and their immediate collaborators; second, we ignored the people at the second location (at least, initially), who had built a sense of personal pride and an internal reputation in the company as agile experts in the course of five years of nurturing XP teams there.

The team members could see that they were performing better than they had done previously, yet they were not happy about it. They didn't have time to be happy about it. They had lost their individual cubicles in favor of a collaborative workspace geared for pairing; the Furniture Police did not consider all the aspects of an effective team work space when they set up the team area, and did not consult the team about their needs. As it turned out, this single factor had a significant impact on team morale. The last time I checked in with them, they had lost one senior team member and two others were emotionally "checked out," showing up for work but not really feeling engaged. They are still delivering value at a higher rate than any other team in the company, but they are doing it mechanically, without a deep personal investment in the process.

The in-house agile experts felt threatened by the new ideas the consultants brought. We caused them to feel that way by failing to engage them properly from the beginning. With the consultants now gone, the in-house agile experts have reasserted themselves as the thought leaders for process improvement. They are imposing "rules" such as they use with their own teams, based on state of the art agile circa 2004. That is when they started to introduce XP, and it is the style of agile work they understand best. In hindsight, I think we should have approached change by engaging these individuals from the start and introducing change at a far more gradual pace. We didn't help them understand how agile practices have evolved in the industry since 2004. As it is, the pattern that is emerging is similar to that at the company in Case 1: A brief flurry of localized extreme high performance followed by a breakdown and re-establishment of the organization's previous equilibrium state. Cold comfort: At least in Case 2 the previous equilibrium state is superior to that in Case 1.

Thinking back over these experiences leads me to another question: One year after the boastful consultants have left a client organization, what does the process look like at that organization? Did the changes "stick?" Consultants often quote client testimonials in their marketing material. What would the clients say one year, two years, or five years after they released those quotes?

The question reminds me of a conversation I had with the CEO of a previous employer, an agile services consultancy. There was one client in particular we used prominently in our marketing materials, who had given us a glowing recommendation. Two years after that successful project, we were reviewing the list of upcoming engagements and prospective engagements one day and I noticed we had never obtained any follow-on work from that particular happy client. I asked him why, if the client had been so happy with the results, they had never brought any more work to us.

His answer was along the same lines as this blog post. Our team had performed at a pace that was far out of sync with the client organization. Our customers could not keep the backlog up to date fast enough to feed our team meaningful work. They threw stories together helter-skelter just so they wouldn't be paying us an hourly rate to have our people sit around and wait for something to do. The experience was so stressful for them that they decided they weren't suited for agile work, and they reverted to traditional methods and traditional services firms.

It occurs to me that we may have served our client better had we adjusted the pace of change and the demands for collaboration so that they were more closely aligned with the client's ability to cope with change. The initial project may have taken a bit longer, but we probably would have won follow-on work from the client because they would have learned how to collaborate effectively with an agile services provider. Instead, they came away feeling as if this "agile" stuff can produce impressive results, but it just isn't for them.

Turning all the knobs up to eleven while ignoring the human costs of doing so may not be the definition of success, after all. Beware of consultants bearing gifts of hyperproductivity.

I recall a comment from Ron Jeffries on the Extreme Programming discussion list some time ago, to the effect that if you haven't gone all the way with agile then you don't have a frame of reference to understand when and how to back off from particular practices without losing the value of the agile approach. (Of course, his comment wasn't as verbose as that. Verbosity is an affliction of mine.) Anyway, I've gone all the way more than once and in more than one context. I think I have a frame of reference to understand when and how to back off. I recently had an initial visit with a new client. Until I understand their organizational culture and their values pretty well, I won't be turning those knobs anywhere near eleven.

UMLet UML diagramming tool

Thu, 22 Oct 2009 13:21:26 -0500

I found a UML diagramming tool that I think is well suited to agile modeling. It's called UMLet, and it runs either standalone or as an Eclipse plugin. The product home page is here: http://www.umlet.com.

The reason I say it's suitable for agile modeling is that it works well, but it isn't slick and it doesn't have ten thousand options for making the diagrams look pretty. When a diagramming tool is too slick, it can be tempting to spend a lot of time fooling with it just because it's fun, and to go into more detail in the diagrams than is appropriate for the agile style of development.

If you like making pretty pictures, or you think our job is to produce highly-polished diagrams rather than working software, or you're a stickler for using the specific icons defined for a particular version of the UML, or if you enjoy playing with diagramming tools to the exclusion of pulling work items through to the "done" state, then you might not appreciate UMLet. If you'd like a simple tool to produce high-level UML diagrams easily, then you might like this tool.

Here are some of the reasons I like UMLet:

It's Open Source software.
It runs on all mainstream platforms: Microsoft Windows, Apple OS X, and Linux.
It runs standalone or as an Eclipse plugin.
It's simple enough to use that it doesn't require a user's manual or a training course.
It supports all the usual suspects - class diagrams, sequence diagrams, use case diagrams, etc. - and is good enough for high-level diagramming without falling in love with its own reflection. It's just crude enough that you won't be tempted to play with it all day long. One might say it's perfect in its imperfection.
It's easy to clip out a diagram and save it as a graphics file, to drop onto your project wiki or into a presentation or document.
It saves the data in XML, so if you have to rescue your diagrams at some future time when UMLet is no longer supported or goes commercial, you at least have some reasonable basis for doing so.
It's easy to check the diagrams into and out of your version control system along with the production code and test suite. Easy to script all that, too.

Here's a crude class diagram I put together in a few minutes' time using UMLet. I'm sure it isn't academically correct, and that's okay with me.

The only problem I've seen with UMLet is that it throws NullPointerExceptions out of the AWT dispatch thread quite frequently. It doesn't seem to do any harm, though.

I'm using UMLet on Ubuntu Linux. Here's how I installed it.

Read It Later plugin for Firefox

Thu, 22 Oct 2009 12:20:59 -0500

When looking for information online, I often come across interesting articles or blogs that I want to read later. I had been bookmarking them in a folder created for the purpose. When I remembered to look there, the items may or may not still have been relevant and timely. In any case, I had to clean out that folder manually, since they weren't really bookmarks I needed to keep for repeated reference. I recall a day when there were over a hundred items in that folder, and I didn't even remember why I had tagged most of them.

One of the things I came across while looking for other information was a Firefox plugin called Read It Later. Its purpose is to manage a special folder in your bookmarks file where you can stick URLs of sites you'd like to read later. This is for items you'd like to read on a one-time basis; it's not a substitute for RSS feeds. It can also save the pages locally for offline reading. It's easy to move items to your permanent bookmarks or delete them when you've read them. The plugin doesn't actually do all that much work, but it's a time-saver and it's very well-behaved code.

Here's a screenshot of a Firefox window with the Read It Later homepage displayed, and showing the elements the plugin adds to the toolbar.

To add an item to the read it later folder, click on the checkmark near the right-hand side of the toolbar. On the right, the drop-down list of saved sites is open in the screenshot. Simple and useful. I've had no problems with it so far. The Read It Later homepage is here: http://readitlaterlist.com/.

Is this a process smell?

Thu, 22 Oct 2009 09:31:44 -0500

I was just scrolling back through old blog posts, trying to decide which ones to keep when I revamp the site, and I came across a post from last February on the subject of tools that provide continuous feedback. I noticed a comment from Lee Winder that I had overlooked until now. He writes,

...we have a system which alerts designers when a new build is ready on the CI machine.

If we have fast turn-around, they could be getting alerts every 10/15 minutes, which is certainly to much if they only update once or twice a day.

Since the alerts are e-mails, I've recommended they filter them into a CI folder, only checking the alerts when they need a new build to see when the last one was generated.

I wish I had seen this earlier. It seems to me the statement, "they could be getting alerts every 10/15 minutes, which is certainly to much if they only update once or twice a day," suggests a process smell. It may or may not be a real problem, but it sounds questionable in the context of agile-style workflow. It may be worth finding out why (1) the team is only updating (checking in?) once or twice a day, and (2) why they think that is satisfactory.

It isn't necessarily a problem. "Smell" doesn't automatically mean "bad;" it just means, "Find out where the smell is coming from, in case it's a problem." It might only be that you neighbor's cooking smells funny. OTOH, it may be that your curtains are on fire. It doesn't hurt to find out for sure.

If this team is updating only once or twice a day, then by implication they're only running their own build once or twice a day. Lee's company does video game development in C++, and the practical limit to the number of updates per day this team can make may be different than for typical business application development in languages like Java and C#. Game development involves several distinct types of programming, and the practical maximum builds per day may be different for this team than for other teams working on the same game. Even so, it might be good to understand exactly why they are limited to just one or two builds per day, just in case the curtains are on fire. Couldn't hurt.

The notifications aren't from this team's own build; they are consuming a build from another team. That team is updating at least every 10 to 15 minutes; not bad. A simpler solution might be to remove this team from the distribution list for that build's automatic notification. There's no need to have the email server churning when the notifications aren't useful. Team members have to go in and delete all the useless emails at some point, too. Maybe it's low enough impact that it doesn't matter, in context. It sounds a bit wasteful, anyway.

An information radiator is sometimes helpful in cases when one team needs the latest stable build from another team. I've seen cases when a large monitor is set up in each team room showing the build status for all the parts of the project that are in flight at the moment. This team would be able to tell at a glance whether they needed to pull a new build. I don't know which tools they're using at Lee's place, but I've seen this done using Big Visible Cruise in conjunction with Cruise Control. It may be possible with other CI servers, as well. Just a thought.

Agiles2009 Conference

Wed, 21 Oct 2009 19:05:16 -0500

Ágiles 2009 took place on October 6-9 in Florianopolis, Brazil. It was the second in a series of agile development conferences bringing leading-edge software development and project management practices to Latin America. The first was held last year in Buenos Aires, Argentina. It was organized by a small group of highly motivated and dedicated people, primarily a group that had met during their university years in Buenos Aires and who are driving agile and lean adoption forward in Argentina, but also including professionals from several other Latin American countries who share a passion for agile development. The first conference was a great success. The second was at least as informative and enriching, if not moreso.

Of course, these are not the only agile- and/or lean-focused events in Latin America, but they are the largest and they seek to bring in international participants. With strong support from the Agile Alliance and from key commercial sponsors, the Ágiles conferences bring together agile and lean practitioners and researchers from across the continent as well as from North America and Europe and help tie together the various local and national agile/lean adoption movements in Latin America.

In addition to Latin American thought leaders in agile and lean adoption, the conference included several well-known figures in the general agile movement, including Diana Larsen, Brian Marick, Joshua Kerievsky, Naresh Jain, David Hussman, and others. There were too many excellent sessions to describe in a blog post, and the quality of those presented by Latin American speakers was on par with that of the better-known speakers. Look to this region for the next generation of leaders in moving the industry forward with effective management and software development methods.

Several participants took some great photos of the event. You can find links to their photo albums on the Latin American Agile Community site: http://sites.google.com/site/comunidadagiles/agiles-2009> I didn't take any great photos, but I took a few mediocre ones. Here's a shot of me with some of the Argentines I met last year, as we began to gather in the lobby prior to the opening keynote:

Left to right: Alejandra Alfonso, Adrián Eidelman, Emilio Gutter, Juan Gabardini, and me.

This one is from Joke Vandemaele's presentation on Power Workshops, a technique she has been using successfully to launch complex agile projects at a company in Belgium. The visuals on the wall are the ones from an actual project that she used as an example.

Brian Marick made himself available for pairing during the conference. He invited anyone who wanted to sit with him to work on an application for his wife's veterinary practice. Here he has grabbed a spot in the lobby and is pairing with one of the conference participants as others look on.

After the final session on Thursday, a rock band made up of people from OnCast Technologies, a Florianopolis-based agile consultancy, put on a concert in the lobby and invited others to join in and jam.

The conference proper was two days long, and was preceded by two days of training courses. These included a TDD & Refactoring course taught by Naresh, a CSM course taught by Alan Cyment, a Certified Product Owner course taught by Alexandre Magno, and a course on retrospectives taught by Diana.

My small contribution was a presentation of my session on Agile Metrics. The presentation continues to be popular. This one was standing room only and went overtime, as usual. Although it seems a rather mundane topic, it is clearly one that is on the minds of many project managers and program managers as they learn different ways of planning and tracking projects.

Florianopolis is quite a nice city. In my spare time I walked around and saw as much of the place as I could. I especially enjoyed the natural scenery around the bay.

The weather was cloudy the whole week. The clouds and mist on the mountaintops in the distance made for an ever-changing scene.

Some parts of the city reminded me of certain European cities. This is the old central market plaza, for instance:

All in all, a great conference with great people, and a fantastic place to visit.

With last year's conference in Argentina and this year's in Brazil, we've all had a chance to try the churrasquerias in both countries. Both are world-famous for their beef. The burning question on the minds of the conference organizers, since most of them come from those two countries, was: Who has the best beef, Argentina or Brazil? To me, the question reminded me of Belgians and Germans debating about who has the best beer. It's basically a moot question, since Ireland has the best beer. (In my humble opinion, anyway.) Sorry to disappoint our gracious conference hosts, but I have to say Venezuela has the best beef. Ágiles2010 will be in Lima, Perú, so the true test must wait for a future year. Fortunately, the future of agile and lean will be a long one.

Grooving to the buzz

Wed, 30 Sep 2009 20:27:04 -0500

An agile team at my current client finally got a proper team work room set up, following lengthy negotiations with the Furniture Police. They had been working in cubicles that were located next to each other arranged as conveniently as you can imagine cubicles being arranged.

Although the cubicle set up wasn't bad, having a true team work space has made a world of difference, even though the whiteboards haven't arrived yet and they've requested wider tables. The team's throughput tripled in the first iteration after the space was set up. The "buzz" of a well-functioning agile team is in the air; it was absent in the cubicle environment, even though it was arranged as reasonably as possible and team members collaborated as closely as they could.

Osmotic communication and promiscuous pairing, managed pomodoro-style, have kept the team's truck number high and have even reduced the need for such basic overhead activities as the daily stand-up. The team has a stand-up when they need to discuss something or when a stakeholder comes to the team room to participate. Otherwise, they communicate seamlessly and continuously about issues and status.

We haven't tried to keep track of every little move the team members make, but I strongly suspect they're enjoying cost savings just by being properly collocated.

I'm more convinced than ever that a proper team work space is a First Domino agile practice. Why would any software development team choose not to work this way, especially after all these years of proven effectiveness in the field?

Rails through thick and thin

Tue, 29 Sep 2009 14:44:21 -0500

It seems as if everyone likes to blog about the fat model and skinny controller idea when they're just learning about Rails development. I'm just learning about Rails development, so here's my blog post on the subject.

The canonical model-view-controller architecture calls for the controller to be lightweight. There are good reasons for this, including flexibility to scale across architectural layers, loose coupling, ease of understanding the code, and more. As a Java developer, I typically follow this guideline when building webapps with MVC frameworks like Struts2. It's only natural for Rails developers to try and conform with this very logical design guideline.

When you generate Rails code, the out-of-the-box solution has very thin models with most of the logic included in the controllers; quite the opposite of generally-accepted good practice for an MVC app. There's a lot of advice out there for people who want to skinny-down the controllers, and there's nothing new about it. For instance, there's a nice write-up by Michael Koziarski dating from 2007. The "before" version of the create method in his ReportsController is a great example of what can happen when you jam too much logic into a controller. In an even earlier example (2006), Jamis Buck shows the value of pushing logic from the controller to the model. His example resulting in the find_recent method in the Person model class is excellent.

But there's another issue to consider. Rails models are based on the ActiveRecord pattern. As a result, Rails models are tightly coupled to the persistence mechanism. By default, the persistence mechanism is a relational database, although this can be overridden. Because of this coupling, Rails models really don't feel quite like a domain layer; at least, not to me. It "feels" as if we need another layer between the controller and the model. I've heard and read Rails developers resist this idea, sometimes strongly. At the same time, other Rails developers say that pushing too much business logic into the models amounts to substituting a new problem for the old.

Besides all that, it's hard to test Rails models without a dependency on the database due to the lack of separation of concerns. There are tools available to mitigate the problem, such as Thoughtbot's FactoryGirl, but you still end up with some tests touching a real database instance. Tools like FactoryGirl make it easier to define fixtures, but they don't really isolate you from the database. In a large app, tests that are dependent on a real database instance can affect build times, and long build times tend to have a domino effect on other good practices.

So, we want skinny controllers, and Rails models are neither a business layer nor a domain layer, properly speaking. What should we do? I think it's legitimate to treat the models as a thin persistence wrapper, and to add either helper classes or a distinct layer between the controllers and the models to function as a business layer, if you need one. You might not need a business layer, since most business applications are almost purely CRUD apps, and don't actually have much "business logic." Let the models focus on one concern — persistence management — and pull the other concern — the domain model as such — into a separate layer. It seems more loosely coupled and cleaner than the default architecture.

Most of the time, the hand-coded methods people add to Rails models are really just handling a find with special conditions (as in Jamis' find_recent example), or saving an object with a change of state, like from "active" to "inactive." It seems to me this isn't necessarily part of the model's responsibility. It feels more like a choice being made by client code. The default model code already handles find conditions passed in from the client, and already knows how to save whatever values the client passes to it. Custom methods that ostensibly make the intent of the code clearer may not actually do so. They may obfuscate the intent of the code by separating the conditions for a persistence request from the context that makes those conditions obvious.

Going with generated code for models, plus a few declarations to invoke built-in functionality like has_many and validates_presence_of, also alleviates the problem of isolating unit tests or specs for model classes. A rule of thumb for unit testing is that we should not test generated code or the plumbing/framework; we should test hand-written code. When you treat Rails models as a thin persistence wrapper, you end up with literally no hand-written code in the model classes. The problem of unit tests that have dependencies on an external database simply disappears, because you don't have unit tests for the model classes — they consist entirely of generated code. Classes in your true domain layer can be isolated for testing just by mocking the models. In effect, you let the model classes adhere to the Single Responsibility Principle simply by omitting domain logic from them, leaving them with the single responsibility to wrap the persistence mechanism. Now the single reason to change is that you need to change the peristence mechanism from a relational database to something else.

I wanted to think through the question of skinny vs. fat to see whether there might be implications for scaling a Rails app. This is what I came up with, FWIW.

The app you get by default from the generators is designed for small-scale, low-volume CRUD apps that live entirely on a single server. Whether you have the bulk of your business logic in a controller or in a model doesn't make all that much difference. A messy method in a model is just as hard to read as a messy method in a controller, and since the whole mess lives on the same server, it's a wash. It's like squeezing a balloon that is half-filled with air. When you squeeze one end, the air bulges into the opposite end. Think of one end as the controller and the other end as the model, and you can see the effect of moving code between the controller and the model. It really comes down to a question of personal preference.

Architectural niceties start to make a difference when you go beyond the level of small-scale, low-volume CRUD apps that live entirely on a single server. How you need to scale depends on the circumstances, and most forms of scaling will not affect the application code at all because they are handled by assets external to the application, like routers or containers. In those cases, it still will not matter whether most of your logic resides in controllers, in models, or in an additional layer between the two.

Let's say we begin with a small-scale, low-volume CRUD app that lives entirely on a single server. Over time, the user base for our app grows. We need to be able to handle a higher workload than we did originally. This can be done by running the Rails app with a facility like Passenger for Apache. Passenger is smart about caching query results and keeping objects instantiated in memory between requests. It enables a Rails app to handle a higher transaction volume in a way that is completely transparent to the application code. In this case, your personal preferences regarding controllers and models neither help nor hinder scalability; you are free to do as you please.

Maybe the need for scaling is a question of availability rather than transaction volume, or in addition to transaction volume. In this case, we can set up a router in front of the application that sends requests to alternate copies of the application running on different servers. The servers may be clustered and may support hot failover, or other features to support high availability. This can be done in conjunction with a facility like Passenger, too. The different instances of the app can access the same back-end data store or different ones, as appropriate. There is no effect on the application code, so your personal preferences are still completely valid.

One of the ways we scale Java webapps is by deploying different architectural layers independently. That way, we can deploy multiple controller instances or multiple business layer instances and so on as needs change. The advent of virtual servers makes this approach all the more feasible and responsive to dynamically-changing workload demands.

You wouldn't normally separate Rails views, controllers, and models physically. If you needed to do so, you could use routes.rb to route different requests to different instances of the business layer or model layer. I can't actually think of a use case for it, but in any case it still doesn't affect the application source code and wouldn't push you toward thin or thick controllers, particularly. The use cases that drive us to separate components in Java webapps can be supported by deploying multiple copies of the whole Rails webapp.

Another scaling solution is to route requests to different servers based on geographical region or company code or some other piece of information that comes in on the request message. You could create a thin controller layer that does nothing other than interpret the relevant code and pass the request through to another controller residing on a different server. When you have different transaction volumes or different quality-of-service levels for different regions or companies (or whatever), this may be a reasonable way to split the traffic across multiple servers. The base application code still doesn't have to change; all you need to do is create a new set of controllers, or a single controller, to pass the requests through to the appropriate instance of the app.

You could conceivably achieve the same goal by defining special routes that included the necessary routing code, rather than introducing another controller layer. In that case, clients would have to include the routing code in the URLs they used to invoke the Rails app. This may or may not be desirable, depending on circumstances. In any case, it has no effect on the application source code, and so your personal preference for skinny or fat controllers is unaffected.

It's possible that the need for scaling has to do with the volume of data transferred to and from the database. We can accommodate that by changing the database adapter and using a different DBMS product; perhaps moving from MySQL to Oracle or UDB. That's transparent to the application code, as well, unless you've nailed your feet to the floor by using stored procedures.

Even if you've nailed your feet to the floor, you can pull them loose again by doing away with the built-in database access in the models and having them call a service layer instead; something like an enterprise service bus, perhaps. Your database resources (or whatever the real source of data happens to be) can be isolated behind the service layer where the details are transparent to your application code. In this case, your personal preference may affect the difficulty of the change.

This is a fairly typical situation in enterprise IT organizations, including those whose feet are not nailed to the floor. Business apps need to pull data from a variety of back-end sources, and the sources may change without notice provided the interface remains stable. If you're using a paper-thin model that acts as nothing more than a persistence wrapper, it's a trivial modification to call the service layer instead of a local database adapter, and your existing unit tests provide a safety net for the changes. If you have a fat model design, you may have more work to do. Realistically, this sort of architectural issue would probably be known at design time, so it's really an academic exercise. When the day comes that there are a lot of legacy enterprise Rails apps around, it may become more common situation.

Personally, I like the skinny model approach because it simplifies testing. I don't mind seeing a bit of business logic in the controllers, and since Rails is a tool for CRUD webapps, there won't be extensive business logic in most cases anyway. If the controllers start to look too fat, it's easy enough to move some of the logic into helper classes. If the helper classes call the models, then you can think of them as a sort of "business layer," although that's really a matter of code organization and readability and not a matter of architecture, since the Rails app will be deployed all in one chunk. Another way to say it (although in a post this long the last thing we need is another way to say it) is that for a Rails app MVC is a design pattern rather than an architectural pattern.

My conclusion about this matter is that it doesn't make a difference one way or another. Structure your Rails app in the way that makes the most sense to you. If you need to scale it later on, it will make little or no difference whether you've written thick or thin controllers. Of course, this conclusion doesn't automatically apply to other languages or frameworks, or to applications other than webapps.

I reserve the right to change my mind without notice as I continue to learn about Rails development.

Some CFEclipse templates

Fri, 25 Sep 2009 12:37:10 -0500

I'm just wrapping up an engagement working with a team that uses ColdFusion. I found that ColdFusion requires a considerable amount of redundant typing, and I created a few Eclipse templates to save keystrokes. The templates work with the CFEclipse plug-in and the unit testing framework MXUnit. I'm sharing them here in case anyone else can use them. They are:

Name	Description
ad	Insert call to assertDatesAreEqual
ae	Insert MXUnit assertEquals in a cfset tag
ef	Insert MXUnit assertFalse in a cfset tag
arg	Insert cfargument tag template
at	Insert MXUnit assertTrue in a cfset tag
ax	Insert try/catch for MXUnit expecting an exception
before	MXUnit setUp function
fail	Insert MXUnit fail in a cfset tag
fun	Insert cffunction tag template
helper	Instantiate a TestHelpers component
if	Insert cfif tag template
init	Insert template for an init function
loop	Insert cfloop tag
ret	Insert cfreturn tag
set	Insert cfset tag
test	MXUnit test function
this=	Insert template cfset assignment from arguments

Download this zip file: cfeclipse_helpers.zip. The archive contains:

cfeclipse_templates.xml
MockSystemClock.cfc
TestHelpers.cfc
TestHelpersTest.cfc

To install the templates, with the CFEclipse view open in Eclipse, go to Window -> CFEclipse -> Editor -> Templates and choose Import. Point to the templates file and import it.

The .cfc files are test helpers that help isolate MXUnit tests that need to mock the system date.

Enjoy.

The new interview

Wed, 23 Sep 2009 12:35:23 -0500

I'm not sure whether this is a US tradition or if it's more general, but job candidates in this country tend to oversell themselves. Americans (and maybe others, as well; I can't speak to that) either believe they are gods or they assume they have to convince hiring managers that they are gods in order to have a chance of getting a job.

Yesterday, the team I'm presently coaching checked out two candidates for temporary contract positions on their project. This team spends very little time talking to candidates, and quickly moves to the audition phase when candidates pair with different team members to complete a small, contrived application. They have learned through experience that this is really the only meaningful part of the process, and that they cannot rely on résumés or interviews (or certifications — pay attention, ye who advocate developer certification programs!) to find out what a person can do. In fact, most of them don't bother to read candidates' résumés. They have learned that there is rarely anything factual in a résumé.

I was present during the interview phase and observed portions of the audition phase, although I didn't comment or interfere. It seemed to me that both candidates were reasonably good at the work. Neither was extraordinary, but they were okay. Clearly, they had at least a moderate level of practical experience in both the technology areas relevant to the project. It seemed to take them a long time to get through the little project, but they did manage to get it done. They probably would have performed adequately on the project.

The team decided the two were not up to par technically. I'm speculating that the candidates may have screwed themselves during the interview phase.

Both did a wonderful job of selling themselves. Based solely on their own words, one would expect each to be a one-of-a-kind expert in the technologies relevant to the project, and an all-around genius generally. By the time they had finished describing themselves, I was wondering why their names aren't household words, like Einstein or Newton. They didn't perform badly in the audition phase, so I have to wonder whether they set expectations too high during the interview phase. There is absolutely no way they lived up to their respective self-portraits. Frankly, I'd be surprised if anyone could have.

I wonder whether this is a difference between the conventional way of interviewing and the more empirical approach that is becoming more and more commonplace in agile software development circles. People used to have to sell themselves to employers strictly on the basis of words — both written and spoken. Nowadays, the words can backfire unless the candidate can confirm their claims when they sit down to write code. Given the extreme degree of exaggeration typically found in résumés, it's unlikely many individuals can live up to their own sales pitch.

These two may well have been selected had they sold themselves as reasonably competent developers with a balance between confidence and humility who enjoyed a collaborative working style and wanted to work with a high-performing team where they could share knowledge and learn from their peers while delivering value to their customers. They didn't.

Maybe next time.