Topic: Change agency
There's a lot of buzz in the community about how to drive high productivity at the grassroots level, and how great it is when we can help develop a high-performing team. There are also those in the community who call for a heavy-handed approach to change; sometimes they use the term "shock therapy" to describe a process of instituting radical changes in team structure and organizational workflow in a short time. Some consultants speak of improvements in productivity in the range of 4x to 10x or greater. Although there has been recent movement away from the term, some consultants still seem to like the idea of hyperproductive teams. People still talk about "spinning up" high-performing teams.
In thinking back over a couple of experiences in team coaching and organizational change, I've begun to realize that there is more to success than just spinning up one or two "hyperproductive" teams. One experience was at the first company where I learned agile methods in 2002, and the other was earlier this year, 2009. In both cases, we achieved astonishing success in creating one team or a small group of teams that performed far outside the norm for their respective organizations.
In each case, the teams in question pushed the envelope of state of the art agile practice. In 2002, that meant two-week iterations, relative story sizing in terms of abstract points, task decomposition, and a daily burndown of tasks. In 2009, it meant a one-week development cadence with an independent release cadence, stories split vertically into small enough chunks that no task decomposition was necessary, no daily burn tracking, "naked planning," fully generalized skill sets, and self-management to the extent they screened and auditioned job candidates and handled their own personnel issues. In both cases, it meant a disciplined approach to Extreme Programming practices and an approach to development that today we could call software craftsmanship. The point is, these teams rode the bleeding edge of whatever the state of the art was at the time. Indeed, one could justifiably say they sharpened that edge. They definitely would have qualified for the label, hyperproductive.
So, these were roaring successes, right?
I used to think so. But I have to wonder: If they were successes, then why did "agile" peter out at the first company after an amazing three-year run of successful projects? Why are the people at the second company unhappy with the new way of working, despite the fact they deliver more business value than before and they can pump far more work through their process than they previously imagined possible?
On reflection, I think the basic reasons are these:
- Local process optimization
- Insensitivity to emotional factors
Case 1
In 2001 I joined a mid-sized financial services company with about 33,000 employees, operating in 6 US states, and with an IT department of about 1,300 people. It was a nightmare of waterfallish processes, crushing bureaucracy, cartoonish fake smiles, dehumanizing toadying, and walled-and-moated functional silos. Less than a year later I was fed up and ready to leave both the company and the IT field. At the urging of a colleague, I joined a small group of employees who wanted to analyze the business value the IT department added to the enterprise, and determine how to improve things. Along the way we discovered the Agile Manifesto. We enlisted help from ThoughtWorks to teach us agile methods. Long story short, in a few months we had built up an internal agile practice that could handle several projects at once, and we were delivering value regularly. The "business side of the house" loved our group. Line of business managers started to demand that the IT department carry out their initiatives using our methods. Things just got better and better...
...for a while. There are many war stories to tell from this time period. For purposes of this post, let's skip to the end of that three-year run. As soon as they could, IT management "took control" of the agile group. Within a week, they had (a) cancelled the practice of locating technical teams on-site with their internal customers; (b) scattered the agile practitioners around in the traditional organization, to dilute their influence; (c) conjured negative performance reviews of key agile proponents in the organization as a way to encourage them to leave the company; some were even put on probationary status, to pressure them to leave; (d) assigned some of the top performers to dead-end jobs babysitting relatively unimportant legacy apps; another tactic to encourage them to leave; and (e) re-established the waterfall process (decorated with a few agile buzzwords) as the sole way of delivering projects for internal customers. One year later, there was nothing left of "agile" but the word, much as the Cheshire Cat faded away and left only his smile behind.
Why would people do this, to the detriment of their own company, the organization whose success fuelled their own bonus programs and retirement plans? At the time, the reasons were incomprehensible to me. I chalked it up to incompetence or, in some cases, maliciousness. In hindsight, the reasons are obvious: We had embarrassed IT management. Where they insisted something would be too costly, too time-consuming, or technically infeasible, we went ahead and did it at reasonable cost, in a short time, and with technical excellence. We demonstrated that everything the traditional IT managers believed to be true of software development was wrong. We went around them, and worked directly with the lines of business. We ignored their standards and procedures. We openly disrespected the chain of command, the org chart. We went around the HR department, as well, and recruited our own people to join the agile teams. This made HR feel disrespected, too, since our behavior demonstrated that we did not trust them to screen job candidates adequately. To add insult to injury, we did all that while wearing blue jeans and working four-day weeks...at a traditional financial institution with a very conservative organizational culture.
It was all amazingly productive and added tremendous value to the enterprise. It was also absolutely intolerable. One year after IT management "took control" of agile, only 4 out of 60 agile practitioners were still employed there. Those four had settled back into quiet, secondary roles in the organization where they could accumulate long-term retirement benefits as long as they kept their mouths shut and didn't make any waves. Everybody (who was anybody) was happy again.
Local process optimization. Our group was physically and philosophically separate from the rest of the IT department. It grew up as a skunkworks operation and only survived as long as it did because our customers wouldn't allow IT management to crush it openly, since they were realizing value from our work. We did not improve the company's IT processes across the board. We only optimized our own work, in isolation from the other 1,300 IT professionals in the company. We were so fully isolated that most of those people probably never even realized anything unusual was going on. If you asked them about it today, I'll bet they wouldn't even know anything like this had happened in 2002-2006. The business managers who learned about the value of agile methods during that time moved on to other jobs, or retired. The organization itself has not gained or retained any deep tacit knowledge of agile practices or their value, and has not fundamentally changed its culture.
Insensitivity to emotional factors. We simply ignored most of the individuals whose support we needed in order to achieve long-term process improvement. They responded in a perfectly normal human way: They bided their time until they could crush the agile initiative, and then they did so quickly and firmly. They "punished the innocent" and made sure anyone who had been part of the agile group either sat down and shut up, or left the company. It's possible that those managers left the company eventually, too, and that a new crop of managers has experimented with agile or lean methods again since 2006. If so, they have not benefited from anything we did in 2002-2006, because internal history was rewritten to expunge all memory of it. This huge waste, this huge loss, is a direct result of our insensitivity to the emotional needs and politically-motivated fears of key management personnel. We succeeded and we failed with the same stroke.
Case 2
Early in 2009 I was engaged, along with another agile coach, to help a certain company improve its business processes and to help a couple of technical teams get up to speed with Extreme Programming practices. This is a smaller company than that in Case 1, with under 1,000 employees working in only two geographical locations. We were working with teams and stakeholders at one location, and most of the technical teams worked at the second location. There, they had been practicing Extreme Programming for five years already, under the guidance of an internal agile coach and with the support of their own management team.
The two of us quickly determined that the main problem in the organization was not the technical teams' use of Extreme Programming, but rather backlog grooming upstream from the technical teams. We focused mainly on that, and did not pay any attention to the teams at the second location. In the meantime, I coached the technical teams at the first location in agile methods, which were new to them.
Long story short, in about three months the teams that were new to agile methods were applying more-advanced techniques and delivering value better than the teams with five years of XP experience behind them. They were using methods I mentioned earlier, which represented state of the art practices circa 2009, and were functioning at a hyperproductive level as compared with the organization's general IT performance. They were applying agile values and principles throughtfully and crafting their own practices accordingly, rather than following "rules" from a book by rote. After five or six months, however, they were floundering. The team was far out of sync with the pace of work the rest of the organization could sustain.
Within their own workstream, they were able to pull work much faster than the stakeholders could keep the backlog up to date. This caused a lot of dissonance in customer collaboration. It also forced the team to revert to story sizing. Whereas relative sizing was a state of the art practice in 2002, as described in Case 1, in 2009 it is no longer a state of the art practice. Here, a team that was already skilled at commitment-based short-term planning without fine-grained estimation had to drop back a step and start sizing user stories, simply because the stories were not in appropriate shape for them to pull; they did not have any idea how big the stories might be or any inkling of potential technical risks until the very day they were expected to start working on them. This was a source of frustration for the team, and also a source of stress for stakeholders, who felt as if they were "failing" in some way although they really didn't have time to do any more than they were already doing.
Local process optimization. We had a single team functioning at a level of performance that was out of sync with the rest of the organization. We introduced change too rapidly and in too localized a way, and caused friction in the organization. This particular team was capable of absorbing that much change that quickly, but because they were out of sync with the rest of the organization their very success created stress and frustration for them and their customers. When they had to back off from leading-edge practices, it made them feel as if they had "failed" with agile, in some sense. That is especially unfortunate in view of the fact that even after they slowed down they were operating at a level of agility beyond that of most agile teams in industry.
Insensitivity to emotional factors. In hindsight, I think this was the area where we failed most spectacularly. We failed on two levels: First, we drove change forward as rapidly as possible and ignored the emotional impact of change on the team and their immediate collaborators; second, we ignored the people at the second location (at least, initially), who had built a sense of personal pride and an internal reputation in the company as agile experts in the course of five years of nurturing XP teams there.
The team members could see that they were performing better than they had done previously, yet they were not happy about it. They didn't have time to be happy about it. They had lost their individual cubicles in favor of a collaborative workspace geared for pairing; the Furniture Police did not consider all the aspects of an effective team work space when they set up the team area, and did not consult the team about their needs. As it turned out, this single factor had a significant impact on team morale. The last time I checked in with them, they had lost one senior team member and two others were emotionally "checked out," showing up for work but not really feeling engaged. They are still delivering value at a higher rate than any other team in the company, but they are doing it mechanically, without a deep personal investment in the process.
The in-house agile experts felt threatened by the new ideas the consultants brought. We caused them to feel that way by failing to engage them properly from the beginning. With the consultants now gone, the in-house agile experts have reasserted themselves as the thought leaders for process improvement. They are imposing "rules" such as they use with their own teams, based on state of the art agile circa 2004. That is when they started to introduce XP, and it is the style of agile work they understand best. In hindsight, I think we should have approached change by engaging these individuals from the start and introducing change at a far more gradual pace. We didn't help them understand how agile practices have evolved in the industry since 2004. As it is, the pattern that is emerging is similar to that at the company in Case 1: A brief flurry of localized extreme high performance followed by a breakdown and re-establishment of the organization's previous equilibrium state. Cold comfort: At least in Case 2 the previous equilibrium state is superior to that in Case 1.
Thinking back over these experiences leads me to another question: One year after the boastful consultants have left a client organization, what does the process look like at that organization? Did the changes "stick?" Consultants often quote client testimonials in their marketing material. What would the clients say one year, two years, or five years after they released those quotes?
The question reminds me of a conversation I had with the CEO of a previous employer, an agile services consultancy. There was one client in particular we used prominently in our marketing materials, who had given us a glowing recommendation. Two years after that successful project, we were reviewing the list of upcoming engagements and prospective engagements one day and I noticed we had never obtained any follow-on work from that particular happy client. I asked him why, if the client had been so happy with the results, they had never brought any more work to us.
His answer was along the same lines as this blog post. Our team had performed at a pace that was far out of sync with the client organization. Our customers could not keep the backlog up to date fast enough to feed our team meaningful work. They threw stories together helter-skelter just so they wouldn't be paying us an hourly rate to have our people sit around and wait for something to do. The experience was so stressful for them that they decided they weren't suited for agile work, and they reverted to traditional methods and traditional services firms.
It occurs to me that we may have served our client better had we adjusted the pace of change and the demands for collaboration so that they were more closely aligned with the client's ability to cope with change. The initial project may have taken a bit longer, but we probably would have won follow-on work from the client because they would have learned how to collaborate effectively with an agile services provider. Instead, they came away feeling as if this "agile" stuff can produce impressive results, but it just isn't for them.
Turning all the knobs up to eleven while ignoring the human costs of doing so may not be the definition of success, after all. Beware of consultants bearing gifts of hyperproductivity.
I recall a comment from Ron Jeffries on the Extreme Programming discussion list some time ago, to the effect that if you haven't gone all the way with agile then you don't have a frame of reference to understand when and how to back off from particular practices without losing the value of the agile approach. (Of course, his comment wasn't as verbose as that. Verbosity is an affliction of mine.) Anyway, I've gone all the way more than once and in more than one context. I think I have a frame of reference to understand when and how to back off. I recently had an initial visit with a new client. Until I understand their organizational culture and their values pretty well, I won't be turning those knobs anywhere near eleven.
