NOOP.NL

Scientific Case for Bottom-Up Rulemaking

In his book Hidden Order: How Adaptation Builds Complexity computer scientist and psychologist John Holland describes a generic pattern for rulemaking in complex adaptive systems (like businesses and software projects).

Performance System
The first part of the pattern is what Holland calls a performance system. It consists of a potentially large number of stimulus-response rules, where the rules are meant to act upon messages received from the environment (or from other rules). The result of applying those rules is that a number of new messages are emitted, either to follow-up rules, or to the external environment.

Being a software developer my mind is filled with plenty of rules for building software. The input that I receive from the environment consists of the things my colleagues are doing (or the things they are only saying they’re doing), the code that I am working on myself, the requirements from the customer, the features and restrictions of the development environment, etc. The many messages from the environment are evaluated, in parallel, using hundreds if not thousands of rules in my mind, both consciously and unconsciously, which results in one or more new actions, like code to be written, changes to existing code, conversations with my colleagues, or discussions with the customer.

I know, this all sounds pretty obvious. But the key concept here is that the performance system consists of many potentially conflicting rules, where different rules are triggered under different circumstances, given different messages from the environment. It is as if the performance system is an ecosystem of rules, where rules are both competing and collaborating with each other, and the “fittest” rules are the ones contributing most effectively to the whole complex adaptive system.

Credit Assignment
The second part of the pattern is called credit assignment. Rules that appear to lead to improved performance of the entire system are credited, which increases their strength within the performance system. Rules that were triggered and subsequently failed to deliver beneficial effects, or even appeared to hurt the total system, will see their strength reduced. The strength of each rule determines the chance of being triggered the next time, for similar input messages.

Credit assignment assures that experience is built up in the system, by strengthening some rules and weakening others. The rules together form an internal model of what the world outside looks like and how the system needs to respond to it. And when the environment changes, strong rules will start failing and weak ones could succeed more often than before. This enables the performance system to adapt to new situations, and to continuously correct and tune its internal model.

Rule Discovery
The last part of the pattern deals with rule discovery: where do the agents in a complex system (or software developers in a project) get their rules from? Holland describes how new rules can be constructed from existing rules by recombination of building blocks. This is essentially how DNA works: by recombination of existing genes and their alleles.

Holland was one of the first to create evolutionary models for rule-based decision-making, which earned him a reputation as the father of genetic algorithms. He not only convincingly described how these performance systems are an interesting conceptual model for complex adaptive systems, but he also showed that they can be implemented easily to create evolutionary algorithms with powerful problem-solving capabilities.

Why am I writing all this?

For one reason only: to help people understand that rule-making is something that must happen bottom-up. And not top-down.

Science proves it.

(image by Syntopia)

This article will be part of the book Management 3.0: Leading Agile Developers, Developing Agile Leaders. You can follow its progress here.

 Twitter -  Subscribe - Newsletter - LinkedIn - SlideShare

Latest, greatest and favoritest posts: Management 3.0: The Era of Complexity Self-Organization vs. Emergence Self-Organization vs. Anarchy

Management 3.0: The Era of Complexity

I know. Management 3.0 is a silly name. We already have Web 2.0, Government 2.0, Project Management 2.0, Enterprise 2.0, and RSS 2.0. And interestingly enough, Gary Hamel’s Management 2.0 was one of the last to jump on the "2.0" bandwagon. So why do we need a "3.0" version number for management?

Actually, we don't.

However, I believe that the "3.0" number conveys the right message about the direction that management is taking in the 21st century. The Management 3.0 name was first used by Terrence Seamon one year ago, though it hasn't been picked up yet by others. Now I think it is time for a breakthrough. Allow me to explain:

Management 1.0 = Hierarchies

Some people call it Taylorism or scientific management, while other people call it command and control. But the basic idea has always been the same: an organization is designed and managed in a top-down fashion, and organizational power is in the hands of the few. Those at the top of the hierarchy have the highest salaries, the biggest egos, the biggest bonuses, and the most expensive chairs. Those at the very bottom have virtually nothing, little money, few responsibilities, and very little motivation to do a good job.

The shareholder value movement, promoted by top economists and business men like Milton Friedman and Jack Welch, was a perfect fit for this kind of management, because it gave shareholders a single wringable neck: the CEO. In exchange for his dangerous position the CEO was allowed to play with risky bonus schemes that had far more effect on personal wealth than organizational performance. As a side effect, and minor inconvenience, these dangerous bonus schemes also triggered a worldwide financial implosion. Oops!

I think we can safely conclude that Management 1.0, even though it is still the most widespread version, used in organizations all over the world, has a number of serious flaws. It is old and outdated and in need of an upgrade.

Management 2.0 = Models

Plenty of smart people realized that Management 1.0 didn't work well enough on its own to produce any good results. And so they created numerous add-on models, most of which dealt with systems thinking and process improvement. Some of these models gained a semi-scientific status, like the Balanced Scorecard, BPR, Six Sigma, Theory of Constraints, and Total Quality Management. Being add-ons to Management 1.0 the models still assumed that organizations were managed from the top. And so they were created to help those at the top of the hierarchy to better design their organization. Sometimes the models worked, sometimes they didn't. Some had a flimsy affair with science, while others had more affinity with religion.

At the same time there were different kinds of models that focused not on systems and science but on craft and art. Ideas like the 7 Habits, the 21 Laws of Leadership, and Good to Great listed basic values, principles, and guidelines for top managers, and then told them to practice and build experience. Again, these models were sometimes right, and sometimes not. Some models were created from research, others from personal experience, but they all had one thing in common: they aimed primarily at the ones with all the power, at the top of the hierarchy.

But working with the add-ons of Management 2.0 means that organizations still suffer from the inherent problems of Management 1.0. The basic architecture for problem solving has not changed. The organization is still seen as a hierarchy!

Management 3.0 = Complexity

In the 80's and 90's we saw the birth and rise of complex adaptive systems theory, or complexity science, first applied to mathematics and biology, and later to economics and sociology. It was a major breakthrough in systems thinking, bridging all sciences, and making some deep thinkers very happy. Stephen Hawking thought it was so important, that he called the 21st century the Century of Complexity. The first decade of this century saw a number of publications, like The Tipping Point, The Black Swan, and Freakonomics, all of them borrowing and popularizing concepts from complexity theory. And management researchers, including Ralph Stacey and David Snowden, found out that social complexity had some interesting things to say about organizational management.

One important insight is that all complex systems are networks and self-organizing systems. People may like to draw their organizations as hierarchies, but that doesn't change the fact that organizations are actually networks. Social networks to be precise. And every time you want to solve a problem in an organization, you must treat it as a network. Because that's what it is. A self-organizing network, not a hierarchy.

Second, complexity theory proves that, ultimately, all models are wrong. No matter how many diagrams, lists, or figures you throw at an organization, in the end they will always be wrong. Potentially useful, yes. But ultimately wrong. And thus, the models of Management 2.0 will always fail. Mind you, there's no need to throw them all away! But neither is it wise to treat any of them as scientific truths, or religious dogmas.

Third, based on both scientific and empirical study, social complexity shows us that management is primarily about people and their relationships. Everything else in an organization pales when compared to the importance of people and their relationships!

Furthermore, we can now conclude that management is science and craft, models and experience. And we can learn that leadership is not "higher than" management. Nonsense! The job of a manager is to be leading and governing the self-organizing systems. To ensure that people are empowered and aligned.

Conclusion

Management 3.0 is just a silly name. Some of you will disagree with me and argue about version numbers. That’s ok, because version numbers are not important.

And many of us already knew that hierarchies are bad and that networks are real. That management is about empowering people, and also giving them a direction. That models only work depending on the context. And that "leadership" is just a trendy name for managers doing the right thing and doing things right.

But network thinking adds a new dimension to our existing vocabulary. The concept of social complex systems makes us realize that we are all participants in self-organizing systems. All of us have to lead and rule in some ways. We are all "3.0 managers".

I think it's nice to have a new name. Names and versions can be powerful. The "3.0" version indicates that management is taking charge again. It usually takes Microsoft three versions of a product to get things right and make their product actually usable. Thanks to social complexity management has, in its third incarnation, finally found a solid scientific foundation for the future. None of the earlier models have to be unlearned, because many of them are still usable. We just have to replace assumptions of hierarchies with networks, and complex systems thinking. Because the 21st century is the Age of Complexity. And Management 3.0 will be the Era of Complexity.

This article will be part of the book Management 3.0: Leading Agile Developers, Developing Agile Leaders. You can follow its progress here.

 Twitter -  Subscribe - Newsletter - LinkedIn - SlideShare

Latest, greatest and favoritest posts: Yes, Good Managers Are Manipulators Managers Are Not Game Designers! Management: A Deceptive Metaphor

Managers Are Not Game Designers!

Conway’s Game of Life has always been a classic example of a complex system. In designing his game John Conway found that some sets of rules were too ordered, while other sets were too chaotic. It took him a while to find a set of rules that was nicely balanced, resulting in systems with complex behavior. Not too ordered, and not too chaotic.

In an interesting Wired article I read that Klaus Teuber took a similar approach when he was designing The Settlers of Catan, one of the most popular board games of all time. Teuber continuously played the game with his family, reconfiguring it again and again, changing the rules, the cards, and the pieces. It took him four years to find a set of rules that was nicely balanced, and that allowed for complex gameplay and heated family competition.

What sets games apart from living systems is their lack of the “adaptive” part. Games do not change their own rules while they are in progress. But living systems do. Complex adaptive systems are systems that are able to find their own way toward that sweet spot of complexity, right between order and chaos, where life blooms and creativity thrives. Scientists call it the edge of chaos. But they also could have called it the edge of order, because it is at the edge between chaos and order where we find complexity. (Never expect a scientist to come up with a name that actually makes sense.)

The question is then who or what is tuning the rules in an organization so that the organization moves towards the edge of chaos, being neither too ordered nor too chaotic? A common misconception (and yes, I plead guilty here) is that managers are somehow responsible for this.

This amounts to a negation of self-organization and emergence. If managers are choosing what “emerges”, then it is not emerging. If they have a blue-print guiding self-organization then it is not self-organization – that is, it is not agents acting purely on the basis of their own local organizing principles, but rather on the basis of simple rules chosen for them. [Ralph Stacey – Complexity and Management]

It is tempting to think of managers as game designers, like John Conway and Klaus Treuber. When the manager chooses the wrong set of rules for the organization, then the system is either too bureaucratic too chaotic. (And if they’re really screwing up it will be dead.) Metaphorically this view is interesting, but scientifically it is hogwash.

This approach completely loses the sophisticated concept of self-organized criticality, in which a system evolves to the edge of chaos through its own internal dynamic, where self-organization produces potentially novel strategies, again through the system’s own internal dynamic. […] The edge of chaos is a dynamic that occurs when certain parameters fall within a critical range – for example critical rates of information flow, degrees of connectivity, and diversity between agents [Ralph Stacey].

Every organization is a complex adaptive system. It’s like a game in which the rules are changed on the fly, and where the job of designing the game is delegated to the participants themselves. Your job as a manager is not to create the right amount of rules in the organization. Your job is to make sure that the people are able to create their own rules together. And it’s their collaborative effort that makes the system find its own way to the edge of chaos. (Or the edge of order, if you prefer.)

As complexity scientist Ralph Stacey pointed out, self-organization takes care of the edge of chaos when certain parameters fall within a critical range. The manager is not a game designer. He does not need to concern himself with the low-level rules of the game. He configures the high-level parameters, like diversity of team members, information flow between people, and connectivity between teams.

In aligning groups of people, the first responsibility of a manager is the construction of a self-organizing system. Don’t try to be John Conway or Klaus Treuber. You may define the squares on the board, but not the rules of the game. When you take rule-making into your own hands, you will significantly influence and frustrate self-organization, and then creativity, innovation and adaptability in the system will suffer.

Don’t be tempted.

This article will be part of the book Management 3.0: Leading Agile Developers, Developing Agile Leaders. You can follow its progress here.

 Twitter -  Subscribe - Newsletter - Bookmarks

Latest, greatest and favoritest posts: If You Want Something Done, Practice Your Patience Leader vs. Ruler: Which One Are You? Great Managers Have No Secrets

Management: A Deceptive Metaphor

The observation from scientists is that rules in a complex system must be tuned for the system to be both stabilizing, lively and creative. A different set of rules leads to a different system, with different behavior.

In an influential paper, Stephen Wolfram, founder of the first journal on complex systems and known for his work on Wolfram Alpha (a “computational knowledge engine”), once proposed a classification scheme which divided cellular automata (mathematical systems consisting of “cells” that can be “dead” or “alive”) into four categories:

• Class I: These are the systems with “doomsday rules”. No matter what pattern of living and dead cells you start out with, everything just dies within a few generations.
• Class II: These systems are a bit more lively, but not much. Each initial pattern quickly collapses to a set of very boring static configurations.
• Class III: These systems are at the opposite extreme: they are too lively. Each initial pattern in the system results in total chaos, with no configuration stabilizing and nothing being predictable.
• Class IV: These are the systems with a set of rules not leading to dead, static or chaotic configurations. Emerging patterns in this category are lively, creative, often surprising, but also stabilizing.

It should not surprise you that, in dynamical systems, class I and II correspond to order, class III corresponds to chaos, and class IV (of which the Game of Life is a famous example) corresponds to complexity. Given that complexity is usually explained as the region between order and chaos, this means that class IV finds itself between II and III. (This strange way of using numbers makes Wolfram’s computational knowledge engine all the more surprising.)

The same universality classification can be used (or should I say abused?) as a metaphor when distinguishing complex systems themselves. Take the human brain, for example. A class I brain would be dead: nothing happens in the brain. A class II brain could be comatose or catatonic: a state of silence or predictable repetitiveness. A class III brain could be insane or epileptic: displaying unpredictable and uncontrollable behavior. And finally, a class IV brain would be the only one considered lively and healthy. To prevent scientists from classifying my brain as a class III specimen, I must stress that I use this categorization only in metaphorical way.

When classifying organizations some people make a similar metaphorical distinction between ordered, chaotic, and complex organizations. (I assume you will forgive me for ignoring the dead organizations for now.)

• In ordered organizations no creativity and innovation are taking place. Nobody is empowered to make his own decisions. Bureaucracy dictates how all work is to be carried out, and fluctuations in organizational behavior are regular and predictable.
• In chaotic organizations there may be plenty of creativity, but not in a structured and predictable way. There is no emerging order in the organization, meaning that people simply empower themselves in getting things done. And everyone does as he or she pleases.
• Complex organizations find themselves right in between. In a complex organization, people cannot empower themselves. They are empowered by managers, who are faced with the challenge of finding a balance between empowerment and alignment, between self-organization and control.

It is said that a system at the balance between chaos and order, finds itself at the edge of chaos.

 

IMPORTANT: This classification of organizations is not scientific, but only a deceptively useful metaphor. I call it deceptive, because the metaphor has led some managers (including me) to conclude that they are responsible for finding the right balance between order and control. But as we shall see in some of my next blog posts, this conclusion is as common as it is wrong…

 Twitter -  Subscribe - Newsletter - Bookmarks

Latest, greatest and favoritest posts: Self-Organization vs. Emergence Leader vs. Ruler: Which One Are You? The Complex Manifesto for Software Development

So Now You’re an Agilist, What’s Next? (Video)

This is the video of the talk I did at the Agile Eastern Europe conference in Kiev, a few months ago. The talk is called "So Now You’re an Agilist, What’s Next?", in which I talk about complex systems and "laws" for software development. (I performed the same talk at several other conferences.)

 

Note: I have already a couple of conferences planned for 2010: the Scandinavian Developer Conference (March), Norwegian Developers Conference (June), and Agile Eastern Europe conference (October) are among them.

Hope to see you there!

 Twitter -  Subscribe - Newsletter - Bookmarks

Latest, greatest and favoritest posts: Self-Organization vs. Emergence Self-Organization vs. Evolution Self-Organization vs. Anarchy

Emergence in Development Teams

When trying to translate the concept of emergence to teams, we can recognize a host of interesting phenomena. The first is the possibility of collective decision making without central planning. Swarm raids of army ants are reported to be among the largest organized operations carried out by any animal. But not a single ant has a picture of the entire operation in its mind. Likewise, no team member has a complete picture of the whole project. And yet it is common for good plans to emerge from the interaction of team members, where each can only work with incomplete information.

From research into human consciousness we can learn that multiple conflicting views can result in a (seemingly) singular view of the entire system. Daniel Dennett and Marvin Minsky both explained that “one stream of consciousness” is an illusion. According to Dennett there are actually “multiple drafts” of consciousness, which Minksy called the “society of mind”. Our brain resolves these multiple competing interpretations of the world into something that we call one identify, or a “self”. For an illusion it works remarkably well. Similarly, the multiple views of the world in a team can resolve into a singular team view. The team identity is an illusion, and yet it works by having a real impact on their projects. Paradoxically, human consciousness works because of the underlying multiple drafts. And the team identity works because of the underlying disparate views. I am sure some people will be glad to know that their diverging opinions may turn out to be crucial for a team identity to emerge. (Just don't blame me next time you get into a fight.)

It is also known that a system can be more than the sum of the parts. Our brains have a stable alpha wave of between 8-12 Hz. It is an accurate clock, though it is constructed from manny sloppy ones, because all the individual neurons have their own rates of discharging, varying between 8 to 12 times per second. And yet the emergent alpha rhythm is more reliable than that of any of the neurons. Similarly, it is not uncommon for a whole team to perform better than the best performance of any of the individual team members. DeMarco and Lister call this a “jelled team”. It is “a group of people so strongly knit that the whole is more than the sum of the parts. The production of such a team is greater than that of the same people in unjelled form.”

Finally, the nature of emergent properties is often unpredictable. Water, consisting of two hydrogen atoms and one oxygen atom per molecule, is subject to state changes like freezing and cooking. There is nothing in the properties of hydrogen and oxygen atoms predicting these properties of water. It is the same with teams. You cannot predict the behavior of a team by analyzing individual team members separately. The emergent behavior of the team is a result of the interactions between the team members.

The only thing you can predict is that they will always try to undermine your profitability by asking for more time, better education, expensive tools, food, or conferences in luxurious resorts.

(image by feserc)

 Twitter -  Subscribe - Newsletter - Bookmarks

Latest, greatest and favoritest posts: Self-Organization vs. Emergence Self-Organization vs. Evolution Self-Organization vs. Anarchy

Self-Organization vs. Emergence

This is the 3rd post in a trilogy about self-organization. The first one was Self-Organization vs. Anarchy, and the second was Self-Organization vs. Evolution. This one is about Self-Organization vs. Emergence.

When a property of a system cannot be traced back to any of the individual parts in the system, it is called an emergent property. Your personality is an emergent property of your brain. It cannot be traced back to individual neurons. Likewise, fluidity is an emergent property of water, and culture is an emergent property of a group of people.

Three aspects are important for a property to be emergent: First of all, supervenience is the observation that the property will no longer exist if you take away the individual parts of the system. For example, your personality disappears when I remove your neurons. (Relax, I won't try and prove it.) Furthermore, the property should not be an aggregate, meaning that it is not simply the result of adding up the properties of the individual parts. For example, a single water molecule has no fluidity. So you cannot simply add up the fluidities of a billion individual molecules to determine the fluidity of water. Finally, there must be downward causality, meaning that the emergent property should influence the behavior of the individual parts. For example, the culture of a group of people influences the behavior of its members. In short, emergent properties are global (to the system), irreducible, and noticeable (to the parts).

 

The boundaries of the sciences rely on emergent levels. Physics resolves into chemistry, which resolves into biology, which resolves into psychology, which resolves into economics. And each science works with the emergent properties generated by previous levels. It also means that each level gives rise to new laws and new insights. Psychology is more than just applied biology, biology is more than just applied chemistry, and chemistry is more than just applied physics. This is why greedy reductionism doesn't work. You cannot explain forgetting your spouse's birthday in terms of faulty atoms or string theory.

Throughout literature there has been some confusion (and disagreements) about self-organization and emergence. Some scientists have defined one in terms of the other, while others claim they are separate concepts. I tend to agree with Peter Corning in that there may be self-organizing systems without emergent properties, and emergent properties in systems that are human-created instead of self-organizing. But it is just a matter of definition. On this blog I choose to use the term emergence for “organized ‘wholes’ composed of functionally distinct ‘parts’ that produce irreducible combined effects”. Even though this blog itself is not a self-organizing system, the impression it will make on you will definitely be an emergent property: the impression is global to blog, irreducible to the individual words, but quite noticeable if you choose to iniate a DDoS attack at my blog after reading.

(image by MikeBlogs)

 Twitter -  Subscribe - Newsletter - Bookmarks

Latest, greatest and favoritest posts: Self-Organization vs. Evolution Self-Organization vs. Anarchy Your Project Has No Goal

The Danger of Lean: Ignoring Social Complexity

After reading three books about Lean software development I feel slightly uncomfortable with the heavy emphasis the Lean community seems to place on process and economy.

(Transparency: I read Implementing Lean Software Development, Managing the Design Factory, and Lean Software Strategies. They were advised to me by several lean experts.)

The only things these Lean books seem to focus on is reducing queue size, reducing cycle time, reducing variability, reducing work in progress, and reducing batch size. It is all inventory this, and control systems that. No wonder the Lean brand appeals more to higher management than Agile. Somehow the word “people” got lost somewhere. (Note: I’ve been told the people-issue is an important topic in the works by Deming and Jeffrey Liker. But I specifically wanted to investigate software development books.)

Try it yourself! Just pick your average Lean software development book, go to the index, and try to find references to concepts like motivation, creativity, innovation, personality, or fun. You probably won’t succeed. And if the content is there at all, it is drowned out by all the stuff about inventory, processes, control, economics, value streams, efficiency, queues, variability, and waste. (I’ll be honest: the Poppendieck book has a chapter called “People”. But this little gem of information pales in comparison to the volume of economical process jargon in the other books, and the many blog posts about Lean software development.)

And I don’t have a problem with this. All that stuff is indeed very valuable!

However, there is a problem when people see Lean as The Only Way.
Because it isn’t.

Alan Shalloway published a blog post The Dangers of Complex Adaptive Systems, where he tried to analyze where complex adaptive systems (CAS) work and where they don’t. I respect Alan a lot, but his blog post completely misses the point about complexity theory. Most systems in the universe are complex systems, and many of them are adaptive. Debating whether or not CAS works is like debating whether or not biology works. It makes no sense. The system just exists. Whether it is a school of fish, a flock of geese, or a team of software developers, they are all complex adaptive systems. You can debate whether certain survival strategies of these systems work. If the strategies don’t work, the systems will need to adapt, or they go extinct. If they don’t go extinct, then obviously the strategies work! That is all part of the theory of complex adaptive systems.

Complexity theory is just a scientific description of the world. It has no “good” or “bad” qualifiers.

But Lean is a prescriptive model. It tells you what you should do when you try to manage the complex system that we call a software development team. It tells you there are good and bad choices.

And that’s where things can go wrong…

The Incompressibility problem of complex systems tells us that every model (including Lean) is inaccurate:

There is no accurate representation of the system which is simpler than the system itself. In building representations of open systems, we are forced to leave things out, and since the effects of these omissions are nonlinear, we cannot predict their magnitude.
– P. Cilliers (2005), Knowing Complex Systems

When you create a model of a system (like a software project), you have to leave things out. Unfortunately, because of the famous Butterfly Effect, in complex systems every little thing that you leave out can still have huge unexpected consequences. And because complex systems are non-linear, you cannot predict which parts of the real system you can safely ignore.

This leads to a simple, but very important conclusion:

All models are wrong, but some are useful.
– Box and Draper (1969), Evolutionary Operation

Lean software development describes a software project as a pull system with a value stream. That means it is a model of the real system, which is more complex. It is a very, very useful model. Unfortunately, it is also wrong because it is inaccurate. But so are all the other models. And some models are worse than others. (Actually, I think Lean probably scores quite well among all those bad models.)

The Lean model ignores the social part of complex systems. I just reviewed all the diagrams in the three books mentioned earlier. I couldn’t find a single diagram or picture with people in it! What are the consequences of ignoring the people-issue in the model? What happens to people’s motivation when you reduce variability? What happens to people’s creativity when you treat the system as a production flow instead of an emerging design effort? How is the process aligned with people’s personalities? And where is the fun? I can tell you, those three books were about as much fun as my last visit to the dentist. Useful, yes. Healthy, sure. Painful, oh God, yes! They certainly didn’t make me look forward to being part of a lean software development team. (I fondly remember the first books about Agile software development. They were about people. And they were fun. There’s certainly more humor on the Agile side…)

Again and again I see emphasized that Lean software development is based on systems thinking. That is correct. And that is exactly the problem. Just have a look at this interesting diagram created by David Snowden, a complexity expert (the red text in the diagram is mine):

Systems thinking is a great tool, but system thinking ignores the unordered parts of the world. The messy bits. The unpredictable stuff that always comes as a surprise, no matter how much time you’ve spent drawing process diagrams and analyzing value streams. In software projects, the proper foundation for management is not systems thinking but social complexity. Social complexity is complexity theory, but applied to people instead of fish and geese.

The danger of Lean is that it overemphasizes process and economy. This makes it deceptively easy for managers to embrace Lean (instead of Agile) and to forget all about people-issues. Again. And the people-over-process principle will be ignored. Again.

Social complexity is the only proper foundation for management of software teams. Systems thinking is outdated. Ralph Stacey, another complexity expert, threw systems thinking in the trash bin. And rightfully so. Goodbye, and good riddance. Systems thinking has served its purpose, but science has moved on.

So yes, Lean has some fantastic ideas, as there are great ideas in Agile too. But neither Lean nor Agile is a complete model of a complex system. The only accurate model of a complex system is the system itself. Which means you must focus on the entire system, including the people. And not only on value streams and waste reduction. It takes a complex mind to recognize the dangers of incomplete models.

I’m writing a book about this stuff. Did I tell you that? I think some managers are going to need it. The book will be useful, and healthy. And fun.

Note: I could just as well have named this post “The Danger of Agile”, focusing on the things that Agile ignores. But there are already so many posts bashing Agile, Scrum, and XP. It wouldn’t have been interesting to write. I’m human too. I want my work to be fun. I think increasing variability is more fun than reducing it.

(picture by Foxtongue)

 Twitter -  Subscribe - Newsletter - Bookmarks

Latest, greatest and favoritest posts: ScrumButs Are the Best Part of Scrum The Complex Manifesto for Software Development Simple vs. Complicated vs. Complex vs. Chaotic

What's in a Word? From Science to Software

In my previous post (Are We Abusing Science?) I wrote that we need to be careful when applying terminology from the sciences to software development. When we don't fully understand scientific terms, while attempting to use them in another domain, we run the risk of being scolded and ridiculed by scientists for our abuse of their cherished terminologies. And we don't want to be shot at with the Large Hadron Collider, now do we?

There are three ways of "borrowing" terms from other disciplines:

Metaphors
We use metaphors to make people understand concepts more easily, without having to make sure that descriptions are accurate and consistent in meaning. For example: in chaos theory the butterfly effect is a metaphor for the unpredictable consequences of small changes. It is a fine example of a metaphor in science. (In fact, in order to get their work funded, scientists often resort to metaphors to make their work somewhat less incomprehensible.)

There are so many metaphors being used in software development that it is hard not to think of any. Bugs, smoke testing, spikes, waterfalls, factories, server farms, black box testing, binary trees, software crashes... Shall I stop already? Such metaphors are very useful for furthering the understanding of complex ideas. But there's a problem with them as well: they always fail at some point when taken too far, and this could result in the metaphors leading someone down false paths.

Strict Terminology
A different way of copying terms from one discipline to another, is to make sure that their meaning is strictly identical in all cases. For example: non-linearity means exactly the same in all sciences, whether it is physics, mathematics, economics, or anything else. (Though whether economics really is a science is still open for debate as far as I'm concerned.)

Perhaps the best example in software development is the word system. Though the word has its origins outside of computer science, the meaning is the same in all disciplines: a set of interacting or interdependent entities forming an integrated whole.

Strict use of terminology is the preferred approach of dogmatic scientists. And I would say that they tend to defend that position religiously, but that would be like practicing suicide with my own words.

Loose Terminology
Sometimes neither metaphors nor strict terminology are helpful. A third approach to applying terms from one discipline to another is to do this more loosely than the strict approach requires. In such a case a term will still carry more or less the same meaning, but it will be adapted, extended, or narrowed down, to better fit the new domain. For example: business management has copied the term culture from anthropology. What a business culture means for a business is pretty much the same as what a social culture means for a society. But there are some important domain-specific differences. Likewise, the word strategy in business strategy has been borrowed (and then adapted) from the military.

Even more interesting: the science of game theory has borrowed the term strategy for evolutionarily stable strategies. Here it appears that scientists too know how to borrow and bend terminology to their advantage. And I'm not one to blame them.

For our own domain, a fine and familiar example would be the word architecture: software architecture, systems architecture, information architecture, and several other kinds of architectures, all have their own specific meanings, slightly adapted from the original word which deals with the design of buildings.

And That's How We Do It
The loose application of terminology, through the borrowing and bending of words from other disciplines, enables us to inherit meaning from elsewhere, while remaining flexible enough to modify that meaning to match our own domain.

It also requires scientists to be flexible enough not to accuse us of abuse of science. But, considering that scientists so often apply metaphors and loose terminology themselves to make their work understandable, I am quite sure we will be able to get away with it.

(image by powerbooktrance)

 Twitter -  Subscribe - Newsletter - Bookmarks

Latest, greatest and favoritest posts:
Self-Organization = Anarchy (Part 1)
Simple vs. Complicated vs. Complex vs. Chaotic
From Complexity Theory to Practical Management

Are We Abusing Science?

In agile software development we regularly hear references to scientific terms such as self-organization and emergence. Next week, when I’m at the Agile 2009 conference in Chicago, I am certain to hear people throwing around terms like self-organizing teams and emergent design.

As I’ve indicated before, these words have their origins in complexity science (plus related sciences). And this happens to be one of my favorite topics.

“For example, an ant colony, the brain, the immune system, a Scrum team, and New York City, are self-organizing systems.” – Ken Schwaber, Agile Software Development with Scrum

“At the heart of complex adaptive systems theory’s relevance to software development is the concept of emergence, and the factors leading to emergent results.” – Jim Highsmith, Adaptive Software Development

“Scrum is not a methodology, a defined process or set of procedures. It's an open development framework. The rules are constraints on behavior that cause a complex adaptive system to self-organize into an intelligent state.” – Jeff Sutherland, Shock Therapy (presentation)

OK, this is all mighty interesting. But… are agilists right in applying complexity science to software development? Do the complexity scientists themselves agree that words like self-organization and emergence not only apply to ant hills, the brain, and the immune system, but also to Scrum teams?

Well, let’s see what scientists have to say about people like us borrowing their scientific terms:

“Abuse of science is…

1. To use scientific terminology without bothering much about what the words actually mean.
2. Importing scientific concepts without giving the slightest empirical or conceptual justification.
3. Displaying superficial erudition by shamelessly throwing around terms where they are irrelevant.
4. Exhibiting a veritable intoxication with words, with a superb indifference to their meaning.”

Sokal, A. and Bricmont, J. (1998), Intellectual Impostures: Postmodern Philosophers’ Abuse of Science

OK, I cheated a little. The rant above was not directed at agilists using (or abusing) complexity science, but to people in general abusing various sciences. Still, it makes one think. And to really hammer it in I have another quote here that hits somewhat closer at home:

“Not unexpectedly, the complexity gurus are most upset with how complexity science terms are loosely, if not metaphorically, defined and tossed in managerial discourse – one [guru] goes as far as to suggest that the books offer many insights for managers, but one should simply black out all references to complexity science.”

Maguire, S. and McKelvey, B. (1999), Complexity and Management: Moving from fad to firm foundations

Ouch!

Yes, I cheated again. This rant was directed at management literature abusing terms from complexity science, not agile literature. But... we are warned.

We have to be careful when carrying over terms from complexity science to other disciplines, including management and software development. For example, when a small issue in a software project unexpectedly turns out to have big consequences, it is all too easy to say that this is a typical example of the butterfly effect. But, without really understanding what the butterfly effect actually means in a scientific context, we might be making ourselves a laughing stock among chaos theorists and complexity scientists!

So, is the term self-organizing team abuse of science?

And is the term emergent design abuse of science?

Personally, I don’t think so. But it would be good to really think things over for a while.

In the next few weeks and months I intend to write some more about ideas and concepts found in complexity science, that we might be able to apply to managing software development. And though I really do have a veritable intoxication with words, I intend to do this with proper regard to their scientific meaning and by providing ample justification.

And, of course, at the Agile 2009 conference I will be talking about these topics too.

If you’re there I hope you will join me!

(images by practicalowl and Knut Burmeister)

 Twitter -  Subscribe - Newsletter - Bookmarks

Latest, greatest and favoritest posts:
Self-Organization = Anarchy (Part 1)
Simple vs. Complicated vs. Complex vs. Chaotic
From Complexity Theory to Practical Management