Agile Management | NOOP.NL

Complexity versus Lean

This is the presentation I made for the LESS2010 conference in Helsinki. I will repeat it again at XP Days Benelux.

Hope you like it.

Update: I was very happy and surprised to hear that the organizers and audience had selected my presentation as the best of the conference. I came there with the intention to provoke the Lean crowd. And they thanked me for it. Apparently, the Lean community is very forgiving. :)

(picture by Bernd Schiffer)

Complexity versus Lean

View more presentations from Jurgen Appelo.

Simplicity: A New Model

Note: This article will be is part of the book Management 3.0: Leading Agile Developers, Developing Agile Leaders.

Ah, simplicity.

We all seem to want it, but we rarely seem to get it.

Many experts have discussed simplicity and complexity. But their contributions often confuse various terms, which hasn’t led to a simplification of the discussion itself.

Here is my attempt to clear things up a little…

What is simplicity?

Simplicity usually relates to the burden which a thing puts on someone trying to explain or understand it. Something which is easy to understand or explain is simple, in contrast to something complicated. - Wikipedia

If you’re going to discuss simplicity, it is important to know the difference between complex and complicated. Not knowing the difference means you might apply exactly the wrong approach to the right problem. (Or, the right approach to the wrong problem.)

I believe the difference needs to be explained using two dimensions. The first dimension is about the structure of a system, and how well we are able to understand it:

• Simple = easily understandable;
• Complicated = very hard to understand.

The second dimension is about the behavior of the system, and how well we are able to predict it:

• Ordered = fully predictable;
• Complex = somewhat predictable (but with many surprises);
• Chaotic = very unpredictable.

The Structure-Behavior Model

My underpants are simple. I found it easy to understand how they work. But my watch is complicated. If I took it apart. it would take me a long time to understand its design and its components. And yet, neither my watch nor my underpants hold any surprises. (At least not for me.) They are ordered, predictable systems.

A three-person software team is simple too. It takes only a few meetings, dinners, and beers to get to know everyone on a team. A city is (usually) not simple but complicated. It takes taxi drivers years to know all its streets, alleys, hotels, and restaurants. And yet, both teams and cities are complex. No matter how well you know them, there will always be surprises. They are predictable to a degree, but you never know for sure what will happen tomorrow.

A double pendulum (two pendulums attached to each other) is also a simple system. It is easy to make and easy to understand. And yet, it undergoes unpredictable chaotic motion due to a high sensitivity to the initial setup of the pendulum. And stock markets are also chaotic. They are by definition unpredictable, or else everyone would know how to make money on stock exchanges, and the whole system would collapse. But, unlike pendulums, stock markets are also extremely complicated. The many different businesses and types of financial properties and transactions make them utterly incomprehensible for a simple guy like me.

Complicated refers to a system’s construction being too intricate to understand, unless you’re an expert, while complex and chaotic refer to a system’s behavior, which is unpredictable to a small or large degree.

What is complicated is not necessarily complex, like two cars in a garage. And what is complex need not be complicated, like two people in a bedroom. (But these people’s behavior in their bedroom can be quite unpredictable.)

• Simplification is the act of making the structure better understandable (moving it from top to bottom in my model.)
• Linearization is the act of making behavior better predictable (moving it from right to left in the model.)

Unfortunately, linearization is (in laymen’s terms) usually confused with simplification. And that’s where the complications start…

How Does This Differ From Other Models?

Cynefin is a model devised by knowledge management scholar David Snowden. It describes a typology of contexts using four domains: Simple, Complicated, Complex, and Chaotic, and is used to guide approaches to decision-making and policy-making.

However, Snowden doesn’t distinguish between the structure and the behavior of systems. This can make discussions a bit complicated (not complex).

Management professor Ralph Stacey created something similar, called the Agreement & Certainty Matrix. It shows Simple, Complicated, Complex, and Anarchy (Chaos) as four areas based in two dimensions: the degree of agreement and the degree of uncertainty.

This model suffers from the same problem. I don’t see complicated and complex as two discrete domains. Instead, I see them as parts of different dimensions. That’s why my Structure-Behavior Model identifies six domains instead of four. And some systems (like cities) can be both complicated and complex.

OK, Now We Can Revisit Simplification

I believe my Structure-Behavior Model is able to simplify discussions around simplicity, and to clear up some misunderstandings…

Everything should be made as simple as possible, but no simpler. – Albert Einstein

With this quote Einstein meant that a system must be made understandable, which means moving it vertically, from the top of the model to the bottom (simplification). However, his addition “but no simpler” maps to the behavior of the system. Einstein warned not to change the system horizontally, because that would change the kind of system (linearization.)

Simplicity is a myth whose time has past, if it ever existed. – Don Norman

In an inspiring article Don Norman discussed the value of having more features in a product, instead of fewer. More features means different behavior, and (often) also a different structure. In my diagram it is both a horizontal and vertical issue. (For example: Google just added Priority Inbox to GMail. This made GMail’s behavior more complex for me. It also complicated the user interface, but I still seem to understand it well enough.) Unfortunately, Don Norman used the term simplification both for linearization of behavior (horizontally) and simplification of structure (vertically).

And so Don complicated his message, which is exactly why many people didn’t understand him. Maybe it would have helped if Don had used pictures:

The goal of visual thinking is to make the complex understandable by making it visible, not by making it simple. - Dan Roam

In his bestselling book The Back of the Napkin Dan Roam suggests to use pictures to make things understandable by drawing pictures. He clearly refers to moving things from complicated to simple (vertically). However, his warning “not to make things simple” is, again, a confusion of terms. What Dan means is that pictures should not change the complexity (behavior, meaning) of something, because that would mess up people’s ability to predict what the pictures are trying to say.

And so…

Yes, by all means, simplify everything that is hard to understand!

But no, you may not want to linearize (“simplify”) something, because the reduced behavior of what you offer may not be what your user had expected.

Thanks for reading. I hope it was simple enough to understand.

Jurgen is an experienced author, trainer, and speaker. Why don’t you hire him to add some spice to your company event or seminar?

 Twitter -  Subscribe - Newsletter - LinkedIn - SlideShare

Latest, greatest and favoritest posts: When You Are Rejected Agile Is a Risk Management Strategy Where There's People, There's Problems

Scientific Case for Bottom-Up Rulemaking

In his book Hidden Order: How Adaptation Builds Complexity computer scientist and psychologist John Holland describes a generic pattern for rulemaking in complex adaptive systems (like businesses and software projects).

Performance System
The first part of the pattern is what Holland calls a performance system. It consists of a potentially large number of stimulus-response rules, where the rules are meant to act upon messages received from the environment (or from other rules). The result of applying those rules is that a number of new messages are emitted, either to follow-up rules, or to the external environment.

Being a software developer my mind is filled with plenty of rules for building software. The input that I receive from the environment consists of the things my colleagues are doing (or the things they are only saying they’re doing), the code that I am working on myself, the requirements from the customer, the features and restrictions of the development environment, etc. The many messages from the environment are evaluated, in parallel, using hundreds if not thousands of rules in my mind, both consciously and unconsciously, which results in one or more new actions, like code to be written, changes to existing code, conversations with my colleagues, or discussions with the customer.

I know, this all sounds pretty obvious. But the key concept here is that the performance system consists of many potentially conflicting rules, where different rules are triggered under different circumstances, given different messages from the environment. It is as if the performance system is an ecosystem of rules, where rules are both competing and collaborating with each other, and the “fittest” rules are the ones contributing most effectively to the whole complex adaptive system.

Credit Assignment
The second part of the pattern is called credit assignment. Rules that appear to lead to improved performance of the entire system are credited, which increases their strength within the performance system. Rules that were triggered and subsequently failed to deliver beneficial effects, or even appeared to hurt the total system, will see their strength reduced. The strength of each rule determines the chance of being triggered the next time, for similar input messages.

Credit assignment assures that experience is built up in the system, by strengthening some rules and weakening others. The rules together form an internal model of what the world outside looks like and how the system needs to respond to it. And when the environment changes, strong rules will start failing and weak ones could succeed more often than before. This enables the performance system to adapt to new situations, and to continuously correct and tune its internal model.

Rule Discovery
The last part of the pattern deals with rule discovery: where do the agents in a complex system (or software developers in a project) get their rules from? Holland describes how new rules can be constructed from existing rules by recombination of building blocks. This is essentially how DNA works: by recombination of existing genes and their alleles.

Holland was one of the first to create evolutionary models for rule-based decision-making, which earned him a reputation as the father of genetic algorithms. He not only convincingly described how these performance systems are an interesting conceptual model for complex adaptive systems, but he also showed that they can be implemented easily to create evolutionary algorithms with powerful problem-solving capabilities.

Why am I writing all this?

For one reason only: to help people understand that rule-making is something that must happen bottom-up. And not top-down.

Science proves it.

(image by Syntopia)

This article will be part of the book Management 3.0: Leading Agile Developers, Developing Agile Leaders. You can follow its progress here.

 Twitter -  Subscribe - Newsletter - LinkedIn - SlideShare

Latest, greatest and favoritest posts: Management 3.0: The Era of Complexity Self-Organization vs. Emergence Self-Organization vs. Anarchy

Management 3.0: The Era of Complexity

I know. Management 3.0 is a silly name. We already have Web 2.0, Government 2.0, Project Management 2.0, Enterprise 2.0, and RSS 2.0. And interestingly enough, Gary Hamel’s Management 2.0 was one of the last to jump on the "2.0" bandwagon. So why do we need a "3.0" version number for management?

Actually, we don't.

However, I believe that the "3.0" number conveys the right message about the direction that management is taking in the 21st century. The Management 3.0 name was first used by Terrence Seamon one year ago, though it hasn't been picked up yet by others. Now I think it is time for a breakthrough. Allow me to explain:

Management 1.0 = Hierarchies

Some people call it Taylorism or scientific management, while other people call it command and control. But the basic idea has always been the same: an organization is designed and managed in a top-down fashion, and organizational power is in the hands of the few. Those at the top of the hierarchy have the highest salaries, the biggest egos, the biggest bonuses, and the most expensive chairs. Those at the very bottom have virtually nothing, little money, few responsibilities, and very little motivation to do a good job.

The shareholder value movement, promoted by top economists and business men like Milton Friedman and Jack Welch, was a perfect fit for this kind of management, because it gave shareholders a single wringable neck: the CEO. In exchange for his dangerous position the CEO was allowed to play with risky bonus schemes that had far more effect on personal wealth than organizational performance. As a side effect, and minor inconvenience, these dangerous bonus schemes also triggered a worldwide financial implosion. Oops!

I think we can safely conclude that Management 1.0, even though it is still the most widespread version, used in organizations all over the world, has a number of serious flaws. It is old and outdated and in need of an upgrade.

Management 2.0 = Models

Plenty of smart people realized that Management 1.0 didn't work well enough on its own to produce any good results. And so they created numerous add-on models, most of which dealt with systems thinking and process improvement. Some of these models gained a semi-scientific status, like the Balanced Scorecard, BPR, Six Sigma, Theory of Constraints, and Total Quality Management. Being add-ons to Management 1.0 the models still assumed that organizations were managed from the top. And so they were created to help those at the top of the hierarchy to better design their organization. Sometimes the models worked, sometimes they didn't. Some had a flimsy affair with science, while others had more affinity with religion.

At the same time there were different kinds of models that focused not on systems and science but on craft and art. Ideas like the 7 Habits, the 21 Laws of Leadership, and Good to Great listed basic values, principles, and guidelines for top managers, and then told them to practice and build experience. Again, these models were sometimes right, and sometimes not. Some models were created from research, others from personal experience, but they all had one thing in common: they aimed primarily at the ones with all the power, at the top of the hierarchy.

But working with the add-ons of Management 2.0 means that organizations still suffer from the inherent problems of Management 1.0. The basic architecture for problem solving has not changed. The organization is still seen as a hierarchy!

Management 3.0 = Complexity

In the 80's and 90's we saw the birth and rise of complex adaptive systems theory, or complexity science, first applied to mathematics and biology, and later to economics and sociology. It was a major breakthrough in systems thinking, bridging all sciences, and making some deep thinkers very happy. Stephen Hawking thought it was so important, that he called the 21st century the Century of Complexity. The first decade of this century saw a number of publications, like The Tipping Point, The Black Swan, and Freakonomics, all of them borrowing and popularizing concepts from complexity theory. And management researchers, including Ralph Stacey and David Snowden, found out that social complexity had some interesting things to say about organizational management.

One important insight is that all complex systems are networks and self-organizing systems. People may like to draw their organizations as hierarchies, but that doesn't change the fact that organizations are actually networks. Social networks to be precise. And every time you want to solve a problem in an organization, you must treat it as a network. Because that's what it is. A self-organizing network, not a hierarchy.

Second, complexity theory proves that, ultimately, all models are wrong. No matter how many diagrams, lists, or figures you throw at an organization, in the end they will always be wrong. Potentially useful, yes. But ultimately wrong. And thus, the models of Management 2.0 will always fail. Mind you, there's no need to throw them all away! But neither is it wise to treat any of them as scientific truths, or religious dogmas.

Third, based on both scientific and empirical study, social complexity shows us that management is primarily about people and their relationships. Everything else in an organization pales when compared to the importance of people and their relationships!

Furthermore, we can now conclude that management is science and craft, models and experience. And we can learn that leadership is not "higher than" management. Nonsense! The job of a manager is to be leading and governing the self-organizing systems. To ensure that people are empowered and aligned.

Conclusion

Management 3.0 is just a silly name. Some of you will disagree with me and argue about version numbers. That’s ok, because version numbers are not important.

And many of us already knew that hierarchies are bad and that networks are real. That management is about empowering people, and also giving them a direction. That models only work depending on the context. And that "leadership" is just a trendy name for managers doing the right thing and doing things right.

But network thinking adds a new dimension to our existing vocabulary. The concept of social complex systems makes us realize that we are all participants in self-organizing systems. All of us have to lead and rule in some ways. We are all "3.0 managers".

I think it's nice to have a new name. Names and versions can be powerful. The "3.0" version indicates that management is taking charge again. It usually takes Microsoft three versions of a product to get things right and make their product actually usable. Thanks to social complexity management has, in its third incarnation, finally found a solid scientific foundation for the future. None of the earlier models have to be unlearned, because many of them are still usable. We just have to replace assumptions of hierarchies with networks, and complex systems thinking. Because the 21st century is the Age of Complexity. And Management 3.0 will be the Era of Complexity.

This article will be part of the book Management 3.0: Leading Agile Developers, Developing Agile Leaders. You can follow its progress here.

 Twitter -  Subscribe - Newsletter - LinkedIn - SlideShare

Latest, greatest and favoritest posts: Yes, Good Managers Are Manipulators Managers Are Not Game Designers! Management: A Deceptive Metaphor

Managers Are Not Game Designers!

Conway’s Game of Life has always been a classic example of a complex system. In designing his game John Conway found that some sets of rules were too ordered, while other sets were too chaotic. It took him a while to find a set of rules that was nicely balanced, resulting in systems with complex behavior. Not too ordered, and not too chaotic.

In an interesting Wired article I read that Klaus Teuber took a similar approach when he was designing The Settlers of Catan, one of the most popular board games of all time. Teuber continuously played the game with his family, reconfiguring it again and again, changing the rules, the cards, and the pieces. It took him four years to find a set of rules that was nicely balanced, and that allowed for complex gameplay and heated family competition.

What sets games apart from living systems is their lack of the “adaptive” part. Games do not change their own rules while they are in progress. But living systems do. Complex adaptive systems are systems that are able to find their own way toward that sweet spot of complexity, right between order and chaos, where life blooms and creativity thrives. Scientists call it the edge of chaos. But they also could have called it the edge of order, because it is at the edge between chaos and order where we find complexity. (Never expect a scientist to come up with a name that actually makes sense.)

The question is then who or what is tuning the rules in an organization so that the organization moves towards the edge of chaos, being neither too ordered nor too chaotic? A common misconception (and yes, I plead guilty here) is that managers are somehow responsible for this.

This amounts to a negation of self-organization and emergence. If managers are choosing what “emerges”, then it is not emerging. If they have a blue-print guiding self-organization then it is not self-organization – that is, it is not agents acting purely on the basis of their own local organizing principles, but rather on the basis of simple rules chosen for them. [Ralph Stacey – Complexity and Management]

It is tempting to think of managers as game designers, like John Conway and Klaus Treuber. When the manager chooses the wrong set of rules for the organization, then the system is either too bureaucratic too chaotic. (And if they’re really screwing up it will be dead.) Metaphorically this view is interesting, but scientifically it is hogwash.

This approach completely loses the sophisticated concept of self-organized criticality, in which a system evolves to the edge of chaos through its own internal dynamic, where self-organization produces potentially novel strategies, again through the system’s own internal dynamic. […] The edge of chaos is a dynamic that occurs when certain parameters fall within a critical range – for example critical rates of information flow, degrees of connectivity, and diversity between agents [Ralph Stacey].

Every organization is a complex adaptive system. It’s like a game in which the rules are changed on the fly, and where the job of designing the game is delegated to the participants themselves. Your job as a manager is not to create the right amount of rules in the organization. Your job is to make sure that the people are able to create their own rules together. And it’s their collaborative effort that makes the system find its own way to the edge of chaos. (Or the edge of order, if you prefer.)

As complexity scientist Ralph Stacey pointed out, self-organization takes care of the edge of chaos when certain parameters fall within a critical range. The manager is not a game designer. He does not need to concern himself with the low-level rules of the game. He configures the high-level parameters, like diversity of team members, information flow between people, and connectivity between teams.

In aligning groups of people, the first responsibility of a manager is the construction of a self-organizing system. Don’t try to be John Conway or Klaus Treuber. You may define the squares on the board, but not the rules of the game. When you take rule-making into your own hands, you will significantly influence and frustrate self-organization, and then creativity, innovation and adaptability in the system will suffer.

Don’t be tempted.

This article will be part of the book Management 3.0: Leading Agile Developers, Developing Agile Leaders. You can follow its progress here.

 Twitter -  Subscribe - Newsletter - Bookmarks

Latest, greatest and favoritest posts: If You Want Something Done, Practice Your Patience Leader vs. Ruler: Which One Are You? Great Managers Have No Secrets

Management: A Deceptive Metaphor

The observation from scientists is that rules in a complex system must be tuned for the system to be both stabilizing, lively and creative. A different set of rules leads to a different system, with different behavior.

In an influential paper, Stephen Wolfram, founder of the first journal on complex systems and known for his work on Wolfram Alpha (a “computational knowledge engine”), once proposed a classification scheme which divided cellular automata (mathematical systems consisting of “cells” that can be “dead” or “alive”) into four categories:

• Class I: These are the systems with “doomsday rules”. No matter what pattern of living and dead cells you start out with, everything just dies within a few generations.
• Class II: These systems are a bit more lively, but not much. Each initial pattern quickly collapses to a set of very boring static configurations.
• Class III: These systems are at the opposite extreme: they are too lively. Each initial pattern in the system results in total chaos, with no configuration stabilizing and nothing being predictable.
• Class IV: These are the systems with a set of rules not leading to dead, static or chaotic configurations. Emerging patterns in this category are lively, creative, often surprising, but also stabilizing.

It should not surprise you that, in dynamical systems, class I and II correspond to order, class III corresponds to chaos, and class IV (of which the Game of Life is a famous example) corresponds to complexity. Given that complexity is usually explained as the region between order and chaos, this means that class IV finds itself between II and III. (This strange way of using numbers makes Wolfram’s computational knowledge engine all the more surprising.)

The same universality classification can be used (or should I say abused?) as a metaphor when distinguishing complex systems themselves. Take the human brain, for example. A class I brain would be dead: nothing happens in the brain. A class II brain could be comatose or catatonic: a state of silence or predictable repetitiveness. A class III brain could be insane or epileptic: displaying unpredictable and uncontrollable behavior. And finally, a class IV brain would be the only one considered lively and healthy. To prevent scientists from classifying my brain as a class III specimen, I must stress that I use this categorization only in metaphorical way.

When classifying organizations some people make a similar metaphorical distinction between ordered, chaotic, and complex organizations. (I assume you will forgive me for ignoring the dead organizations for now.)

• In ordered organizations no creativity and innovation are taking place. Nobody is empowered to make his own decisions. Bureaucracy dictates how all work is to be carried out, and fluctuations in organizational behavior are regular and predictable.
• In chaotic organizations there may be plenty of creativity, but not in a structured and predictable way. There is no emerging order in the organization, meaning that people simply empower themselves in getting things done. And everyone does as he or she pleases.
• Complex organizations find themselves right in between. In a complex organization, people cannot empower themselves. They are empowered by managers, who are faced with the challenge of finding a balance between empowerment and alignment, between self-organization and control.

It is said that a system at the balance between chaos and order, finds itself at the edge of chaos.

 

IMPORTANT: This classification of organizations is not scientific, but only a deceptively useful metaphor. I call it deceptive, because the metaphor has led some managers (including me) to conclude that they are responsible for finding the right balance between order and control. But as we shall see in some of my next blog posts, this conclusion is as common as it is wrong…

 Twitter -  Subscribe - Newsletter - Bookmarks

Latest, greatest and favoritest posts: Self-Organization vs. Emergence Leader vs. Ruler: Which One Are You? The Complex Manifesto for Software Development

So Now You’re an Agilist, What’s Next? (Video)

This is the video of the talk I did at the Agile Eastern Europe conference in Kiev, a few months ago. The talk is called "So Now You’re an Agilist, What’s Next?", in which I talk about complex systems and "laws" for software development. (I performed the same talk at several other conferences.)

 

Note: I have already a couple of conferences planned for 2010: the Scandinavian Developer Conference (March), Norwegian Developers Conference (June), and Agile Eastern Europe conference (October) are among them.

Hope to see you there!

 Twitter -  Subscribe - Newsletter - Bookmarks

Latest, greatest and favoritest posts: Self-Organization vs. Emergence Self-Organization vs. Evolution Self-Organization vs. Anarchy

Emergence in Development Teams

When trying to translate the concept of emergence to teams, we can recognize a host of interesting phenomena. The first is the possibility of collective decision making without central planning. Swarm raids of army ants are reported to be among the largest organized operations carried out by any animal. But not a single ant has a picture of the entire operation in its mind. Likewise, no team member has a complete picture of the whole project. And yet it is common for good plans to emerge from the interaction of team members, where each can only work with incomplete information.

From research into human consciousness we can learn that multiple conflicting views can result in a (seemingly) singular view of the entire system. Daniel Dennett and Marvin Minsky both explained that “one stream of consciousness” is an illusion. According to Dennett there are actually “multiple drafts” of consciousness, which Minksy called the “society of mind”. Our brain resolves these multiple competing interpretations of the world into something that we call one identify, or a “self”. For an illusion it works remarkably well. Similarly, the multiple views of the world in a team can resolve into a singular team view. The team identity is an illusion, and yet it works by having a real impact on their projects. Paradoxically, human consciousness works because of the underlying multiple drafts. And the team identity works because of the underlying disparate views. I am sure some people will be glad to know that their diverging opinions may turn out to be crucial for a team identity to emerge. (Just don't blame me next time you get into a fight.)

It is also known that a system can be more than the sum of the parts. Our brains have a stable alpha wave of between 8-12 Hz. It is an accurate clock, though it is constructed from manny sloppy ones, because all the individual neurons have their own rates of discharging, varying between 8 to 12 times per second. And yet the emergent alpha rhythm is more reliable than that of any of the neurons. Similarly, it is not uncommon for a whole team to perform better than the best performance of any of the individual team members. DeMarco and Lister call this a “jelled team”. It is “a group of people so strongly knit that the whole is more than the sum of the parts. The production of such a team is greater than that of the same people in unjelled form.”

Finally, the nature of emergent properties is often unpredictable. Water, consisting of two hydrogen atoms and one oxygen atom per molecule, is subject to state changes like freezing and cooking. There is nothing in the properties of hydrogen and oxygen atoms predicting these properties of water. It is the same with teams. You cannot predict the behavior of a team by analyzing individual team members separately. The emergent behavior of the team is a result of the interactions between the team members.

The only thing you can predict is that they will always try to undermine your profitability by asking for more time, better education, expensive tools, food, or conferences in luxurious resorts.

(image by feserc)

 Twitter -  Subscribe - Newsletter - Bookmarks

Latest, greatest and favoritest posts: Self-Organization vs. Emergence Self-Organization vs. Evolution Self-Organization vs. Anarchy

Self-Organization vs. Emergence

This is the 3rd post in a trilogy about self-organization. The first one was Self-Organization vs. Anarchy, and the second was Self-Organization vs. Evolution. This one is about Self-Organization vs. Emergence.

When a property of a system cannot be traced back to any of the individual parts in the system, it is called an emergent property. Your personality is an emergent property of your brain. It cannot be traced back to individual neurons. Likewise, fluidity is an emergent property of water, and culture is an emergent property of a group of people.

Three aspects are important for a property to be emergent: First of all, supervenience is the observation that the property will no longer exist if you take away the individual parts of the system. For example, your personality disappears when I remove your neurons. (Relax, I won't try and prove it.) Furthermore, the property should not be an aggregate, meaning that it is not simply the result of adding up the properties of the individual parts. For example, a single water molecule has no fluidity. So you cannot simply add up the fluidities of a billion individual molecules to determine the fluidity of water. Finally, there must be downward causality, meaning that the emergent property should influence the behavior of the individual parts. For example, the culture of a group of people influences the behavior of its members. In short, emergent properties are global (to the system), irreducible, and noticeable (to the parts).

 

The boundaries of the sciences rely on emergent levels. Physics resolves into chemistry, which resolves into biology, which resolves into psychology, which resolves into economics. And each science works with the emergent properties generated by previous levels. It also means that each level gives rise to new laws and new insights. Psychology is more than just applied biology, biology is more than just applied chemistry, and chemistry is more than just applied physics. This is why greedy reductionism doesn't work. You cannot explain forgetting your spouse's birthday in terms of faulty atoms or string theory.

Throughout literature there has been some confusion (and disagreements) about self-organization and emergence. Some scientists have defined one in terms of the other, while others claim they are separate concepts. I tend to agree with Peter Corning in that there may be self-organizing systems without emergent properties, and emergent properties in systems that are human-created instead of self-organizing. But it is just a matter of definition. On this blog I choose to use the term emergence for “organized ‘wholes’ composed of functionally distinct ‘parts’ that produce irreducible combined effects”. Even though this blog itself is not a self-organizing system, the impression it will make on you will definitely be an emergent property: the impression is global to blog, irreducible to the individual words, but quite noticeable if you choose to iniate a DDoS attack at my blog after reading.

(image by MikeBlogs)

 Twitter -  Subscribe - Newsletter - Bookmarks

Latest, greatest and favoritest posts: Self-Organization vs. Evolution Self-Organization vs. Anarchy Your Project Has No Goal

The Danger of Lean: Ignoring Social Complexity

After reading three books about Lean software development I feel slightly uncomfortable with the heavy emphasis the Lean community seems to place on process and economy.

(Transparency: I read Implementing Lean Software Development, Managing the Design Factory, and Lean Software Strategies. They were advised to me by several lean experts.)

The only things these Lean books seem to focus on is reducing queue size, reducing cycle time, reducing variability, reducing work in progress, and reducing batch size. It is all inventory this, and control systems that. No wonder the Lean brand appeals more to higher management than Agile. Somehow the word “people” got lost somewhere. (Note: I’ve been told the people-issue is an important topic in the works by Deming and Jeffrey Liker. But I specifically wanted to investigate software development books.)

Try it yourself! Just pick your average Lean software development book, go to the index, and try to find references to concepts like motivation, creativity, innovation, personality, or fun. You probably won’t succeed. And if the content is there at all, it is drowned out by all the stuff about inventory, processes, control, economics, value streams, efficiency, queues, variability, and waste. (I’ll be honest: the Poppendieck book has a chapter called “People”. But this little gem of information pales in comparison to the volume of economical process jargon in the other books, and the many blog posts about Lean software development.)

And I don’t have a problem with this. All that stuff is indeed very valuable!

However, there is a problem when people see Lean as The Only Way.
Because it isn’t.

Alan Shalloway published a blog post The Dangers of Complex Adaptive Systems, where he tried to analyze where complex adaptive systems (CAS) work and where they don’t. I respect Alan a lot, but his blog post completely misses the point about complexity theory. Most systems in the universe are complex systems, and many of them are adaptive. Debating whether or not CAS works is like debating whether or not biology works. It makes no sense. The system just exists. Whether it is a school of fish, a flock of geese, or a team of software developers, they are all complex adaptive systems. You can debate whether certain survival strategies of these systems work. If the strategies don’t work, the systems will need to adapt, or they go extinct. If they don’t go extinct, then obviously the strategies work! That is all part of the theory of complex adaptive systems.

Complexity theory is just a scientific description of the world. It has no “good” or “bad” qualifiers.

But Lean is a prescriptive model. It tells you what you should do when you try to manage the complex system that we call a software development team. It tells you there are good and bad choices.

And that’s where things can go wrong…

The Incompressibility problem of complex systems tells us that every model (including Lean) is inaccurate:

There is no accurate representation of the system which is simpler than the system itself. In building representations of open systems, we are forced to leave things out, and since the effects of these omissions are nonlinear, we cannot predict their magnitude.
– P. Cilliers (2005), Knowing Complex Systems

When you create a model of a system (like a software project), you have to leave things out. Unfortunately, because of the famous Butterfly Effect, in complex systems every little thing that you leave out can still have huge unexpected consequences. And because complex systems are non-linear, you cannot predict which parts of the real system you can safely ignore.

This leads to a simple, but very important conclusion:

All models are wrong, but some are useful.
– Box and Draper (1969), Evolutionary Operation

Lean software development describes a software project as a pull system with a value stream. That means it is a model of the real system, which is more complex. It is a very, very useful model. Unfortunately, it is also wrong because it is inaccurate. But so are all the other models. And some models are worse than others. (Actually, I think Lean probably scores quite well among all those bad models.)

The Lean model ignores the social part of complex systems. I just reviewed all the diagrams in the three books mentioned earlier. I couldn’t find a single diagram or picture with people in it! What are the consequences of ignoring the people-issue in the model? What happens to people’s motivation when you reduce variability? What happens to people’s creativity when you treat the system as a production flow instead of an emerging design effort? How is the process aligned with people’s personalities? And where is the fun? I can tell you, those three books were about as much fun as my last visit to the dentist. Useful, yes. Healthy, sure. Painful, oh God, yes! They certainly didn’t make me look forward to being part of a lean software development team. (I fondly remember the first books about Agile software development. They were about people. And they were fun. There’s certainly more humor on the Agile side…)

Again and again I see emphasized that Lean software development is based on systems thinking. That is correct. And that is exactly the problem. Just have a look at this interesting diagram created by David Snowden, a complexity expert (the red text in the diagram is mine):

Systems thinking is a great tool, but system thinking ignores the unordered parts of the world. The messy bits. The unpredictable stuff that always comes as a surprise, no matter how much time you’ve spent drawing process diagrams and analyzing value streams. In software projects, the proper foundation for management is not systems thinking but social complexity. Social complexity is complexity theory, but applied to people instead of fish and geese.

The danger of Lean is that it overemphasizes process and economy. This makes it deceptively easy for managers to embrace Lean (instead of Agile) and to forget all about people-issues. Again. And the people-over-process principle will be ignored. Again.

Social complexity is the only proper foundation for management of software teams. Systems thinking is outdated. Ralph Stacey, another complexity expert, threw systems thinking in the trash bin. And rightfully so. Goodbye, and good riddance. Systems thinking has served its purpose, but science has moved on.

So yes, Lean has some fantastic ideas, as there are great ideas in Agile too. But neither Lean nor Agile is a complete model of a complex system. The only accurate model of a complex system is the system itself. Which means you must focus on the entire system, including the people. And not only on value streams and waste reduction. It takes a complex mind to recognize the dangers of incomplete models.

I’m writing a book about this stuff. Did I tell you that? I think some managers are going to need it. The book will be useful, and healthy. And fun.

Note: I could just as well have named this post “The Danger of Agile”, focusing on the things that Agile ignores. But there are already so many posts bashing Agile, Scrum, and XP. It wouldn’t have been interesting to write. I’m human too. I want my work to be fun. I think increasing variability is more fun than reducing it.

(picture by Foxtongue)

 Twitter -  Subscribe - Newsletter - Bookmarks

Latest, greatest and favoritest posts: ScrumButs Are the Best Part of Scrum The Complex Manifesto for Software Development Simple vs. Complicated vs. Complex vs. Chaotic