How to Architect for Agility and Emergence in Complex Adaptive Systems

Emergent systems do not necessarily require human intervention.  For instance, many systems emerge on the basis of swarm intelligence. There are situations in which coordinated behaviors occur without prior planning through communication. They are both common and effective in multi-agent systems – including both biological and computational.  Self-organizing complex systems typically are comprised of a large number of frequently similar components or events. Through their process, a pattern at the global-level of a system emerges solely from numerous interactions among the lower-level components of the system.  The rules specifying interactions among the system’s components are executed using only local information, without reference to the global pattern, which, as in many real-world problems is not easily accessible or possible to be found. In biological systems such emergence is represented by Stigmergy is coordination through changes to the environment. Stigmergy as a kind of indirect communication and learning by the environment found in social insects is a well know example of self-organization, providing not only vital clues in order to understand how the components can interact to produce a complex pattern. Biological strategies are used to create a system of human collaboration. The information and its organization emerge later – as a consequence of the system being put into operation.

For traditional organizational theorists as well as business executives (and Enterprise Architects), biological systems will seem to break all the rules of both logic and commonsense. Traditional organizational theory, business strategy, management techniques and enterprise architecture have been based upon the premise that collaborative systems have to be ordered and brought under control.  This seemed the only rational way to be able to specify results, to set and achieve goals and targets. The idea that a collaborative system should be deliberately designed to be unstable and chaotic would seem to be utter madness. Yet, this is the way biological systems work.  However, biological systems can be highly organized and efficient, and possess abilities to process some kinds of information even better than the most powerful computer systems. 

Philip Agre offers a substitute for planning, participatory improvisation “Improvisation, like Planning, involves ideas about what might happen in the future. Improvisation differs from Planning in that each moment’s action results, effectively, from a fresh reasoning-through of that moment’s situation.”    Sense is retrospectively made of the interaction through a process Weick calls “sensemaking.” This is consistent with the rethinking of EA in emergent enterprise architecture. 

This proffers a revised approach to doing EA based on what the Gartner group initially described as emergent EA.  

They contrast this with the traditional model as follows:

This respecification has led to a critique about prevailing system of systems thinking, using document-based approaches and UML-based methodologies to model contents as static, in-coherent, system-perspective views. The traditional technical architectures developed using artifacts from the TOGAF Architecture Content Metamodel and the various types of models represented in FEAF, DoDAF, MoDAF and similar approaches and methodologies, while useful as specifications for integrating current systems and the project portfolios, are now developing the dynamic situation awareness architectures required in mission analyses and planning, as well as the complex nature of the post modern enterprise. 

We need to be aware there are often “good organizational reasons for bad organizational behavior.” What we see at the macro level as a problem often has justifiable reasons when looked at closely.  The bad behavior is emergent. As one moves from the global to the local or shop floor level we often “lose the phenomenon.”  The enterprise arises from local interaction of often-independent units that exist within a common environment. Each unit or entity interacts with its immediate environment according to a set of low order rules. The combined effects of these lower order interactions within an environment gives rise to higher order organizational phenomenon or organizational culture. Organizational culture emerges from localized interactions, and is grounded at the local level. Hence, culture is highly resistant to change. Changing culture and transforming the enterprise entails re-specifying local level rules rather than simply imposing change from the top. This is the particular value of Active Knowledge Modeling where structures are recognized as dynamic, emergent and adaptive; yet capable of description. Utilizing an emergent enterprise architecture and characterized by such dynamic models proffers a mechanism to initiate positive change.

I referred to the postmodern view of organizations and to postmodern organizations, and that this has implications for enterprise architecture.  In the traditional or what we can refer to here as modernist approaches of enterprises society is characterized as rational, stable and well ordered.  However, in contrast the postmodern approach we contrast value as counter rational, reflexive, and other-oriented global and require networked models for organizational study. In the postmodern organization we see flexible structures where workers need continual learning – which can be understood as emergent knowledge spaces.  In the postmodern enterprise, self-managing teams replace top-down management and quality is part of the knowledge process.  Rather than management being a role at a point in time it is now team or project based. The very concept of a stable blueprint for the enterprise needs to be replaced by one that encompasses agility and recognizes emergence as the basis for changing and to-be models.  

This has implications in EA and particularly for the Architecture Development Methodology (ADM) in TOGAF.  In the postmodern non-traditional architecture view the “to be” is a moving target, and the continual iterations of the ADM Phases fluctuate as the enterprise adapts to transitional architectural steps. Consequently, the areas of focus transform into ellipse change cycle patterns.  Interaction and interference from different areas of focus is to be expected, and requirements management takes on a “hive” or swarm intelligence structure. This raises a new set of concerns:

  • Is there a need to change the language of architecting?
  • Is there a need for a more dynamic modeling approach showing adaptation over time?
  • Perhaps architecture artifacts of the future will become more like films, only with the animation drawn out over a much longer period of time?
  • Should we model clusters, swarms, and centroids of architectures, with some level of coupling (tightly or loosely) and generalized functionality across the sub-architectures (for example, generalized audit trails for SOA-based systems)?
  • Will future focus be increasingly on user experience driven design, while constrained by QoS necessities and cost of resources?
  • What is the role of knowledge modeling as artifacts? 
  • How can we create an acquisition process that allows for emergence?

Given these issues we need to consider architecture as a crucible of that entails the trade-offs between resource availability and constraints, along with meeting organizational/enterprise goals and user experience. Agile is a set of actions that focuses and models the results to meet acceptance criteria, while enabling ongoing adaptation and accounting for emergence.  Architecture is the representative artifact of the solution. We must consider how agile augments and changes architecture methods to produce solutions that embrace greater user experience, while enabling adaptation. We must recognize how architecture brings focus and system discipline to agile.

Authored by Dr. Beryl Bellman, Principal & Senior Instructor

1. Philip Agre Computation and Human Experience, Cambridge University Press (July 28, 1997)

2. See Hatch, 2006, Organizational Theory: modern, symbolic and postmodern perspectives