As we know now, the world is full of complex systems. However much we wish, we can’t avoid them. We see them in our organizations. They are there in our societies. They can be experienced in ‘cloud bursts’ and in flash floods. We even find them in our families.
They are nagging and they are wicked at times. Wicked in the sense that it often leaves us baffled preventing us from acting skillfully.
The question is how do we deal with them?
However, the idea of prediction is a bit different from our usual idea of prediction. Our usual idea about prediction is, if we know sufficiently about the behavior at any point of time, we would be able to predict the behavior of a system any time in the future.
But that is not what bothers us about understanding and dealing with complex systems. And it is fair to say that such predictions are absolutely impossible with complex systems. Therefore, with any complex system all we are interested about is to detect the appearance of a ‘black swan‘. A black swan is some sudden unexpected change in the behavior of a system that disrupts the system and brings it crashing to the floor. The crash of 2008 provides an extreme example of a black swan. But such examples are rather common in our lives. Our careers crash. Organizations crash. Our health suddenly crashes. A ‘cloud bursts’. Or for that matter any system is likely to be disrupted any time by the sudden appearance of black swans.
So what might we do about it? Does it help if we notice the change in behavior of each agent or individuals that form the part of the system? For example, does it help if we watch individual behavior of employees in an organization? Or for instance, does it help if we monitor individual performance of school or college students? We know that such methods hardly help improve the system though we are enslaved by such methods by ‘blind faith’. A complex system would keep doing what it does. That is its role or purpose.
The good news is that the behavior of any complex system can be monitored and predicted for black swans, a little in advance, before it strikes us with full force to bring the system to its knees.
In order to predict black swans we need to know of one very peculiar phenomenon of any complex system. That is a complex system behaves linearly for most of its time when it is free from a black swan or an outlier. Then as a black swan slowly creeps into the system the system suddenly behaves non-linearly. When it behaves linearly it gives us a false sense of security. We feel everything is fine and hunky dory and would stay like that forever. We take pride in our design.
Lulled by our false sense of security, we then forget that non-linearity is just waiting to strike us. And when it strikes we are so much confounded that we rush like headless chickens to ‘fix’ the ‘problem’. And believing in our superior intelligence, we keep ourselves busy ‘fixing’ problem after problem till we drop dead from such heroic efforts.
Mathematically speaking, while linear behavior of any complex system follows Gaussian distribution; the nonlinear behavior follows some sort of power law. So, it is the mix of the two, never one thing or the other.
By understanding this phenomenon clearly, we can ‘predict’ a black swan or an outlier very easily. It is deceptively simple. Simpler than what we perceive it to be.
It is liberating too. Once the presence of a black swan is detected, much before it actually happens, we are left with enough time on our hands to deal with it effectively and skillfully. It also leaves us with the possibility to dramatically improve the system big time.
The truth is there is no ‘randomness’ anywhere. The concept of ‘randomness’ is a big illusion at best.
So, then, what are we waiting for to improve our lives?