Predicting Black Swans – Part II

In the earlier post we dealt with the concept of predicting a ‘black swan‘.

In this post, I intend to explore the concept a bit more: what exactly we monitor to notice a ‘black swan’ in time?

In doing so we would be forced to consider the natural response of a system.

The starting point of our exploration would be to understand how any system, as a whole, whether natural or engineered, would disturbed by a ‘black swan’.  A system is disturbed in three possible ways, which are as follows:

a) A system loses energy till it reaches a tipping point

b) A system gains more and more energy till it crosses the point of system resilience

c) A part of a system emits more energy than it is normally supposed to, that is going beyond the linear response of the part.

So the natural way to watch a system to expect a ‘black swan’ in time, is to keep a tab on the ‘energy’ of a system in the following ways:

a) Monitor the entropy of a system. As a system functions the entropy of a system gradually rises till it hits a threshold limit indicating the appearance of a ‘black swan’ or an outlier.

b) Monitor the energy gain of a system till it crosses the ‘resilience’ point to give birth to a ‘black swan’, outlier or a ‘wicked problem’.

c) Monitor critical parts of a system for excess emission of energy till it goes beyond the linear response of a part.

It is useful to remember that energy is transferred in ‘quanta‘ or in packets of energy. Therefore, it is natural to expect jumps of energy levels as we record by capturing the different manifestation of energy levels on monitoring trend charts. So when a ‘jump’ is big enough to cross a threshold limit or resilience point or linear response level indicated by its presence outside the Gaussian distribution range  we can be quite sure that a ‘black swan’ or an outlier or a ‘wicked problem’ would soon arrive on the scene. We call such an indicator as a signal.

Therefore, the central idea is to capture such signals in time, just before a ‘black swan’ makes it way to appear on the scene to dominate and change the system.

However, the question is how early can we detect that signal to effectively deal with the inherent ‘black swan’ in a system, which is yet to appear on the scene?

That would be explored in the next post.

Predicting BlacK SwaNs

As we know now, the world is full of complex systems. However much we wish, we can’t avoid them.  We see them in our organizations. They are there in our societies.  They can be experienced in ‘cloud bursts’ and in flash floods. We even find them in our families.

They are nagging and they are wicked at times. Wicked in the sense that it often leaves us baffled preventing us from acting skillfully.

The question is how do we deal with them?

It is easy to deal with any complex system if we are able to predict their behavior over time.

However, the idea of prediction is a bit different from our usual idea of prediction. Our usual idea about prediction is, if we know sufficiently about the behavior at any point of time, we would be able to predict the behavior of a system any time in the future.

But that is not what bothers us about understanding and dealing with complex systems. And it is fair to say that such predictions are absolutely impossible with complex systems. Therefore, with any complex system all we are interested about is to detect the appearance of a ‘black swan‘.  A black swan is some sudden unexpected change in the behavior of a system that disrupts the system and brings it crashing to the floor. The crash of 2008 provides an extreme example of a black swan. But such examples are rather common in our lives. Our careers crash. Organizations crash. Our health suddenly crashes. A ‘cloud bursts’. Or for that matter any system is likely to be disrupted any time by the sudden appearance of black swans.

So what might we do about it? Does it help if we notice the change in behavior of each agent or individuals that form the part of the system? For example, does it help if we watch individual behavior of employees in an organization? Or for instance, does it help if we monitor individual performance of school or college students? We know that such methods hardly help improve the system though we are enslaved by such methods by ‘blind faith’. A complex system would keep doing  what it does. That is its role or purpose.

The good news is that the behavior of any complex system can be monitored and predicted for black swans, a little in advance, before it strikes us with full force to bring the system to its knees.

In order to predict black swans we need to know of one very peculiar phenomenon of any complex system. That is a complex system behaves linearly for most of its time when it is free from a black swan or an outlier. Then as a black swan slowly creeps into the system the system suddenly behaves non-linearly. When it behaves linearly it gives us a false sense of security. We feel everything is fine and hunky dory and would stay like that forever. We take pride in our design.

Lulled by our false sense of security, we then forget that non-linearity is just waiting to strike us. And when it strikes we are so much confounded that we rush like headless chickens to ‘fix’ the ‘problem’. And believing in our superior intelligence, we keep ourselves busy ‘fixing’ problem after problem till we drop dead from such heroic efforts.

Mathematically speaking, while linear behavior of any complex system follows Gaussian distribution; the nonlinear behavior follows some sort of power law. So, it is the mix of the two, never one thing or the other.

By understanding this phenomenon clearly, we can ‘predict’ a black swan or an outlier very easily. It is deceptively simple. Simpler than what we perceive it to be.

It is liberating too. Once the presence of a black swan is detected, much before it actually happens, we are left with enough time on our hands to deal with it effectively and skillfully. It also leaves us with the possibility to dramatically improve the system big time.

The truth is there is no ‘randomness’ anywhere. The concept of  ‘randomness’ is a big illusion at best.

So, then, what are we waiting for to improve our lives?