Go back

How Agile methodologies mitigate cognitive biases that lead projects to failure

Agile Software

By Walter Abrigo, Managing Director at Santex

I want to emphasize in this article how the existence of two cognitive biases, which are almost always present in our daily lives, position agile methodology practices as one of the most adaptable frameworks for project monitoring and management, in general. This is especially true when the context of the given project development is complex, has changing requirements that are poorly defined, and where innovation, competitiveness, flexibility, and productivity combined are critical to achieving the desired results.  

Cognitive biases

    1. The emotional aspect of our decisions and choices.

  1. The fallacy of planning.

By reviewing each of these biases, we are able to see how people’s behavior fits better and more consistently within the structure of Agile methodologies.

Our decisions and choices are emotional

The following cases demonstrate how, in our every day decision making, we often forget the Base Rates (or the true distribution of events). Additionally, we strive to make sense of representative stereotypes, we seek causes and explanations, and we have a natural aversion to losing whenever there is something at risk.

First Case: Forgetting the Base Rates (the true distribution of events)

Tom is extremely intelligent, although he lacks true creativity. He needs order and clarity, and prefers systematic organization. He has a strong competitive drive and seems to have little interest and sympathy for others. He does not enjoy dealing with other people. Although he’s self-centered, he has deep moral awareness.

Let’s order the following nine areas of expertise according to the probability that Tom would be a student in any of these fields. We’ll use one for the most likely and nine for the least likely.

    • Business Administration

    • IT

    • Engineering

    • Humanities and Education

    • Law

    • Medicine

    • Physics and Biology

  • Social Sciences and social work

Most will agree that Tom fits well with the stereotypes of smaller groups of students, like IT and engineers, but would fit poorly into larger groups, like humanities and education, social sciences and social work. This is an example of how we substitute the probabilities of the Base Rates for representative stereotypes.  

Second Case: Prejudices based on stereotypes

Linda is thirty-one years old. She’s single, outspoken, and very bright. She majored in Philosophy and, when she was a student, she was very concerned about the issues of discrimination and social injustice. She participated in several anti-nuclear protests. Given this information, which of the following scenarios fits best with Linda’s personality?

    1. Linda is a bank teller.

  1. Linda is a bank teller and activist for the feminism movement.

Most will agree that Linda is well suited to the role of “bank teller and feminist.” The stereotypical teller may not be a feminist, so including this detail adds more emphasis to the description. Nonetheless, both feminist bank tellers and regular bank tellers share the common fact that they coexist in the world of ‘bank tellers.’

P(teller)=P(feminist teller) + P(teller not feminist).

Third Case: Seeking causes

Take the genders of six children born one after the other in a hospital. The sequence of boys and girls is random. Each event (birth) is independent of the other, and the number of boys and girls born in the hospital in the last hour has no effect on the gender of the next child. Now consider three possible sequences (M = male, F = female):

    1. MMMFFF

    1. FFFFFF

  1. MFMMFM

Are these sequences equally probable? The intuitive answer is, “of course not!” but that is false. Because each event is independent and the results M and F are both (approximately) equally likely, all possible sequences for the six births are as likely as any other. Now that we know that this conclusion is true, it seems counterintuitive because only the third sequence appears to be completely random. Our minds are built with associative machinery that continuously seeks causal relationships, and this tendency leads to serious error in our evaluation of sequences that are truly random.

We are hunters of patterns, believers in a coherent world in which regularities (like a sequence of six girls) are not accidentally produced, but rather the effect of a particular cause or someone’s intention.

Fourth Case:  We are willing to risk more when it comes to losses than gains.

Situation 1: Imagine a group of people where each one has $3,000 and you give them a choice between:

    1. Receiving another $1,000, or

  1. Flipping a coin and playing the $1,000 for double or nothing: if they win they’ll receive an additional $2,000, but if they lose they get nothing.

What would you choose?

Situation 2: Imagine a group of people where each one has $5,000 and you give them a choice between:

    1. Giving up $1,000, or

  1. Flipping a coin to play $1,000 for double or nothing:  If they lose, they give up $2,000, but if they win they don’t lose any money.

What would you choose?

Most of us in situation one prefer option one and most of us in situation two prefer option two. The interesting thing here is that the odds of the four options are identical, but differ considerably in our minds. We are more willing to take a risk when it comes to LOSSES and are more reluctant to take a risk when it comes to GAINS.

The fallacy of planning

The fallacy of planning is one manifestation of an omnipresent optimistic bias. Almost all humans see the world as less harmful than it really is, our skills better than what they really are, and our goals easier to achieve than they really are. We also tend to exaggerate our ability to predict the future, which exudes optimistic overconfidence.

When we complete a successful project, we assume that it was due to our accurate and detailed planning of controlled variables. We forget the random variables that impacted us positively. We assume the cause of success was within the plan, and we are the performers.  

When we finish a project and it is unsuccessful, we assume that this was due to the presence of external uncontrollable variables, not foreseen from the beginning which affected us negatively. The cause of failure is out of our hands, and we are not the performers.   

Agile methodologies mitigate these biases

Having raised the existence of these two cognitive biases (the emotional side of our decision-making and the fallacy in our planning), we see two aspects of Agile methodologies that make them in the most effective way to mitigate the biases: valuing people and response to change.

By realizing that our decisions are more emotional than they are rational, we place more value on individuals and their interactions than we do tools and processes. This allows us to communicate more empathetically and understand the emotion behind our choices.

Regarding the fallacy of planning, by putting more value on response to change, rather than following a plan, we can better detect the random variables that may arise and impact the results.

In this way, we can realize the importance and value that Agile methodologies have in reducing the noise and deviations that may occur during the development of a project.

Sources

KAHNEMAN, D. (2011) Thinking, Fast and Slow. Debate Editorial.

About the Author Walter Abrigo is a Managing Director at Santex. In addition to his large academic career, he possess market expertise in several organizational processes such as management control, change and strategy, recruiting and staffing as well as performance and engagement.

You can read the spanish version of this article published in “Pulso Social”.