Project Optimism Bias - Friend or Foe?

The belief that the future will be much better than the past and present is known as optimism bias. It's the tendency for project portfolio management professionals to ignore lessons learned and believe that they are less likely to repeat the same mistakes or experience negative events than others and to act on that optimistic belief that “It won't happen to me!” This thinking often leads to a phenomenon called the planning fallacy in which predictions about how much time will be needed to complete a future task display an optimism bias and underestimate the actual time needed.

This innate confidence or attitude that something is the case is articulated by Tali Sharot of TedTalk fame, where she states that we are more optimistic than realistic. To enable progress, we need to be able to imagine alternative realities – better ones – and we need to believe that we can achieve them. Such faith helps to motivate people directing, managing and delivering a project or programme to pursue organisational (including strategic) objectives and key results.

This optimism bias is defined as the difference between a person's expectation and the outcome that follows. If expectations are better than reality, the bias is optimistic; if reality is better than expected, the bias is pessimistic. This denotes the glass half full or half empty conundrum. The extent of the optimism bias is thus measured empirically by recording an organisation’s expectations before a project event unfolds and contrasting those with the required outcomes and forecast benefits that transpire since reality sometimes does not meet customer expectations. 

In a project environment, optimism bias is the demonstrated systematic tendency, whether unconsciously or deliberately, for project business cases to overstate forecast benefits, and understate the timescales and costs, both capital and operational including delivery complexity. To combat this, the United Kingdom Treasury developed the Green Book on how to appraise policies, programmes and projects.  They provide further guidance through Better Business Cases™ for Better Outcomes using the Five Case Model that ought to be used by those accountable within an organisation using public resources (i.e. people, assets, materials, funding and services) in developing programme and project investment proposals. Since this guidance describes how to successfully scope, analyse, plan, procure and manage delivery to achieve value for money and social return on investment in terms of public outcomes, services and trust.

The companion guide, APMG-International’s Managing Benefits – Optimizing the Return from Investments also notes that psychologists and other researchers have identified a number of cognitive biases and organisational factors that impact our ability to assiduously produce accurate and reliable forecasts. This leads to shortfalls in benefits realisation where in many cases the stated benefits in the business case were unfortunately overstated to begin with. Depending on the organisational culture, it may be a reflection of the perceived need to present best case scenarios to have any success in achieving the required whole of life funding estimate for capital, service and maintenance.

Professor Bent Flyvbjerg at Oxford University calls this strategic misrepresentation; defined as the planned, systematic, deliberate misstatement of costs and benefits to get projects approved. It is for this very reason the UK government espouses that project plans should allow for optimism bias by increasing the costs and timescales estimates when information is unknown early in the lifecycle and prior to an investment decision. As such, the residual use of optimism bias to measure continued uncertainties should be minimal and generally no more than 2% for a standard capital scheme.

Image: 5 Case Model

It is for the above reasons that good decisions are based on having sufficient, objective, accurate and timely data and information on programme and project costs, timescales, benefits and risks. As weaknesses in the scrutiny and appropriateness of data and modelling techniques mendaciously distort the information on which investments are approved and mask the business, service and external risks. This is why a business case must never be perceived or used as a medium for simply gaining approval. Better Business Cases™ using the Five Case Model provides decision makers and stakeholders with a proven framework for structured thinking and assurance that the investment proposal:

• Provides strategic fit and is supported by a compelling case for change (Strategic Case)
• Delivers best public value to society, including wider social and environmental effects (Economic Case)
• Is commercially viable and attractive to the supply side (Commercial Case)
• Is affordable and is fundable over time (Financial Case)
• Demonstrates robust arrangements are in place for the delivery, monitoring and evaluation of the scheme, including feedback into the organisation’s strategic planning cycle (Management Case)

Over-optimism persists where weaknesses in information are ignored and little effort is made across the organisation to benchmark and develop robust estimates or to be honest and transparent about the assumptions made on limited data and information. It's, therefore, critical to organisational performance to start acknowledging that optimism bias exists! One leading entity, Transport for NSW has recognised in their Cost-Benefit Analysis Guide “ways to avoid cognitive bias when developing initiatives is to think in terms of outcomes, exploring a wide range of [viable] options early in development, and avoiding the preferring of an option without the necessary evidence.” A starting point for benchmarking and estimation is to establish a standard unit of measure. The most commonly used measure is a unit of local currency in present day prices (known as “real prices”). Increases in prices due to inflation or other sources of cost escalation should not be included in the values of future benefits and costs.

Additionally, programme and project justification is often predicated on assumptions (typically without evidence) about desirability, feasibility and/or achievability. As such, business cases should distinguish information without any substantiating evidence (assumptions) from that based on probability and empirical data (presumption). As such, explicit adjustments or allowances should be made to the estimates of a project's costs, benefits and duration. These estimates ought to be based on empirical data from past or similar projects, and adjusted for the unique characteristics and complexities of the initiative. This is why forecast benefits stated in business cases (used to justify programme and project viability, achievability and desirability) are better understood if the benefits realisation plan and the business case are approved by the host organisation at the same time.

As optimism is biased to the extent that people expect good outcomes without working to ensure those outcomes occur. Organisations and appraisers alike should acknowledge that optimism bias in business cases exist and seek ways to adjust for it since actual and estimates show the accuracy between prediction and the reality of the programme and project costs, timescales, benefits and risks against complexities in delivery.

Image: Optimism Bias diagram