Research Discussion Paper – RDP 2026-05 Real GDP Forecasts by International Organisations

The paper examines the properties of real gross domestic product (GDP) forecasts by international organisations. These organisations commit substantial resources to their macroeconomic forecasts, with a forecast round taking multiple weeks of work by dozens of economists. This high amount of effort reflects the importance of the forecasts for the decisions of international organisations and others, and hence their importance for economic outcomes more broadly.

International organisations use their forecasts to inform key policy decisions. The International Monetary Fund (IMF) and World Bank draw on the published forecasts for an economy when deciding whether to make a loan or grant to that economy. The European Commission (EC) uses its growth forecasts when setting its annual budget. The IMF, World Bank, EC and Organisation for Economic Co-operation and Development (OECD) all rely on their growth forecasts when giving macroeconomic policy advice to national governments.

International organisation forecasts are also important due to their effects on external users. Central banks and fiscal authorities make their own growth forecasts to inform their policy decisions, but are also users of international organisation forecasts. A survey of national governments shows that a substantial majority agree that IMF forecasts ‘are valuable inputs to the economic policy process in my country’ (Genberg and Martinez 2014b, Figure 5). Other external users include financial market participants, with JP Morgan and Citigroup both reporting that clients pay attention to IMF forecasts (Genberg, Martinez and Salemi 2014). Since international organisation forecasts influence the expectations of both governments and the private sector, they affect investment and other decisions. Empirical work finds they have causal effects on economic outcomes. Afonso (2010) shows that EC growth forecasts influence bond yields, while Beaudry and Willems (2022) find that if the IMF makes overly optimistic growth forecasts, this causes an economic contraction later, which they attribute to governments and the private sector accumulating excessive debt in response to the optimistic forecast.

Forecasting is challenging, and best practice is to rigorously evaluate forecasts to identify opportunities for improvement. Both academic researchers and international organisations have investigated the properties of real GDP forecasts, as will be discussed in the literature review. Comparisons of accuracy hold forecasters accountable, and assist users in choosing how much attention to pay to different forecasters. It is also useful to test if the forecasts are as accurate as possible given the information available, a concept called ‘forecast rationality’ in the literature.^[1]

This paper makes three contributions to the literature. First, we compare the forecast accuracy for pairs of forecasters and test the rationality of forecasts using a large dataset of forecasts. The dataset comprises vintages of forecasts by four international organisations (the IMF, the World Bank, the EC and the OECD) and, for reference, the average forecast of respondents to the monthly survey by Consensus Economics. One strength of our dataset is that it covers as many economies as possible, allowing us to identify features of the forecasts that are general, rather than specific to particular groups of economies. Another strength is that it covers forecasts as recently as 2025, which makes the results relevant to current practice at international organisations. For all pairs of forecasters analysed, we find statistically significant differences in accuracy for only a few economies. In contrast, for each forecaster, we find statistically significant departures from forecast rationality for the majority of economies. The nature of this irrationality varies by economy, but often takes the form of optimistic bias, revisions that are too large, or forecasts that are too extreme.

Second, we extend the literature on comparing accuracy by deriving a new decomposition. We show that the difference in squared errors of a pair of forecasts can be decomposed into contributions from disagreement between the forecasts and the accuracy of the average forecast. Forecast accuracy tends to differ by more at longer horizons. Using our decomposition, we show that this occurs because of lower accuracy at longer horizons, and to a lesser extent, more disagreement at longer horizons.

Finally, we investigate two explanations for the rejection of rationality that may be especially relevant to international organisations. One explanation is that international organisations often make forecasts that are conditional on future outcomes of a set of explanatory variables. We derive conditions under which this results in the failure of rationality tests. Another explanation is that organisations may make ‘modal’ forecasts, which means they report the most likely outcome rather than the expected value of the outcome. Often the economy is expected to grow moderately, but there is a small chance of negative growth due to a financial crisis or other shock. The modal forecast is simply for moderate growth, while the expected value takes into account the chance of a crisis. We present empirical evidence suggesting that making modal forecasts could introduce a small optimistic bias in the forecasts, but the magnitude is small.

The remainder of this paper is organised as follows. Section 2 reviews the literature, while Section 3 describes our dataset. Section 4 compares the accuracy of the forecasters, including presenting our new decomposition of differences in forecast accuracy. The results of rationality tests are presented in Section 5. Section 6 investigates two explanations for the rejection of rationality tests. Finally, Section 7 concludes.

2.1 Comparing accuracy across forecasters

A number of papers have compared the accuracy of real GDP growth forecasts by different international organisations. Some papers compare measures of forecast performance, but do not formally test for differences in accuracy. For example, Hong and Tan (2014) find that the United Nations growth forecasts performed slightly better than those by the IMF and World Bank over 2000 to 2012. Fioramanti et al (2016) find that EC growth forecasts were more accurate than market consensus and performed comparably to those of other international organisations over 2000 to 2014. Other papers formally test for accuracy. Timmermann (2007) finds that there are few economies where IMF and Consensus had statistically significant differences in accuracy. Melander, Sismanidis and Grenouilleau (2007) also find few economies where the IMF, OECD, Consensus or EC had significant differences in performance, using a sample of European economies from 1990 to 2005. Abreu (2011) uses a sample of nine advanced economies over 1991 to 2009, and finds few statistically significant differences in accuracy when comparing international organisations (IMF, EC, OECD) to private-sector forecasters (Consensus or The Economist). More recently, Akgun et al (2024) compare the accuracy of IMF and OECD forecasts from 1998 to 2016, finding few economies for which the forecasts differed in accuracy.

The lack of statistically significant differences in accuracy in the previous literature may partly reflect the small sample sizes used in many of those studies. This argument was made, for instance, by Timmermann (2007). Our paper includes more recent forecasts than earlier studies, increasing our sample size. Another explanation for the lack of significant differences is that the forecasts themselves tend to be quite similar. This similarity could arise because the forecasters independently make similar forecasts to one another due to having similar information sets. It could also arise if forecasters deliberately adjust their forecasts to be closer to others. The empirical evidence on whether forecasters deliberately ‘herd’ by making their forecasts more similar to one another is mixed (Clements 2018).

2.2 Forecast rationality

The literature describes a number of related notions of rationality. We follow Elliott, Komunjer and Timmermann (2008) in defining a forecaster to be ‘rational’ if their forecasts minimise the conditional expectation of the forecaster's loss function given their information set. To formalise this, let:

• $y_{t+h}$ be realised real GDP growth in period $t+h$
• $y_{t+h}{}_{|t}$ be a forecast made in period t for real GDP growth in period $t+h$
• $v_{t+h}{}_{|t}\equiv y_{t+h}-y_{t+h}{}_{|t}$ be the forecast error, which is the actual less the forecast.
• $L:ℝ\to {ℝ}_{\ge 0}$ be the forecaster's loss function
• I_t is the forecaster's information set at period t

The forecaster is rational if:

Here, the conditional expectation is computed using the conditional probabilities implied by the true data-generating process (DGP). This is quite a stringent definition, as it assumes that the forecaster has a correctly specified model. Any mis-specification, such as the forecaster viewing growth as stationary when it is instead subject to breaks in the mean, would violate forecast rationality.

The literature often focuses on the special case where the forecaster's loss function is the squared error, $L\left(v_{t+h}{}_{|t}\right)=v_{t+h|t}^2$. Jointly assuming forecast rationality and a squared error loss function is equivalent to assuming that the forecast equals the conditional expectation of real GDP growth given the forecaster's information set. This is called ‘strong efficiency’ by Nordhaus (1987):

If the forecaster's forecasts equal the conditional expectation of growth given their information set, then the forecast errors must be orthogonal to any vector x_t of variables known at time t. This orthogonality condition is called ‘strong rationality’ by Stekler (2004) and is the basis for the regression-based rationality tests common in the literature.^[2] If the forecast error satisfies Equation (1), where $u_{t+h}$ is a mean-zero error, then orthogonality states that the parameter vector $\theta$ is the zero vector:

The literature on growth forecasts contains various regressions of forecast errors on information available at the time of the forecast. Much of this literature tests the rationality of the average of private-sector forecasters (Coibion and Gorodnichenko 2012, 2015) or individual private-sector forecasters (Bordalo et al 2020; Broer and Kohlhas 2024). However, a number of papers test the rationality of international organisations' forecasts for multiple economies, as we do.^[3] These papers include rationality regressions with a variety of explanatory variables, including:

• Regressions of forecast errors on an intercept to test for bias. The results are mixed. For the IMF, some papers have found an optimistic bias for many economies (Artis 1996; Timmermann 2007), while others have found optimistic biases for some economies and pessimistic biases for others (Schavey and Beach 1999; Takagi and Kucur 2006). For the OECD, Vogel (2007) finds that current-year forecasts are unbiased, while year-ahead forecasts have an optimistic bias. Tsuchiya (2023) finds that the same is true of World Bank forecasts since the global financial crisis. Finally, for the EC, Melander et al (2007) finds no evidence of bias for all but a few economies.
• Regressions of forecast errors on past revisions to test if the forecaster tends to revise forecasts by too much or too little. Ashiya (2006) performs this test on IMF and OECD forecasts for G7 economies, rejecting rationality in 40 per cent of cases, typically because the revisions are too small. In contrast, Aktuğ and Rezghi (2025) find that IMF real GDP growth forecasts tend to be overrevised.
• Regressions of forecast errors on an intercept and forecast. This can be used to test if the forecasts are ‘too extreme’ in the sense that they should be scaled up or scaled down in magnitude. Equivalently, one can regress actual outcomes on an intercept and forecast, as in Mincer and Zarnowitz (1969).^[4] Tsuchiya (2023) finds that this test is rejected for one-year-ahead forecasts by the World Bank, while Vogel (2007) finds the same for the OECD.
• Regressions of forecast errors on errors known at the time. If forecast errors are positively correlated with past forecast errors known to the forecaster, then the forecaster is failing to learn from their mistakes. Timmermann (2007) finds that IMF forecast errors have positive sample autocorrelations, but do not formally test the null of no autocorrelation. Melander et al (2007) do not reject the null of rationality for EC forecasts for most economies.
• Regressions of real GDP growth forecast errors on actual outcomes or forecasts for other explanatory variables. For example, Ghirelli, Pérez and Santabárbara (2025) show that IMF and European Central Bank forecasts for real GDP are correlated with their forecasts for monetary policy. Carrière-Swallow and Marzluf (2023) show that IMF real GDP growth forecast errors are correlated with financial variables.

We test the rationality of multiple international organisations and the Consensus mean for as many economies as possible. In contrast, much of the literature tends to test the rationality of only one or two international organisations. A notable exception is Abreu (2011), who tested the rationality of forecasts for nine advanced economies over 1991 to 2009. A key takeaway from Abreu (2011) is that the departures from rationality tend to be similar across forecasters.^[5] This is useful, but leaves open the possibility that Abreu's findings are specific to these economies during the pre-financial-crisis period. By covering more economies over a longer time period, we can have more confidence that we have identified general features of international organisations' forecasts.

The literature also studies the consequences of departures from rationality for the forecasts. For example, Clements (2022) finds evidence that irrationality in forecasts contributes to persistent differences in accuracy across forecasters. Some literature studies the implications of irrationality for the behaviour of the economy. Bordalo, Gennaioli and Shleifer (2018) show that a model of irrationality called ‘diagnostic expectations’ can explain credit cycles, while Bordalo et al (2021) show that this model can explain asset price bubbles.

2.3 Explaining rejection of rationality

An extensive literature provides explanations for the rejection of the null of rationality in standard forecast rationality tests. Much of that literature relates to consumers, firms or financial market participants. However, the explanations for irrationality at international organisations may be quite different, given that they make forecasts using different methods and face different incentives. Our paper investigates two explanations that are based on specific features of international organisations, and of similar institutions such as central banks and ministries of finance.

The first explanation is that international organisations often make ‘conditional forecasts’, defined as forecasts that are conditional on assumed future paths for a set of explanatory variables. In our paper, we characterise the circumstances under which making conditional forecasts will cause standard rationality tests to be rejected. We show that these tests may be rejected because the assumptions about the explanatory variable may be irrational, or because the ‘response function’ that relates growth to the explanatory variables has certain properties, such as strict convexity or strict concavity.

A related paper, Faust and Wright (2008), develops a method that tests if a set of observed growth forecasts are rational given a set of observed interest rate assumptions. Their approach is useful, as forecasters who make conditional forecasts will be interested to know if they are making rational conditional forecasts. However, their method can only be applied where the assumptions about the explanatory variable are observed, which is not true of some forecasters, and not true of assumptions about qualitative developments such as a war. Moreover, their method can only be used when the relationship between growth and the explanatory variable is linear, which may not be the case for some variables, such as those related to the climate (e.g. temperature). For this reason, researchers are likely to continue running standard rationality tests on conditional forecasts. In such cases, our theoretical results provide guidance on whether the conditional nature of the forecasts can explain the failure of these standard tests.^[6]

The second explanation is that international organisations may be reporting the conditional mode of future growth, rather than the conditional mean. Reporting the conditional mode is rational if one's loss function applies no penalty to being exactly correct (to the published precision of the forecasts, usually 1 decimal place) and applies the same penalty to being incorrect by any amount. If the conditional distribution is symmetric and unimodal, then the mean and mode will coincide, otherwise the mean and mode may differ. Two papers have tried to explain optimistic bias in growth forecasts in a similar way. Firstly, Kangur et al (2019) find that IMF forecasts for the euro area have an optimistic bias. They suggest that the IMF may be making modal forecasts, which will tend to be higher than mean forecasts because the distribution of growth outcomes is negatively skewed. Secondly, Burgess et al (2021) find that the distribution of real GDP per capita growth is negatively skewed in most economies, and that this could in principle explain the bias.^[7] They then show that, empirically, mean growth and median growth tend to be similar, and argue on this basis that the bias in IMF forecasts cannot be fully explained by the skew of growth outcomes.^[8] Our paper advances on these previous efforts in two ways:

1. We argue that the practice of making modal forecasts could lead to bias (as noted by Kangur et al (2019)), but we additionally argue this practice could cause the failure of rationality tests by causing the forecast errors to be correlated with variables known on the forecast date.
2. We attempt to quantify the effect of making modal rather than mean forecasts on bias. Burgess et al (2021) did a similar exercise, but they compared the mean with the median instead. However, this provides little information about the effect of making modal forecasts, because even when the mean and median are similar, the mode could be quite different.^[9]

We examine annual real GDP growth forecasts at horizons of up to two years ahead. To this end, we assemble a large dataset incorporating the forecasts of prominent international organisations and the average of private-sector forecasters.

A strength of our dataset is that it covers as many economies as possible, and contains forecast vintages as recently as December 2025. The large set of economies allows us to draw conclusions about international organisations' forecasts in general, rather than worrying that our conclusions only apply to forecasts for a specific unusual group of economies, such as G7 economies. The inclusion of recent forecast vintages that were unavailable to previous researchers increases our sample sizes, which increases the relevance of our results to current practice at international organisations, and increases the power of our tests.

We use short-term forecasts for annual real GDP growth from four international organisations:

• The IMF publishes forecasts in the World Economic Outlook (WEO). A full WEO is released in each April and each October, which we call the ‘Spring’ and ‘Fall’ editions respectively, and provides forecasts for almost all economies. A WEO update is published in each January and July, which provides forecasts for selected economies and groups of economies.
• The World Bank publishes forecasts in the Global Economic Prospects (GEP) publication each January and June. The GEP provides forecasts for emerging and developing economies, and for selected advanced economies.
• The EC publishes forecasts for European Union member states and for selected other economies each Spring and Autumn, and sometimes at other times as well.
• The OECD publishes forecasts for OECD members and major non-members in the Economic Outlook (EO) publication each December and June, and occasionally at other times.

We also use the average of the forecasts made by respondents to the Consensus survey. The respondents are a mix of global institutions (e.g. global banks) and economy-specific institutions (e.g. local banks or think tanks). Around 20–30 forecasters are surveyed for each economy.

Table 1 summarises the forecasts available. This rich variety of data ensures a well-rounded view on the forecast performance of all major institutional and private economic actors, allowing us to draw general conclusions about the economic forecasting profession in the last 30 years. Both WEO and Consensus increased the number of economies for which forecasts are available over time, which means our evaluation is able to study more economies than similar evaluations conducted earlier. The GEP and EC produced forecasts for a mostly constant set of economies over time.

To assess forecast performance we must compute forecast errors at different horizons. We define the forecast horizon as the length of time from the publication date of a forecast to the end date of the target year (see Appendix A.1). We compute forecast errors as the latest vintage of the actual outcome minus the forecast (see Appendix A.2), except for some rationality tests, as discussed in Section 5.2.1.

4.1 Method

We compare the accuracy of different forecasters using Diebold-Mariano (DM) tests (Diebold 2015). We use WEO forecasts as our benchmark, and compare each other forecaster to the WEO. We perform this comparison separately for each economy and forecast horizon. WEO forecasts are available for a comprehensive set of economies, so using them as our benchmark maximises the number of economies for which we can perform comparisons.

In a DM test, the null hypothesis is that the pair of forecasters have equal forecast accuracy (i.e. equal expected loss), while the alternative is unequal forecast accuracy.

A ‘loss differential’ is the difference between the loss of the two forecasters, $d_{t+h|t}\equiv L\left(v_{t+h|t}^{\text{other}}\right)-L\left(v_{t+h|t}^{\text{WEO}}\right)$. The null hypothesis of the DM test is equivalent to stating that, in a regression of the loss differential on an intercept, the intercept is zero:

Under the assumption that the loss differential is covariance-stationary, the OLS estimator of the intercept is asymptotically normal. We perform the DM test by performing a two-sided t-test of the null that $\mu =0$. We use heteroskedasticity and autocorrelation consistent (HAC) standard errors, as we want to allow for the possibility that the loss differentials are serially correlated.^[10]

There are two further econometric complications. Firstly, our approach is only asymptotically valid, but the sample sizes in macroeconomic forecast evaluation are often modest. As Kiefer, Vogelsang and Bunzel (2000) explain, conducting t-tests with HAC standard errors in finite samples leads to rejecting null hypotheses more often than the chosen level of significance. The importance of this issue depends on the forecasters being compared. As shown in Table 1, our dataset includes many forecast vintages for the IMF, OECD and Consensus, but fewer for the EC and World Bank.

Secondly, the forecasts by international organisations and the private sector are often informed by models whose parameters are estimated on finite samples of input data. Even where the forecaster relies solely on judgement, they can be viewed as using an implicit mental model that is informed by the finite sample of input data. Diebold (2015) discusses alternative tests that aim to address this issue, and argues that using a DM test rather than the alternative tests is appropriate provided that the forecast errors are approximately covariance-stationary.

4.1.1 Timing issues

When computing loss differentials for an economy, we must ensure that the two forecasters record that economy's growth rates over the same period. Fortunately, WEO, EO, EC and Consensus all record each economy's annual growth rates over the same 12-month periods (see Appendix A). There are some economies where WEO and GEP record growth rates over different periods, so we do not compute loss differentials for these economies.^[11] As an additional check, whenever both forecasters have data on actual outcomes, we check that the actual outcomes are the same.^[12]

We also need to ensure that the loss differentials are computed with forecasts made around the same time. When comparing Consensus with an international organisation, we always compute loss differentials using forecasts made in the same month for the same target year. For example, if we wanted to compute a sequence of 14-month-ahead loss differentials, we'd compute one differential as ‘October 2020 WEO forecast for 2021’ with ‘October 2020 Consensus forecast for 2021’, and compute the next differential as ‘October 2021 WEO forecast for 2022’ with ‘October 2021 Consensus forecast for 2022’.

When comparing international organisations with each other, we compute loss differentials using forecasts made in the same or adjacent months for the same target year. For example, we could compute one differential as ‘October 2020 WEO forecast for 2021’ with ‘November 2020 EC forecast for 2021’, and compute the next differential as ‘October 2021 WEO forecast for 2022’ with ‘November 2021 EC forecast for 2022’. A limitation of this approach is that one forecaster will have an additional month of information relative to the other, giving them an unfair advantage in our comparisons of forecast accuracy. In practice, EC always has one more month of information than WEO, GEP has the same or less information than WEO, and the OECD sometimes has more information and sometimes less (see Table 2).^[13]

4.2 Results

4.2.1 Diebold-Mariano tests for individual economies

We conduct a DM test for each economy and horizon. We report the p-value of each DM test. In addition, we report the ratio of the root mean square forecast error (RMSFE) of one forecaster to the RMSFE of the WEO.^[14]

For the vast majority of economies, we find no statistically significant difference at a 5 per cent level in forecast accuracy (Figure 1). For the one-year horizon there are a handful of economies where Consensus and EC are found to outperform or underperform WEO, and at the two-year horizon there are a few economies where each forecaster outperforms or underperforms.

Since we perform DM tests separately for dozens of economies with a 5 per cent level of significance, even if the null hypothesis is true we would expect around 5 per cent of economies to show a significant result. This would be just two or three economies for EC, EO and Consensus, and zero or one economies for GEP. Hence, the handful of economies with statistically significant differences could be spurious. If one of the forecasters was genuinely better or worse in general, we would expect to see that forecaster outperform or underperform for many economies, but this is not the case. The results for each individual economy at a one-year-ahead horizon are available in Table B.1.

The lack of significant results could occur because there is genuinely no meaningful difference in performance, or because our sample sizes are too small to detect any performance differences. Our paper benefits from having access to more recent forecast vintages than earlier studies, increasing the power of our tests (see Section 2.1). Nevertheless, the sample sizes are modest. This is especially true for comparisons of GEP or EC, whose forecast vintages start in 2010 and 2011 respectively.

4.2.2 Cross-economy evidence

The hypothesis tests considered each economy in isolation. A natural question is whether one can better detect differences in forecast performance by drawing on the results for many different economies. Unfortunately, it is not straightforward to use the observed performance for different economies in a formal hypothesis test. The reason is that actual outcomes are correlated across economies (e.g. high growth in France tends to coincide with high growth in Germany), and forecasts tend to be correlated across economies too (e.g. if the IMF expects high growth in the United States it likely expects high growth in Mexico too). As such, the forecast errors for different economies do not provide independent sources of information.

Nevertheless, considering the results for many economies together provides suggestive evidence for differences in performance between forecasters. This suggests that, while tests of differences in forecast accuracy typically fail to reject the null at present (Section 4.2.1), future studies with longer sample periods may find differences more often.

First we compare Consensus to WEO. For each horizon, we sort economies from the lowest ratio of Consensus RMSFE to WEO RMSFE to the highest. At zero or one-year horizon, roughly 75 per cent of economies had a lower Consensus RMSFE than WEO RMSFE (Figure 2). This provides suggestive evidence that Consensus mean forecasts are more accurate than WEO in general. Of course, absent a formal hypothesis test, we cannot say how likely it is that Consensus outperformed WEO by this much over the sample period simply by chance. If the Consensus mean is more accurate in general, it might reflect that combining forecasts of different forecasters tends to improve accuracy (e.g. Clements and Harvey 2009).

Second, we compare EO to WEO using the same exercise. At a one-year horizon, around 75 per cent of economies had a higher EO RMSFE than WEO RMSFE (Figure 3). This suggests WEO may be more accurate than EO in general. This cannot be attributed to differences in the timing with which each forecaster publishes their forecasts, because although the EO has a one month timing advantage in some vintages, it has a one month timing disadvantage in other vintages (Table 2).

We also compare both EC and GEP to WEO. These do not have a clear pattern in either direction. These point estimates are noisier, as the comparisons are based on a smaller set of vintages than EO or Consensus.

4.3 Decomposing differences in forecast performance

A loss differential measures the difference in accuracy of two forecasts for the same outcome. We show that in the case of squared error loss, a loss differential can be decomposed into two components: the accuracy of the average forecast and the disagreement between forecasters. This decomposition can be applied to any comparison of pairs of forecasters, as it is not specific to a particular variable (such as real GDP growth) or a particular type of forecaster (such as international organisations).

Theorem 1. Suppose the loss differential between a forecaster c and another forecaster b is computed with squared error loss:

Then the absolute loss differential can be written:

Proof. See Appendix B.1.

If we assume that the loss differential, the error in the average forecast, and the absolute difference in forecasts are all non-zero, we can take natural logs, giving:

Before taking logs, we omit observations for which the absolute loss differential or either component is less than 0.01. We can use this decomposition to understand how the relative performance of a pair of forecasters for an economy depends on the forecast horizon. To do this, we compute sample means of the log absolute loss differentials across all forecast dates, and sample means of the components.^[15] Figure 4 shows this decomposition for the Consensus and WEO forecasts for the United States. At longer horizons, the log absolute loss differential is less negative, which means the absolute loss differential is larger. Our decomposition shows that this occurs because longer horizons have both larger amounts of disagreement and larger errors in average forecasts.

The decompositions for individual economies are not too informative in our application. As discussed in Section 4.2.1, we find statistically significant differences in accuracy for only a few economies, possibly reflecting the modest sample sizes of forecasts. However, the decompositions for individual economies may be informative in future studies with larger sample sizes, including studies on other variables or types of forecaster.

We also consider the results for many economies together, as this may reveal patterns that are hard to detect for individual economies. To do this, we perform the decomposition for each individual economy, and then summarise the results by estimating a panel regression (Equation (2)). The dependent variable is the log absolute loss differential for a particular economy i, forecast month t and horizon h. The explanatory variables are economy fixed effects and dummies for each horizon. We include a horizon dummy for each horizon h = 1,...,H, which means the base case is a horizon of h = 0 years.^[16] The economy fixed effects ${\alpha }_i$ are included because the magnitude of loss differentials is likely to differ across economies.^[17]

We then perform two more panel regressions with the same explanatory variables, but where the dependent variable is a term from our decomposition (Equation (3)).

Figure 5 shows the estimated coefficients on the horizon dummies for the comparison of Consensus to WEO. Figures 6, 7 and 8 do the same for comparisons involving EC, GEP and EO respectively. Across all four pairs of forecasters, we find that, relative to the zero-year-ahead base case, the log absolute loss differentials are larger at a one-year horizon, and larger still at a two-year horizon. Our results show that, over the sample period, this occurred because the errors in the average forecast were larger at longer horizons, and to a lesser extent, because disagreement between the forecasters was larger at longer horizons. Whether this is true in the population is hard to assess.^[18] It makes sense that the forecast errors and disagreement are both larger at longer horizons, since forecasters have less information at these horizons. In particular, the forecasts at a zero-year horizon typically benefit from having data for some quarters of the year.

Our results suggest that practitioners should be cautious when interpreting the size of loss differentials at different horizons: if the difference in squared errors becomes larger in magnitude at longer horizons, this does not necessarily indicate changes in the relative performance of the two forecasters, but rather could arise simply because both forecasters are less accurate at longer horizons.

5.1 Methods of testing forecast rationality

We perform rationality regressions on fixed-horizon forecasts to test the joint assumption of rationality and squared error loss. Specifically, we test the null that the parameter vector is the zero vector in Equation (1).

The key assumption is that the error in this regression is covariance stationary.^[19] However, we allow for the possibility that the error in this regression is heteroskedastic and autocorrelated.^[20] Under our assumptions, the OLS estimator of $\theta$ is asymptotically normal, so we can perform asymptotically valid t-tests and Wald tests of hypotheses concerning $\theta$ using a HAC estimator of the covariance matrix of the OLS estimator (see the introduction of Kiefer et al (2000)). We use the HAC estimator in the ‘sandwich’ package in R with the default options (Zeileis 2004), just as we did in the DM tests.

Our rationality test raises similar econometric issues to the DM test (as discussed in Section 4.1), which is to be expected given that both are tests about the coefficients of a regression involving forecast errors. Firstly, the test has an asymptotic justification, but as with any evaluation of macroeconomic forecasts, we only have modest sample sizes. This problem is less severe here than some other papers, as our sample includes recent forecast vintages that were unavailable to researchers in previous years. Secondly, the forecasts may be produced using explicit models that are estimated on finite samples, or using judgement that is informed by finite samples. Some authors, such as West and McCracken (1998), have shown that even in the presence of parameter estimation error, one can test hypotheses about the forecast errors that would arise absent that error. However, we are not able to conduct such an exercise as there is no single explicit model for each forecaster. Hence we do not try to account for estimation error in model parameters, following Rossi and Sekhposyan (2016).

In brief, we find that international organisations' growth forecasts fail rationality tests for most economies. The source of the failure of the test varies by economy. In many cases, it arises because the forecasts are too extreme, the forecast revisions are too large, or the forecasts have an optimistic bias.

5.2 Regression specifications

5.2.1 The general specification

Our general regression specification is:

This regression includes the following elements:

• The dependent variable, $v_{t+h|t}$, is the error in the forecast for period (t + h) made in period t.
• The forecasts for the target year. This includes the forecast that was used to compute the error, $y_{t+h|t}$, and one earlier forecast, $y_{t+h|t-1}$.

• Errors in h-step-ahead forecasts. We let p denote the number of periods it takes for statistical agencies to publish the actual real GDP growth outcome after the end of a period. This implies that, in period t, the latest actual known to the forecaster is $y_{t-p}$. Our regression includes the latest forecast error known to the forecaster, $v_{t-p|t-p-h}$, and one earlier error, $v_{t-p-1|t-p-h-1}$.

In the notation of Equation (1), the vector x_t comprises the intercept, forecasts for the target year and errors in h-ahead forecasts, and the vector of parameters is $\theta =\left(\alpha ,{\beta }_0,{\beta }_1,{\gamma }_0,{\gamma }_1\right)$. The rationality test is a Wald test of $H_0:\theta =0$ against $H_1:\theta \ne 0$.

It would be straightforward to add more lags of forecasts or errors to this regression. We choose not to, as it may result in the parameters being estimated very imprecisely given the modest sample sizes of macroeconomic forecasts available.

When we estimate these regressions, we are careful to ensure the regressors only reflect information available to the forecaster at the time. For example, in one observation in this dataset, the dependent variable is the error in the April 2020 WEO forecast for 2022. The regressors include two forecasts for that target year: the April 2020 WEO forecast for 2022, and the January 2020 WEO forecast for 2022. Both of these forecasts would have been known to the IMF at the time. The regressors also include two errors in two-year-ahead forecasts. As of April 2020 WEO, if the most recent actual for a particular economy is 2019, the two latest two-year-ahead errors are the error in the April 2018 forecast for 2019 and the error in the April 2017 forecast for 2018. These two errors are computed with the April 2020 vintage of actuals, rather than the latest vintage.^[21]

5.2.2 Interpreting the general specification

Interpreting the parameters of the rationality regression is difficult. In particular, the intercept is not the forecaster's unconditional bias, but rather is the bias conditional on past forecasts and past errors all being zero. To facilitate interpretation, it is useful to rearrange Equation (4) as follows.^[22]

The right-hand-side terms each have a nice interpretation.

• The first term is the unconditional bias, which is the expectation of the forecast error. This differs from the intercept, which is the conditional bias given that all the explanatory variables are zero.
• The contribution of demeaned forecasts tells us if the forecasts tend to be ‘too extreme’ or ‘not extreme enough’. For example, suppose the coefficient on the latest forecast is positive ${\beta }_0>0$. If the forecaster predicted higher growth than they usually do, $y_{t+h|t}>E\left[y_{t+h|t}\right]$, then the forecast error is likely to be positive, $v_{t+h|t}>0$, so the forecast error would have been smaller if the forecast were even higher. Conversely, if the forecaster had predicted lower growth than usual, the forecast error would likely be negative, so the forecast error should have been even lower. For this reason, we describe ${\beta }_j>0$ as indicating that the jth most recent forecast was not extreme enough, and ${\beta }_j<0$ as indicating that it was too extreme.
• The contribution of demeaned forecast errors tells us if the forecaster's h-period-ahead errors are related to the latest h-period-ahead-errors known on the forecast date, which would indicate that the forecaster is not making efficient use of the information contained in their past errors. For example, the forecaster may be slow to recognise that a change in the mean of the series has occurred, and hence persistently overpredict growth over a long period.

We cannot estimate the rearranged regression directly, because this would require knowledge of the population mean of forecasts and the population mean of errors. Instead, we estimate the original regression (Equation (4)), but we bear the rearranged regression in mind when interpreting the coefficients. Since we cannot estimate the rearranged regression directly, we cannot use it to test the hypothesis of zero unconditional bias. Instead, we need to estimate a regression of errors on an intercept alone, as in Section 5.3.3.

5.3 Results

5.3.1 General specification

We perform a Wald test of the null hypothesis of forecast rationality (all parameters equal zero) against the alternative of irrationality. For most forecasters, we reject rationality of one-year-ahead and two-year-ahead forecasts at a 5 per cent level for most economies (Figure 9). The exception is EO, where we only reject rationality for half of economies at one-year-ahead and a minority of economies at two-years-ahead. The samples of forecasts available for EO are smaller than for WEO and Consensus, which reduces the power of tests for EO.

Having found widespread irrationality, we now investigate the source of irrationality. First, we perform Wald tests of the null that the coefficients on past forecasts are zero, which we reject for most economies (Figure 10). As explained in Section 5.2.2, this indicates that the forecasters have a tendency to make forecasts that are too extreme or not extreme enough.

Second, we perform Wald tests of the null that the coefficients on past errors are zero. Some forecasters, including the IMF, OECD and EC, only review their forecast performance every few years (Genberg et al 2014), so it is plausible that they would fail to make efficient use of any of the information contained in their recent errors. We reject the null that the coefficients on past errors are zero for a minority of economies (Figure 11). Since we conduct a separate hypothesis test at the 5 per cent level for each economy and forecaster, we would expect the Wald test to be rejected for 5 per cent of economies if the null hypothesis were true for all economies. These results provide tentative evidence that a failure to make use of the information in past errors is a source of irrationality for some forecasters for some economies, but suggest this issue is not too widespread.

5.3.2 Intercept and past forecasts specification

The Wald tests of the general specification suggest that failing to account for information in past errors is a relatively unimportant source of irrationality. For this reason, we consider a special case of the general specification where these past errors are removed:

We can view this regression through the lens of forecast revisions, which are changes in forecasts for a given target year from one vintage to the next. We rearrange the specification with forecasts only to obtain a specification with a forecast revision and a past forecast (Equation (7)).

The coefficient on the latest forecast revision tells us if the forecast revisions tend to be too large or too small. For example, if ${\Psi }_0>0$, it implies that if the forecaster revised up their forecast in the last period, $\left(y_{t+h|t}-y_{t+h|t-1}\right)>0$, then the forecast error is likely to be positive, $v_{t+h|t}>0$, which implies that the forecast error would be smaller if the upward forecast revision had been larger. Hence whenever ${\Psi }_0>0$ we say that the latest forecast revision tends to be too small, indicating that the forecaster tends to underreact to the news received between t – 1 and t.^[23]

We test rationality in Equation (7) by performing a Wald test of the null that all coefficients are zero (Figure 12). We reject rationality for many.

We know that this irrationality is not solely due to bias, because if we do a Wald test of the coefficient on the revision and on the past forecast, we reject the null for the majority of economies for all forecasters. To understand the role of revisions versus past forecasts, we do t-tests of each, noting that for some economies neither variable may be individually significant even though the two variables are jointly significant.

The t-tests of the coefficient on the past forecast show that this coefficient is insignificant in most economies (Figure 13). Where the t-tests are significant, the forecasts are typically too extreme. When the forecaster expects higher-than-average growth, they could improve accuracy by making a less optimistic forecast, and when the forecaster expects lower-than-average growth, they could improve accuracy by making a less pessimistic forecast. Halperin and Mazlish (2025) find, using regressions that pool across economies, that Consensus mean forecasts are too extreme. Their finding is consistent with our results, which show that Consensus mean forecasts are too extreme for a sizable minority of economies.

The t-tests show that the coefficient on revisions is significant in many economies (Figure 14). However, the direction of the inefficiency differs. In each international organisation, it is more common for revisions to be too large than for them to be too small. This is similar to the finding that individual private-sector forecasters make overly large forecast revisions (Bordalo et al 2020; Broer and Kohlhas 2024). For the average of Consensus forecasts, it is more common for revisions to be too small, consistent with previous evidence that the average of private-sector forecasts has overly small forecast revisions (Coibion and Gorodnichenko 2012, 2015). However, our finding contrasts with Halperin and Mazlish (2025), who find that average private-sector forecasts for real GDP have overly large revisions at horizons of two or more years.

An alternative to regressing errors on forecasts (as we have just done) would be to regress forecast revisions on their own lags. Nordhaus (1987) shows that if forecasters incorporate all new information each time they update their forecasts, then the sequence of forecast revisions for a fixed target date will be serially uncorrelated. Such a sequence of forecast revisions is said to satisfy ‘weak efficiency’, which is a necessary condition for rationality. Hadzi-Vaskov et al (2021) find, using WEO real GDP growth forecasts, that the sequence of revisions for a given target year are negatively autocorrelated. This is consistent with our finding that WEO revisions are too large for a sizable minority of economies.

5.3.3 Intercept only specification

We also estimate a specification with the intercept alone. By doing this we are not claiming that the revision and past forecast are absent from the true model. As we saw earlier, the forecast revision and past forecast are jointly significant in most economies. Rather, we estimate this specification so that we can test for unconditional bias by performing a t-test of the intercept, as is standard in the literature. Although we could provide point estimates of unconditional bias in each economy using the previous specifications, we were not able to formally test if it was equal to zero.

For all forecasters, we find that there is no significant unconditional bias for most economies at the one-year or two-year horizon (Figure 15), though bias is more common at a two-year horizon for GEP and WEO. Where bias is present, it is almost always optimistic rather than pessimistic.

To summarise, we have shown that international organisations' growth forecasts fail rationality tests for the majority of economies. The reason for the rejection of this test differs by economy, but common reasons are that the forecasts are too extreme (Figure 13), revisions being too large (Figure 14), and optimistic bias (Figure 15).

In Section 5, we conducted rationality tests for the forecasts from international organisations and Consensus by regressing forecast errors on an intercept, past forecasts for the target year and past forecast errors. Where the null hypothesis of the rationality test is false, then at least one of the following must be true:

• the forecaster does not satisfy forecast rationality;
• the forecaster must not have a squared error loss function; or
• the vector of explanatory variables, x_t must not be in the forecaster's information set.

If the null is false then this is likely because the forecaster does not satisfy forecast rationality or does not have squared error loss, rather than because the explanatory variables are absent from their information set. The reason is that the explanatory variables in our regressions are the forecaster's own forecast or the errors in their own forecasts, which are almost certainly known to the forecaster.

An extensive literature provides explanations for why different types of agents, such as consumers or firms, may make irrational forecasts. However, the explanations for irrationality at international organisations may be quite different, given that they make forecasts using different methods and face different incentives. In this section we consider two explanations that are based on specific features of international organisations:

1. International organisations make forecasts that are conditional on assumed future values of a variety of explanatory variables. Such forecasts violate forecast rationality under fairly general conditions (Section 6.1).
2. International organisations may report a ‘modal’ forecast, rather than a mean forecast (Section 6.2).

6.1 Conditioning assumptions

6.1.1 The rational expectations approach

Future growth, $y_{t+h}$, depends on the future values of a variety of explanatory variables, $x_{t+h}$, such as crude oil prices, exchange rates, nominal interest rates or government spending. As discussed in Section 5, if a forecaster has rational expectations and a squared error loss function, then their forecast will be a conditional expectation of growth given their information set.

By the law of iterated expectations, this rational forecast equals:

That is, the forecaster can make a rational forecast by:

1. Computing a conditional expectation of future growth given a value of the future explanatory variables, $E\left[y_{t+h}|x_{t+h},I_t\right]$, for each possible value of the explanatory variables; and then
2. Averaging this conditional expectation, $E\left[y_{t+h}|x_{t+h},I_t\right]$, over the various possible paths of the explanatory variables, weighted by the conditional probabilities of each possible value of the explanatory variables given the information set.

For example, one would compute the expected path for growth for each possible scenario for crude oil prices, and then compute a weighted average of those different paths for growth weighted by the likelihood of each crude oil price scenario.

6.1.2 The current approach at international organisations

International organisations make forecasts that are conditional on assumed future values of the explanatory variables.

In each WEO, the IMF publicly states its assumptions for exchange rates, oil prices, bond yields, monetary policy and fiscal policy. In addition, the IMF makes assumptions about other issues on an ad hoc basis. In the April 2021 WEO, for instance, the IMF made assumptions about COVID-19 vaccination and testing (IMF 2021).

The EC and OECD follow a similar practice of making assumptions about future explanatory variables such as exchange rates and then making growth forecasts conditional on those assumptions (Genberg et al 2014). Though central banks don't form part of our dataset, it's worth noting that they follow this practice as well (e.g. Kohler 2023). It's unclear what practices private-sector forecasters follow, as their forecasting processes are rarely as well documented publicly as those of international organisations.

6.1.3 Current practice typically differs from rational expectations

We will use the term ‘current practice’ forecast to refer to the conditional expectation of future growth $y_{t+h}$ conditional on an assumed future value of the explanatory variable $\widetilde{x_{t+h}}$.

Current practice will typically be inconsistent with rational expectations. This is true whether the assumed values of the explanatory variables are rational forecasts or not. We will consider each case in turn.

Case with rational forecasts for explanatory variable

First, suppose that the forecaster's assumed future value of the explanatory variable is a rational forecast for that variable, $\widetilde{x_{t+h}}=E\left[x_{t+h}|I_t\right]$. Then the current practice forecast will be:

Whether the current practice forecast of growth coincides with the rational forecast depends on two factors. Firstly, what is the relationship between future growth and the future explanatory variable? This relationship is described by the ‘response function’, $r\left(x_{t+h}\right)=E\left[y_{t+h}|x_{t+h},I_t\right]$. Secondly, what is the conditional distribution of the future explanatory variable given the information set? This is described by the conditional probability density function (pdf) $p\left(x_{t+h}|I_t\right)$.

The next theorem provides sufficient conditions for the current practice forecasts to be rational. For simplicity, the theorems and proofs consider a scalar explanatory variable. They condition on a fixed information set, omit that information set and the time subscripts from the notation, and denote $\mu \equiv E\left(x\right)$. We assume that $\mu$ and $E\left[r\left(x\right)\right]$ are finite.

Theorem 2. The rational forecast will equal the current practice forecast if:

1. the response function r is affine, that is, $r\left(x\right)=a+bx$ for constants a and b; or
2. the response function r has 180 degree rotational symmetry about the point $\left(\mu ,r\left(\mu \right)\right)$, and the conditional pdf p is symmetric about its mean, $p\left(\mu -z\right)=p\left(\mu +z\right)$.

Proof. See Appendix C.

To give intuition for the theorem, the rational forecast of y considers the possibility that x could be above its mean and that x could be below its mean. Under the assumptions of this theorem, these two considerations have exactly offsetting effects, so the rational forecast coincides with the current practice forecast, which assumes that x will equal its mean. Consider case (b) where r satisfies the rotational symmetry assumption. As illustrated by Figure 16, if the explanatory variable is a distance u above its mean, the response function would change by $\left(r\left(\mu +u\right)-r\left(\mu \right)\right)$. If the explanatory variable were the same distance below its mean, the response function would change by $\left(r\left(\mu -u\right)-r\left(\mu \right)\right)$. The rotational symmetry assumption states that these two changes are the same in size and opposite in sign.

Since the future explanatory variable has a symmetric distribution about its mean, it is just as likely to be above its mean by u units as below its mean by the same amount. Hence, the rational forecast equals the current practice forecast.

If either of these symmetry conditions fail, then the current practice forecast may differ from the rational forecast, as we now show.

Theorem 3. Suppose the future explanatory variable has a non-degenerate distribution.

• If the response function r attains a strict global minimum (maximum) at the rational forecast for the explanatory variable, $\mu$, then the rational forecast will be strictly greater than (strictly less than) the current practice forecast.
• If the response function r is strictly convex (strictly concave), then the rational forecast will be strictly greater than (strictly less than) the current practice forecast.

Proof. See Appendix C.

To illustrate, consider the dependence of real GDP growth on rainfall, which may be important in low-income economies where agriculture is a large share of GDP. Typically, the kinds of agriculture undertaken in a given location are those that provide high yields when rainfall is typical for that location. For example, rice in areas with high rainfall and wheat in areas with lower rainfall. Hence it may be approximately correct to assume that real GDP growth in a low-income economy attains a strict global maximum at the average level of rainfall, or that growth is strictly concave in rainfall. If either of these assumptions is true, Theorem 3 implies that the rational forecast will be lower than the current practice forecast.

Case with irrational forecasts for explanatory variables

The preceding discussion assumed that the assumed future values of the explanatory variables are rational forecasts for those variables. This is likely to be approximately true when those assumptions are based on financial markets, as when crude oil price assumptions are taken from crude oil price futures. However, when assumptions are made in other ways they may be far from rational forecasts.

Assumptions about future fiscal policy will often differ greatly from rational forecasts. Where the IMF or EC are providing financing to an economy, they assume that the economy will fully implement any policies it agreed to as a condition of the financing (Genberg et al 2014). It is unlikely that expecting full compliance is a rational forecast, given that economies often comply only in part. For other economies, the IMF and EC approach:

assumes that policy changes will occur during the forecast period only if they have been legislated in advance or if there is substantial knowledge that suggests they will occur. Otherwise, policy is assumed to remain constant during the forecast period. (Genberg et al 2014, pp 35–36)

This is also unlikely to be a rational forecast, as it ignores the systematic tendency of governments of advanced economies to vary their fiscal policy to smooth out business cycle fluctuations.

6.2 Modal forecasts

6.2.1 Which forecasters make modal forecasts?

For a given conditional distribution of future growth, a forecaster could report the mean, the mode, the median, or some other summary statistic. Surveys of professional forecasters typically do not specify the type of point forecasts respondents should provide. For example, Elliott et al (2008) notes that the US Survey of Professional Forecasters does not specify the type of point forecasts that respondents should provide. Similarly, international organisations refer to their forecasts using vague terms such as ‘central forecast’.

It's likely that at least some international organisations see themselves as making modal forecasts. One piece of evidence is that some major central banks make modal forecasts.^[24] This suggests international organisations may do the same, given that they use similar forecasting models, are staffed by economists with similar training, and follow similar internal processes during forecasting rounds.

Some private-sector forecasters may also see themselves as making modal forecasts. This view was held, for instance, by JP Morgan (Davies 2017).

6.2.2 How would modal forecasts affect rationality tests?

Consider a forecaster who is able to compute the conditional distribution of growth given the information available on the forecast date.

• The forecaster could report the conditional expectation of growth, $y_{t+h|t}=E\left[y_{t+h}|I_t\right]$. In this case, their forecasts would satisfy rationality tests. That is, their forecast errors $v_{t+h|t}$ would be orthogonal to any vector of known variables x_t (see Equation (1)).
• If the conditional distribution is unimodal, the forecaster could instead report the conditional mode of growth, $y_{t+h|t}=\text{Mode}\left[y_{t+h}|I_t\right]$. If the conditional distribution is also symmetric, then the mode would coincide with the mean, and these forecasts would satisfy rationality tests. However, if the conditional distribution is asymmetric, the mode could differ from the mean, and the forecasts may not satisfy rationality tests.

Modal forecasts may fail rationality tests due to bias (as recognised in Kangur et al (2019)). However, modal forecasts may also fail rationality tests due to correlations of forecast errors with other explanatory variables (which has not been recognised previously). Biased forecasts will cause the rejection of rationality tests whenever the vector x_t includes a constant, because the bias would lead to the coefficient on that constant being non-zero. As shown in Bekaert and Popov (2019) and Burgess et al (2021), past growth outcomes have been negatively skewed in most economies. If the conditional distribution of future growth given the forecaster's information set is also negatively skewed, then the mode will often exceed the mean, in which case modal growth forecasts will have an optimistic bias.

Modal forecasts may result in correlations between forecast errors and other explanatory variables. For example, suppose that growth is a realisation from one of two non-overlapping distributions: a ‘crisis’ distribution centred on –5 per cent, or a ‘non-crisis’ distribution centred on 2 per cent. Let the probability of a crisis next year be low. Assume that the conditional mode remains the mode of the non-crisis distribution even if the crisis probability varies. The mean will then depend on both distributions, while the mode will depend only on the non-crisis distribution. Suppose some variable in x_t, such as the debt-to-GDP ratio, is positively correlated with the probability of a crisis next year. Then that conditional expectation of growth will be a decreasing function of that variable, while the conditional mode of growth will be unrelated to that variable. This implies that, if the forecaster makes a modal forecast, their resulting forecast errors will be negatively related with that variable, causing the failure of rationality tests.

6.2.3 Empirical evidence on whether making modal forecasts introduces bias

We have just argued that the practice of making modal forecasts could result in biased forecasts. We now show empirically that modal growth outcomes tend to exceed mean growth outcomes, which suggests that this bias might arise in practice, but the effect is small. We also raise a methodological limitation that affects both our method and the similar method of Burgess et al (2021). Due to this limitation, it remains possible that making modal forecasts contributes greatly to bias or not at all.

Conceptual overview of method

Suppose that for any given economy and forecast horizon, the conditional mode for a period t equals the sum of a constant $\mu$, the conditional mean $E\left[y_t|I_{t-h}\right]$, and a mean-zero error e_t.

The constant $\mu$ is the expected difference between conditional mode and mean. This parameter determines whether a hypothetical forecaster that knew the conditional distribution of future growth would make biased forecasts if they reported the conditional mode rather than the conditional mean. Our aim is to estimate this parameter, so our empirical strategy is focused on growth outcomes (which are relevant to the hypothetical forecaster) rather than on the growth forecasts of any particular organisation.

Ideally, we would compute the mean and mode of the conditional distribution of future growth for a sample of periods. Using this sample, we could estimate Equation (8), giving us an estimate of the amount of bias that modal forecasting causes for that economy, $\widehat{\mu }$. We could also test the hypothesis that modal forecasting does not introduce bias, $\mu =0$, against the alternative that it does.

Unfortunately, we do not observe the conditional distribution of growth given particular information sets. Rather, we observe a set of growth outcomes, which provides information about the unconditional distribution of growth. Hence, we need to use our model of conditional distributions (Equation (8)) to make a prediction about unconditional distributions. We do this by taking expectations of Equation (8) over possible information sets.^[25]

We then make the further assumption that the unconditional mode, Mode[y_t], equals the expectation of the conditional modes, $E\left[\text{Mode}\left[y_t|I_{t-h}\right]\right]$. This assumption is a limitation of the method, as discussed below. With this assumption:

Given a sample of growth outcomes, we would like to estimate the unconditional mode and the unconditional mean, as the difference between them would provide an estimate of $\mu$. A challenge is that an economy's growth may be a non-stationary process, so its mean and mode may vary over time. For example, some east Asian economies such as Japan experienced high ‘catch-up’ growth followed by more moderate growth. To allow for this, we assume that an economy's growth is the sum of a deterministic component, b_t, and a stationary stochastic component, z_t.

Substituting this into Equation (9), and recognising that deterministic components can be taken outside of the mode or expectation operator, we have:

Hence, to estimate how much making modal forecasts introduces bias, $\mu$, we compute the estimated mode of detrended growth less the sample mean of detrended growth. This is somewhat similar to Burgess et al (2021), who compare the sample mean of detrended growth to the sample median of detrended growth.

Implementation and results

We use annual real GDP growth data from the WEO database for all economies whose time series has at least 40 observations. To estimate the stochastic component, we convert the growth rates into log levels, compute detrended log levels, and then compute growth rates of the detrended log levels. We detrend log levels using a two-sided Hodrick-Prescott filter (Hodrick and Prescott 1997), with the parameter value $\lambda =6.25$, following the recommendation for annual data given by Ravn and Uhlig (2002). Figure 17 compares actual growth rates, y_t, and our estimates of the stationary stochastic component, z_t, for Japan.

We estimate the mode of detrended growth using the half-sample mode estimator as implemented by the ‘modeest’ R package (Poncet 2019). This estimator assumes that the data are drawn from a continuous unimodal distribution, but does not require the assumption of any specific parametric distribution, and has the advantage of being insensitive to outliers (Bickel and Frühwirth 2006). We estimate the mean of detrended growth using the sample mean.

We find that 91 of 145 economies have an estimated mode above the sample mean. These differences are small in magnitude however. On average across all 145 economies, the estimated $\widehat{\mu }$ is 0.16 percentage points.

As a robustness check, we also compute the sample skewness of detrended growth for each economy:

Skewness is important because if the distribution of the detrended growth is unimodal, then its mode will typically exceed its mean when the distribution is negatively skewed. We find that 107 of 145 economies have negative sample skewness. This supports our finding that the unconditional mode exceeds the unconditional mean in most economies. Without this complementary evidence, one might have wondered if our results were driven by the specifics of our mode estimator.

Limitations

Our method assumes that the unconditional mode, Mode [y_t], equals the expectation of the conditional mode, $E\left[\text{Mode}\left[y_t|I_{t-h}\right]\right]$. This assumption holds exactly if the conditional mode does not vary with respect to variables in the information set. This is possible in principle. Returning to the example of a crisis introduced above, suppose that a crisis is very unlikely when the debt-to-GDP ratio (the variable entering the information set) is low and that a crisis is somewhat unlikely when the debt-to-GDP ratio is high. For both values of the debt-to-GDP ratio, the crisis will probably not occur, and the conditional mode of growth does not vary with respect to the variable in the information set.

It is more likely, however, that the conditional mode of growth does vary with respect to the information set. In this case, the assumption may or may not hold. We will illustrate this in the extreme case where the information sets fully determine growth. Suppose that there are three possible information sets: ‘pre-bust’, ‘ordinary’ and ‘pre-boom’, which guarantee that growth will be –2, 2 or 6 per cent respectively. First we consider the symmetric case, where ‘pre-bust’ and ‘pre-boom’ each occur with probability $\frac{1}{4}$ and ‘ordinary’ occurs with probability $\frac{1}{2}$. In this case, the assumption holds:

In practice, business cycles are likely to be asymmetric. Suppose that the probability of ‘pre-bust’ is $\frac{1}{4}$, of ‘ordinary’ is $\frac{3}{4}$ and of ‘pre-boom’ is 0. Then:

As this discussion illustrates, unconditional modes may differ from the expected conditional mode in the face of asymmetric business cycles, which is a situation we are interested in. Hence, one should be cautious about the results of our method (which suggest that modal forecasts may contribute slight optimistic bias), but also about any other effort to explain forecast bias using the unconditional distribution of growth. In particular, comparing the sample median and sample mean of growth as in Burgess, Langendorf, Ippolito and Pielke (2021) provides an unbiased estimate of the difference between the conditional median and conditional mean only if one assumes that $E\left[\text{Median}\left[y_t|I_{t-h}\right]\right]=\text{Median}\left[y_t\right]$, which is subject to an analogous critique.

This paper provides new evidence and theory concerning real GDP forecasts by international organisations. The paper begins by comparing the forecast accuracy of four international organisations and the average of Consensus forecasts, finding statistically significant differences for only a few economies. It presents a new decomposition of differences in accuracy into contributions of disagreement and the accuracy of the average forecast. This decomposition is then used to show that, on average across economies, differences in accuracy are larger at longer forecast horizons due to both greater disagreement and lower accuracy of the average forecast. We then turn to forecast rationality. For most forecasters, we reject the null of the rationality test for most economies. Often the irrationality takes the form of optimistic bias, overly large revisions or forecasts that are too extreme. Finally, we contribute to the literature that attempts to explain why rationality is often rejected in these tests. We focus on two explanations: the use of conditioning assumptions and the practice of making modal forecasts.

Our findings have important implications for the practices of international organisations. Firstly, we find that the international organisations are quite similar to one another in terms of accuracy and the ways in which they fail rationality tests. Secondly, our findings highlight the importance of forecasters being transparent, including about their conditioning assumptions and whether they see themselves as making modal forecasts. This will assist users in interpreting the forecasts. Knowing that a forecaster is making modal forecasts, for instance, will tell users that they should not view the forecasts as incorporating tail risks like financial crises. Transparency will also make it easier to choose among competing explanations for the failure of rationality tests, which will help organisations determine what actions to take in response.

The lessons of this paper can also be applied to other types of forecasters. For example, one can ask whether differences in the accuracy of different private-sector forecasters are larger at longer horizons, and if so, use our decomposition to assess if this reflects greater disagreement or lower accuracy. Or one can investigate the effects of making conditioning assumptions or modal forecasts at central banks or fiscal authorities.

A.1 Computing forecast horizons

We define the horizon of a forecast as the number of days from the publication date of the forecast to the end date of the target year. The publication date of a forecast is assumed to be the 15th day of the publication month.^[26] International organisations and Consensus forecast real GDP growth in calendar years for most economies, but forecast growth in fiscal years in others (see Table A1 for a list). For example, WEO forecasts US real GDP growth in calendar years, so in the October 2019 WEO, the end date of the first target year will be 31 December 2019. However, WEO forecasts Indian real GDP growth in fiscal years, so in the October 2019 WEO, the end date of the first target year will be 31 March 2020.

When comparing forecasters or testing the rationality of forecasts, we report results for a set of forecasts with a specific horizon. In the terminology of Clements (1997), we perform ‘rolling event’ regressions rather than ‘fixed event’ regressions, because our regressions are estimated on samples of forecasts with a fixed horizon and a varying forecast date. However, a subtlety is that we round the forecast horizons to ensure we have adequate sample sizes. For example, when we test the rationality of two-year-ahead forecasts by EC, we estimate our regression on all EC forecasts whose horizon in days is between 548 days and 913 days. An alternative to pooling together forecasts with similar horizons is to construct a set of artificial fixed-event forecasts, an approach that has been critiqued by Yetman (2018).

A.2 Computing forecast errors

A.2.1 Source of actual outcomes

For WEO, EO and EC, we compute forecast errors by comparing the institution's forecast with their own actuals. The forecast databases for these institutions provide vintages comprising actual outcomes and forecasts. To distinguish between actuals and forecasts, we need to make an assumption about how long it takes for statistical agencies to publish actual outcomes after the end of the target year. Our assumption is that this takes 90 days. To illustrate, consider WEO vintages of real GDP growth in India. WEO reports Indian real GDP growth over fiscal years that end on 31 March of each year. Growth for fiscal year 2016/17 will be published on 29 June 2017, which is 90 days after the end of the fiscal year. As discussed above, we assume the publication date of a forecast is the 15th day of the publication month. Hence, the June 2017 WEO vintage is assumed to have been published on 15 June 2017, so we assume that the value reported by WEO for 2016/17 is a forecast rather than an actual outcome.

The GEP database also contains both forecasts and actuals, so they need to be distinguished as above. However, the data on actuals has too short a history to enable the calculation of forecast errors. Hence, we compute errors in GEP forecasts by comparing them with WEO actuals. One might be concerned that the forecasts for an economy may be recorded over a different year as the actuals, since they are taken from different sources. Unfortunately, there are some economies for which the GEP forecasts are recorded over different years to the WEO actuals, and we are unable to compute forecast errors in these economies (see Table A1 for the specific economies).

Consensus provides forecasts, but not actuals. Hence, we compute errors in Consensus forecasts by comparing them with WEO actuals. Fortunately, Consensus uses the same years as WEO for all economies where it makes forecasts.

A.2.2 Rationale for use of latest vintage of actual

For all forecasters, we compute errors using the latest vintage of the actual at the time of writing. An alternative approach would have been to take the first vintage of the actual. Revisions to real GDP data can be substantial (Genberg and Martinez 2014a), so this would lead to different results. Our choice of approach rests on two claims:

1. International organisations should aim to forecast the best estimate of true real GDP growth. True growth determines other economic variables and welfare, so a forecast for the best estimate of true growth is the best guide to policy.
2. Later vintages of actuals are better estimates of ‘true’ growth than earlier vintages. A major source of revisions to real GDP data is that the statistical agency receives additional data over time. For example, a statistical agency may produce the first vintage of a component of GDP based solely on survey data, but produce more accurate later vintages using tax data as well. This makes later vintages better estimates of true growth. Another source of revisions is changes to international and national standards for the measurement of GDP. Whether such revisions also make the data a better estimate of true growth is less clear. One view is that the changed standards better approximate some ideal notion of real GDP. Another view is that the later vintages are estimates of a slightly different variable to that which the forecaster had in mind.^[27] In our view, even if one is concerned about changing standards, this second claim would still hold because the later vintages of GDP benefit so much from additional data.

Taken together, these claims imply that international organisations should aim to forecast the latest vintage of the actual, which makes this latest vintage the proper standard by which the accuracy and rationality of their forecasts should be assessed.

A limitation of our approach is that it reduces the comparability of earlier and later errors, since forecasts from earlier vintages will be compared to actuals that have been revised many times, while forecasts from later vintages will be compared to actuals that have been revised only a few times. On balance, however, we judge that it is appropriate to compute errors using the latest vintage of the actuals.

B.1 Proof for Section 4.3

Proof of Theorem 1. The loss differential can be given the following multiplicative decomposition:

Hence the absolute loss differential can be decomposed as:

B.2 Estimation method for panel regressions in Section 4.3

We combine Equations (2) and (3) into a single equation by defining a new variable, $z_{t+h|h}^k$, where k is one of l (loss differential), e (error in average forecast) or d (disagreement)

Substituting in the constraints gives:

Rearranging gives the following equation. We estimate it with OLS.

Using the OLS estimates of ${\widehat{\alpha }}_1,{\widehat{\alpha }}_2,{\widehat{\beta }}_1,{\widehat{\beta }}_2$, we compute ${\widehat{\gamma }}_1={\widehat{\alpha }}_1-\log \left(2\right)-{\widehat{\beta }}_1$ and ${\widehat{\gamma }}_2={\widehat{\alpha }}_2-{\widehat{\beta }}_2$. To test a hypothesis about ${\gamma }_1$ or ${\gamma }_2$, we can perform a Wald test of the appropriate restriction on $\left({\alpha }_1,{\alpha }_2,{\beta }_1,{\beta }_2\right)$, using a HAC estimator of the covariance matrix of $\left({\alpha }_1,{\alpha }_2,{\beta }_1,{\beta }_2\right)$.

B.3 Full results of comparisons of accuracy