RDP 2013-07: An Empirical BVAR-DSGE Model of the Australian Economy 5. Results

The forecasting results for the BVECMX model and the BVARX model that excludes the error-correction terms are shown in Table 4. The results suggest that including the error-correction terms improves the accuracy of the investment forecasts at all horizons, but leads to some deterioration in the export forecasts. This is not surprising because, in the data, exports and output do not appear to be cointegrated (at least with the parameters implied by the model; for further detail see Dungey and Pagan (2009)).

Table 5 shows the performance of the BVECMX relative to our benchmark models, namely the SOE Minnesota VAR, the univariate autoregressions, the DSGE model and the DSGE and Minnesota forecasts combined with equal weights.^[28]

BVARs with Minnesota priors have been shown to forecast well (for example Litterman (1986)). In this case, for several variables, such as growth in exports or output, the SOE Minnesota VAR forecasts more accurately than the BVECMX. The relative accuracy of forecasts of growth in exports appears to be related to the inclusion of the error-correction terms in the BVECMX.

These results imply that the SOE Minnesota VAR outperforms the DSGE model at long horizons, except for interest rates and investment. For inflation, it is only when the structure of the model is relaxed to some extent, as in the BVECMX, that the forecasting performance becomes more comparable to that of the statistical models. There are several reasons why this may be the case. First, it could be particular to the DSGE model we have used. While we have attempted to tailor the baseline small open economy model to Australia to some extent, much could be done to further enrich it. Second, the fit and/or forecasting performance of DSGE models might generally be better for relatively closed economies, such as the United States, and it could be intrinsically more difficult to forecast small open economies or commodity exporters.

Another aspect to note is that the performance of the SOE Minnesota prior is very similar to that of the univariate autoregressive models. For some variables the autoregressive (AR) models themselves are probably similar to simply assuming the forecast is equal to the average growth rate; for example, over the inflation-targeting period there is little persistence evident in quarterly output growth in Australia, and consequently the autoregressive coefficients are insignificant.

The performance of the forecasts formed by equally-weighting the DSGE and Minnesota VAR forecasts is broadly equivalent to that of the BVECMX. Computationally, this is a much simpler way of combining information from the DSGE and the VAR.

Another metric for evaluating the forecasts is their bias, i.e. do they over- or under-predict on average? This test is shown in Table 6. The forecasts for growth in exports and output are biased. Over this period, there was an investment boom in the mining sector, but there was not a commensurate increase in export growth. Consequently, productivity in the mining sector declined and the model forecasts were for stronger export growth than eventuated.

Table 6: Bias – BVECMX Model

Includes error-correction terms

Variable	Quarterly	Year-ended
Δ Exports	1.21*	4.03*
Δ Investment	1.11	3.47
Growth	0.38*	1.23*
Inflation	−0.09	−0.55*
Interest rates	0.02	0.34
Δ Real exchange rate	−1.05	−3.90
Notes: * denotes significance at the 5 per cent level; forecast error is defined as the forecast minus the actual value; heteroskedasticity and autocorrelation consistent (HAC) standard errors for the year-ended forecasts were used; year-ahead interest rate forecasts are for the level

The rapid growth in investment was a response to a terms of trade boom from around 2003 onwards, and forecasters, including the RBA, under-predicted the extent of growth in the terms of trade (see Kearns and Lowe (2011)). Given these developments, and the fact that movements in the terms of trade and the real exchange rate are typically quite correlated, it is somewhat surprising that the bias in the forecasts for the change in the real exchange rate, while in the expected direction, is not significantly different from zero.

Output growth at both horizons is over-predicted on average. The demeaning of the data probably contributes to this bias. At the beginning of the sample there was strong growth as the economy emerged from the early 1990s recession. Also, since at least 2005, Australia's productivity performance has been relatively weak compared to the growth achieved over the 1990s.^[29] As a result, the growth rate to which our forecasts are assumed to revert may be too high, even though we have allowed economy-wide technology to be non-stationary. Of course it is difficult to know whether the apparent change in the average rate of output growth will be permanent, especially given the considerable structural change occurring in the economy due to the terms of trade boom.

The bias in the inflation forecasts in the near-term is slight, but it is larger one-year-ahead. One period contributing to this result is immediately prior to the global financial crisis, when inflation in Australia increased noticeably. The BVECMX model initially fails to predict this increase, and its one-quarter-ahead inflation forecasts are progressively revised upwards throughout the period. The SOE Minnesota model also does not predict the initial increase. One possible reason for the sharp increase in inflation is that there might be a threshold level of capacity utilisation above which inflation increases strongly (Lowe 2011). As all of the models we have considered are linear, none can capture such dynamics.

5.1 Sensitivity Analysis

5.1.1 Prior sensitivity

The forecasting performance is relatively insensitive to the looseness of the prior, governed by the parameter λ (Table 7). For some variables, such as the interest rate and inflation, the forecasts made using different values of λ are highly correlated. This may in part reflect the strong persistence in these variables. The variations in the RMSEs that do occur as λ changes are not consistent across all of the variables or horizons; for example, for output growth one-quarter-ahead, tighter priors outperform looser priors, whereas one-year-ahead a higher λ performs better. For inflation, when the prior is tightly held (λ = 1) the forecasts are most accurate, although the impact of changing λ is small. Accordingly, focusing on the results for output and inflation, it is not clear what λ to choose; in the near-term a low λ is appropriate, whereas for longer horizons a higher value might be desirable. Alternatively, if λ was chosen with reference to the forecasting performance of all variables then λ = 1 would be selected.

5.1.2 The Calvo parameter

As previously discussed, the estimated Calvo parameter for domestically produced goods is very high in this model. To examine the implications of this we re-estimated the model, calibrating the parameter at the mean of the prior previously used in estimation (0.75). The results, which are shown in Table 8, suggest that the main impact is that the near-term inflation forecasts become less accurate if the prior is tightly held (i.e. λ = 1). This suggests a looser prior may be appropriate.

5.1.3 Excluding the exports error-correction term

In the results presented in Table 4 it appears that including both error-correction terms leads to a deterioration in the performance of the forecasts for export growth, but improves those for investment growth. Re-estimating the model excluding the exports cointegrating term improves the export forecasts, but has little impact on the other variables (Table 9).

5.1.4 Extending the estimation period

The estimation sample used was the inflation-targeting period (1993:Q1 onwards). Hence, the models for the early samples in the forecasting exercise were estimated with only a small amount of data, which may make the use of a relatively tight prior for the VAR preferable. To test the importance of sample size, we re-estimate the BVECMX over a longer sample (1986:Q2 onwards). To do this we regress all variables against a constant and a dummy for the inflation-targeting period, rather than demeaning the data, which is akin to allowing the intercept to shift for all equations in the VAR in 1993:Q1. Given that other countries, such as the United Kingdom and Canada, also deflated at around this time and are included in the aggregates we use for the foreign variables, we also allow for breaks in these series. The DSGE parameters obtained are similar to the short sample estimates, and are given in Table D1.

In general, the BVECMX estimated over the longer sample tends to forecast more accurately; for example, the ratio of its RMSE for the one-quarter-ahead inflation forecasts to that for the original sample is 0.85 (when λ = 1). However, the accuracy of the AR models, for example, also increases when estimated over the longer sample (and adjusting for possible breaks in the same way). Overall, it appears that in the short to medium term the BVECMX model forecasts inflation more accurately than an AR model when the long sample is used. In contrast, output growth is forecast less accurately by the BVECMX, but the relative performance of the BVECMX model generally improves (Table 10). It is also noticeable that a looser prior improves the accuracy of the output growth forecasts, but causes both the inflation and, particularly, the interest rate forecasts to deteriorate.

5.2 The Global Financial Crisis

The major economic event during our forecast evaluation period is the global financial crisis. Consequently, it is interesting to consider the performance of the BVECMX around this time. Examining the forecasts, it is apparent that the early forecasts for the crisis period were for around trend growth. Throughout the first half of 2008, the mean one-quarter-ahead forecasts were generally revised lower, reflecting the soft growth outcomes which occurred. The BVECMX does not forecast the sharp drop in output that occurred in the December quarter of 2008. The SOE Minnesota VAR similarly predicted moderate growth during this period.

One possible explanation as to why the mean forecasts from these models missed the extent of the impact of the global financial crisis is that they do not contain enough financial variables, for example, credit aggregates and spreads are not included in the model. An obvious shortfall of the BVAR-DSGE methodology is that it is necessary to have a DSGE model that incorporates these variables. Developing this aspect of these models is currently a very active area of research.^[30] The Minnesota VAR we have used to generate the large economy forecasts is also very simple and could be expanded or generalised. Edge and Gürkaynak (2010) review the forecasting performance for the United States economy of the Smets and Wouters (2007) DSGE model, a Minnesota VAR in the observable variables of the DSGE and Blue Chip (a monthly survey of business economists). They found that ‘[a]lthough all the forecasts clearly first miss the [United States'] recession, and then miss its severity, the Blue Chip forecasts in general fare better as the quarter to be forecasted gets closer, and especially when nowcasting’ (p 235). This improved performance is likely to be due to the fact that the analysts included in the Blue Chip forecasts can utilise additional timely information in constructing their forecasts, which are typically not observed or even included in these models.

A second explanation of why the BVECMX model's mean output growth forecasts fail to capture the crisis is that they are not very persistent, and quickly revert to the average growth rate. While this lack of persistence is a characteristic of the data, it makes it difficult for the model to explain fluctuations in growth.^[31] None of the models that were considered forecast output growth well over this period.

Footnotes

A comparable table for the model excluding the error-correction terms is Table D1. [28]

For one discussion of trends in Australian productivity over the 2000s see Eslake (2011). [29]

The financial accelerator model links credit spreads to the net worth of the borrower, and consequently potentially amplifies the impact of shocks (see, for example, Bernanke, Gertler and Gilchrist (1999)). However, Pagan and Robinson (2012) review the ability of several models including financial frictions to predict recessions and find that it is poor. [30]

In the pre-sample period for the SOE Minnesota VAR, the first-order autocorrelation coefficient was approximately 0.3. However, over the estimation period this drops to 0.1 and is no longer statistically significantly different from zero. Tulip (2009) finds that for the United States, using Greenbook forecasts, during the Great Moderation period ‘… the predictable component of fluctuations in output and inflation has virtually disappeared’ (p 1217). [31]