CGE Models vs Educated Guesswork: The Case of the EU-Korea FTA

A September 2016 research note by the European Commission on the effects of the EU-Korea Free Trade Agreement has recently been picked up in social media by various commentators.

Amid rising skepticism around trade agreements and a tendency to question established wisdom, its 2-fold message is balm for the economist’s soul: 1) Trade agreements are good and 2) CGE models do a great job predicting their effects. Comparing observed trade flows with ex-ante CGE predictions, the study concludes that:

“The present analysis adds to deliberations surrounding CGE modelling by examining the EU-Korea FTA and pointing to sound projections against observed data at the aggregate level and in the largest sectors.”

These results have found a thankful audience in trade policy circles, notably among those advocating for the ratification/conclusion of “new-generation” trade deals such as CETA, TTIP, or the TPP. A closer look at the data, however, reveals that both claims cannot be substantiated without further analysis.

Using the same database (COMEXT) as in the Commission’s study, figure 1  plots the observed trade flows against ex-ante CGE forecasts. The CGE 1 trajectory describes the baseline 1 predictions of the original CEPII/ATLASS impact assessment study from 2010, in which it was assumed that no further FTAs were to be signed by both economies, among other assumptions. This is the model against which the Commission’s study evaluates the observed data. CGE 2 describes the baseline 2 predictions, in which some trade diversion through other concluded trade agreements policies were assumed. The vertical line marks the time of entry into force of the EU-Korea FTA.  The CGE 1 trajectory is steeper, due to the lower assumed trade diversion effects, compared with trajectory CGE 2.

Clearly, the CGE forecasts do not match the observed outcome. Both projections significantly underestimate the actual observed flows. Nevertheless, the Commission’s study claims that the impact assessment study offered “sound projections”, while concluding that

“the EU-Korea FTA generated greater-than-expected EU export growth.”

In other words, the Commission reads figure 1 as: The CGE projections are good, but the FTA has been even better.

But how good were these predictions really? In order to get a better grasp on the accuracy of the CGE projections and the impact of the EU-Korea FTA, we look at what trajectory EU exports to Korea had taken before the FTA. While the past may not predict the future, the historical trajectory can offer a rudimentary means of comparison and should always be included in such type of analysis.  So let us “zoom out” of figure 1 to enlarge the time horizon. Figure 2 does the job and puts figure 1 in historical context. Having the broader picture, both the effect of the FTA and the predictions of its effects based on CGE modeling appear in a different light: The observed post-FTA data simply follow their pre-agreement path. In contrast, the CGE forecasts predicted a weakening of export growth.

Would the projections have been better had they been based on different models? The authors of the 2010 CEPII/ATLASS impact assessment study undertook their study with 2004 GTAP data. Putting ourselves into their shoes and acknowledging the data constraints they were facing, we generate an alternative prediction using a much simpler method.

We predict export volumes on the basis of a simple linear regression on the time trend of exports before the onset of the financial crisis (dotted vertical red line in figure 3), which we define as September 2008, when Lehman Brothers filed for bankruptcy. The ensuing economic downturn was largely exogenous to any economic forecasting at the time.  We chose 2004 as the starting date, as the EU has undergone its most significant expansion at that time. Our predictions are represented by the orange dashed fitted line in figure 3. We observe a good fit with the actual data after the entry into force of the EU-Korea FTA (right of the solid vertical line) – a fit that is also more accurate than either prediction of the CGE exercise (whose trajectories are displayed in the same colors as in the previous graphs).

Figure 4 provides an overview of the accuracy of the different forecasts for each calendar year. The dashed horizontal line indicates a perfect fit, where observed trade volumes exactly match the prediction. The green bar reflects our forecasts based on pre-financial crisis data, the yellow and red bars display the results of the CGE 1 and CGE 2 models. The Commission concludes that the CGE predictions were good, and the FTA even better. The quite constant export growth rates pre- and post-FTA, as well as the sizable error margin of CGE projections amounting to over 20%, remain unaccounted for. In contrast, a simple linear extrapolation of historical export data would have predicted observed exports to Korea in 2015 with half as much of an error (10%).

The insight that EU exports to Korea have persisted at their historical growth rate does not let us conclude that the EU-Korea FTA had no effect.  In the absence of a sensible counterfactual, we cannot reasonably make such a claim. Conversely, however, noting that their growth rate has outpaced unreasonably low CGE projections does not allow the inverse conclusion that it has been a success. The fact that a simple linear extrapolation of historical trade flows would have provided a more accurate result than the CGE-based forecasts should make us pause when applying such models to future agreements. In times of rising anti-globalization sentiment and looming protectionist battlegrounds, the stakes are too high to jump to conclusions.