CHAPTER 5 CTA Performance Evaluation with Data Envelopment Analysis Gwenevere Darling, Kankana Mukherjee, and Kathryn Wilkens W e apply data envelopment analysis to a performance evaluation frame- work for CTAs. The technique allows us to integrate several perform- ance measures into one efficiency score by establishing a multidimensional efficient frontier. Two dimensions of the frontier are consistent with the standard Markowitz mean-variance framework, while additional risk and return dimensions include skewness and kurtosis. We also illustrate a method of analyzing determinants of efficiency scores. Tobit regressions of efficiency scores on equity betas, beta-squared, fund size, length of manager track record, investment style (market focus), and strategy (discretionary vs. systematic) are performed for CTA returns over two time frames represent- ing different market environments. We find that the efficiency scores are negatively related to beta-squared in both time periods. Results also indi- cate that emerging CTAs (those with shorter manager track records) tend to have better efficiency scores as defined by the DEA model used in our study. This relationship is strongest during the period from 1998 to 2000, but not statistically significant during the period from 2000 to 2002. For both time periods, fund size is not related to efficiency scores. INTRODUCTION Industry performance reports for commodity trading advisors (CTAs) present multiple performance measures such as return, standard deviation, drawdowns, betas, and alphas. Investors and fund managers recognize the importance of considering a multitude of performance measures to analyze fund risk from various perspectives. It is particularly important for the growing alternative investment class of managed futures, which have dif- 79 c05_gregoriou.qxd 7/27/04 11:07 AM Page 79 ferent risk/return profiles from those of traditional mutual funds as well as those of many hedge fund strategies. For all asset classes, however, the aca- demic literature has done little to offer a comprehensive framework that incorporates multiple risk measures in an integrated fashion (Arnott 2003). Too often, studies focus on single measure of risks, arguing for one relative to another. “Managed futures” are a subset of hedge funds that uses futures con- tracts as one among several types of trading instruments (including swaps and interbank foreign exchange markets) and for which futures are a means, rather than an end, with which to implement their strategy. The name wrongly suggests that futures are the dog rather than the tail. Man- aged futures encompass the broad set of individual commodity trading advisors (CTAs). CTAs are also unfortunately named because, on balance, most of their trading is in the financial markets, not the commodity mar- kets. Like any other class of alternative investments, managers are repre- sented by a variety of styles and substyles. For example, there are systematic and discretionary CTAs, CTAs who exclusively try to capture trends, those who identify countertrend opportunities, and those who combine the two approaches. 1 In this study we look at the performance of CTAs based on multiple criteria using data envelopment analysis (DEA). DEA establishes a multidimensional efficient frontier and assigns each CTA an efficiency score whereby 1 (or 100 percent) indicates perfect efficiency and scores lower than 1 represent rela- tively less efficient CTAs based on the performance criteria chosen. The criteria we choose as bases for performance evaluation are monthly returns, kurtosis, minimum return, skewness, standard deviation of returns, and percentage of negative monthly returns. Although there are many other possibly appropriate criteria, those not included here are likely either to be redundant with variables included or to not make sense in an optimization framework. Criteria that make sense in this framework are those that are desirable to maximize or minimize across various market conditions. This aspect leads us to reject equity betas as a criterion in the DEA model, for example, because CTAs may desire a higher beta in up-market environ- ments but negative betas in down-market periods. In addition to applying the DEA methodology to evaluate CTA per- formance, we explore the relationship between the efficiency scores and fund size, investment style and strategy, length of the manager’s track 80 PERFORMANCE 1 Another important dimension of styles is the time frame. There are long-term, short-term, and medium-term traders and those who combine time frames. c05_gregoriou.qxd 7/27/04 11:07 AM Page 80 record, and measures of the covariance of CTA returns with equity market returns. We ask: ■ Do emerging hedge fund managers 2 really do better than larger, estab- lished managers? ■ Is there a relationship between efficiency scores and equity markets, and if so, does the market environment impact the relationship? ■ Do strategies (systematic, discretionary, trend-based) or styles (diversi- fied, financial, currency, etc.) matter in different market environments? We analyze monthly CTA returns in two different market environ- ments: over 24 months beginning in 1998, when equity market returns are predominantly positive, and over 24 months beginning in 2000, when they are more often negative. We find that emerging managers perform better than well-established managers in the sense that funds with shorter track records have a greater efficiency score. Fund size and manager tenure are weakly positively correlated. In contrast with the conventional wisdom, however, larger funds have better efficiency scores. These results provide some insight into capacity issues concerning optimal fund size. The fund size and manager tenure coefficients are, however, statistically significant only during the first (1998–2000) time period, indicating that capacity issues may be less important during flat equity markets. For both time periods, squared equity beta is inversely related to the efficiency scores and the coefficient is highly significant. This result appears to be influenced by the risk-minimizing design of our DEA model. The style dummy variable (diversified versus nondiversified) was not a significant fac- tor impacting efficiency scores. The systematic strategy variable was signif- icant, but only during the second (2000–2002) down-market period. We consider these results as preliminary because several issues may be affecting their significance. Notably, when our sample size is broken down by invest- ment style and strategy, the number of CTAs representing each group is very small. Nevertheless, we believe that the approach is a promising avenue for further research. The next section of this chapter provides a background discussion on var- ious risk measures and performance evaluation issues. The variables chosen as inputs to the DEA model and the regression model are then discussed in the context of prior research, and the data are described. The variable descrip- CTA Performance Evaluation with Data Envelopment Analysis 81 2 We consider managers with short track records to be emerging CTAs. This cate- gory is distinctly different from managers who invest in emerging markets. c05_gregoriou.qxd 7/27/04 11:07 AM Page 81 tion is followed by an explanation of the DEA methodology and Tobit regres- sions used to explore determinants of the efficiency scores obtained from the DEA model. Results are presented and the final section concludes. RISK MEASURES AND PERFORMANCE EVALUATION A multitude of investment fund performance models and metrics exist in part because some measures are more appropriate for certain purposes than others. For example, the Sharpe ratio is arguably more appropriate when analyzing an entire portfolio, while the Treynor ratio is appropriate when evaluating a security or investment that is part of a larger portfolio. 3 The multitude of per- formance measures and approaches also suggests that more than one meas- ure of risk may be needed to accurately assess performance. Conversely, some measures can be redundant. For example, Daglioglu and Gupta (2003b) find that returns of hedge fund portfolios constructed on the basis of some risk measures are often highly correlated, and sometimes perfectly correlated, with returns of portfolios constructed on the basis of others. Burghart, Dun- can, and Liu (2003) illustrate that the theoretical distribution of drawdowns can be replicated with a high degree of accuracy given only a manager’s aver- age return, standard deviation of returns, and length of track record. In this section we begin by briefly reviewing some of the traditional portfolio performance measures and analysis techniques. We review single parameter risk measures based on modern portfolio theory, we discuss expanded performance models that account for time-varying risk, discuss concerns over assuming mean-variance sufficiency, and consider multifactor models of style and performance attribution. This short review exposes a plethora of performance measures. The question of appropriateness and redundancy is revisited in the section that describes the data used in this study. The current section also discusses the seemingly paradoxical issue of using benchmarks to evaluate absolute return strategies 4 and concludes with a discussion of potential determinants of performance. Alpha and Benchmarks Traditional asset managers seek to outperform a benchmark, and their per- formance is measured relative to that benchmark in terms of an alpha. 82 PERFORMANCE 3 The Sharpe measure is appropriate when analyzing an entire portfolio, because the standard deviation, or total risk, is in the denominator whereas beta is the denomi- nator of the Treynor measure, and beta measures the systematic risk that will con- tribute to the risk of a well-diversified portfolio. 4 Absolute return strategies seek to make positive returns in all market conditions. In contrast, relative return strategies seek only to outperform a benchmark. c05_gregoriou.qxd 7/27/04 11:07 AM Page 82 While CTAs follow absolute return strategies that seek to make positive returns in all market conditions, benchmarks now exist for CTAs and other hedge fund strategies. Before considering benchmarks for absolute return strategies, we first review the concepts in the context of traditional asset management. Jensen’s (1968) alpha is generally a capital asset pricing model (CAPM)-based performance measure of an asset’s average return in excess of that predicted by the CAPM, given its systematic risk (beta) 5 and the market (benchmark) return. Alphas also may be measured relative to addi- tional sources of risk in multi-index models. Whereas various single-index models are based on the CAPM and assume that security returns are a function of their co-movements 6 with the market portfolio, multi-index (or multifactor) models assume that returns are also a function of additional influences. 7 For example, Chen, Roll, and Ross (1986) develop a model where returns are a function of factors related to cash flows and discount rates such a gross national product and infla- tion. The purposes of multi-index models are varied and, in addition to performance attribution, include forming expectations about returns and identifying sources of returns. Sharpe (1992) decomposes stock portfolio returns into several “style” factors (more narrowly defined asset classes such as growth and income stocks, value stocks, high-yield bonds) and shows that the portfolio’s mix accounts for up to 98 percent of portfolio returns. Similarly, Brinson, Singer, and Beebower (1991) show that rather than selectivity or market timing abilities, it is the portfolio mix (allocation to stocks, bonds, and cash) that determines over 90 percent of portfolio returns. However, Brown and Goetzmann (1995) identify a tendency for fund returns to be correlated across managers, suggesting performance is due to common strategies that are not captured in style analysis. Schneeweis and Spurgin (1998) use various published indexes (Gold- man Sachs Commodity Index, the Standard & Poor’s 500 stock index, the CTA Performance Evaluation with Data Envelopment Analysis 83 5 Within the Markowitz (1952) framework, total risk is quantified by the standard deviation of returns. Tobin (1958) extended the Markowitz efficient frontier by adding the risk-free asset, resulting in the capital market line (CML) and paving the way for the development of the capital asset pricing model, developed by Sharpe (1964), Lintner (1965), and Mossin (1966). The CAPM defines systematic risk, measured by beta (b), as the relevant portion of total risk since investors can diver- sify away the remaining portion. 6 Usually CAPM-based performance models describe covariance with the market portfolio, however, as noted earlier, they can attempt to describe coskewness and cokurtosis as well. 7 Arbitrage pricing theory (APT) establishes the conditions under which a multi- index model can be an equilibrium description (Ross, 1976). c05_gregoriou.qxd 7/27/04 11:07 AM Page 83 Salomon Brothers government bond index, and U.S. dollar trade-weighted currency index, the MLM Index 8 ) with absolute S&P 500 returns and intramonth S&P return volatility in a multifactor regression analysis to describe the sources of return to hedge funds, managed futures, and mutual funds. The index returns employed in the regression analysis are intended to be risk factors that explain the source of natural returns. The explana- tory variable, absolute equity returns, captures the source of return that derives from the ability to go short or long. Returns from the use of options or intramonth timing strategies are proxies for the intramonth standard deviation. The MLM Index, an active index designed to mimic trend- following strategies, is used to capture returns from market inefficiencies in the form of temporary trends. Seigel (2003) provides a comprehensive review of benchmarking and investment management. Despite the fact that CTAs and many hedge fund managers follow absolute return strategies, various CTA benchmarks now exist, as described by Seigel (2003). Addressing Time-Varying Risk Single-parameter risk measures are problematic if managers are changing fund betas over time, as they would if they were attempting to time the mar- ket. For example, when equity prices are rising, the manager might increase the fund’s beta and vice versa. Although market risk can be measured if the portfolio weights are known, this information is generally not publicly available and other techniques must be employed. 9 84 PERFORMANCE 8 Mount Lucas Management Index TM is based on a concept conceived in 1988 of an index methodology that involves changing (commodity) market sides long and short to measure economic return. 9 Treynor and Mazuy (1966) added a quadratic term to the basic linear regression model to capture nonlinearities in beta resulting from market timing activities. Kon and Jen (1978, 1979) use a switching regression technique. Merton (1981) and Hen- riksson and Merton (1981) develop nonparametric and parametric option-based methods to test for directional market timing ability. The nonparametric approach requires knowledge of the managers’ forecasts. The more commonly employed parametric approach involves adding an extra term to the usual linear regression model and is CAPM based. Ferson and Schadt (1996) note that fund betas may change in response to changes in betas of the underlying assets as well as from changing portfolio weights. They modify the classic CAPM performance evaluation techniques to account for time variation in risk premiums by using a conditional CAPM framework. This method removes the perverse negative performance often found in earlier tests and suggests that including information variables in perform- ance analysis is important. c05_gregoriou.qxd 7/27/04 11:07 AM Page 84 Mitev (1998) uses a maximum likelihood factor analysis technique to classify CTAs according to unobservable factors. Similarly, Fung and Hsieh (1997b) also use a factor-analytic approach to classify hedge funds. In both cases, the results identify general investment approaches or trading strate- gies (e.g., trend-following, spread strategies, or systems approaches) as sources of returns to these alternative investment classes. Factor analysis and multifactor regression analysis differ in their approach to identifying the factors (benchmarks) that serve as proxies for risk. In multifactor regression analysis, the factors are specified in advance. Factor analysis will identify funds that have common yet unobservable factors, although the factors can be inferred from the qualitative descriptions of the funds. While this may seem redundant, the clustering of funds is done independently of the qualitative descriptions in a formal data-driven process. The data envelopment analysis methodology used in this chapter, and described in more detail in Wilkens and Zhu (2001, 2004), incorporates multiple criteria and “benchmarks” funds or other securities according to these criteria. This is distinctly different from multifactor analysis. Here benchmarks are not risk factors but rather are efficient securities as defined in n dimensions where each dimension represents risk and return criteria. Recently Gregoriou (2003) used the DEA method in the context of bench- marking hedge funds. Skewness and Kurtosis: Questioning Mean-Variance Sufficiency The standard CAPM framework assumes that investors are concerned with only the mean and variance of returns. Ang and Chau (1979) argue that skewness in returns distributions should be incorporated into the perform- ance measurement process. Even if the returns of the risky assets within a portfolio are normally distributed, dynamic trading strategies may produce nonnormal distributions in portfolio returns. Both Prakash and Bear (1986) and Stephens and Proffitt (1991) also develop higher-moment performance measurements. Fishburn (1977), Sortino and van der Meer (1991), Marmer and Ng (1993), Merriken (1994), Sortino and Price (1994), and others also have developed measures that take into account downside risk (or semivariance) rather than the standard deviation of returns. Although some differences exist among these measures, the Sortino ratio captures their essence. Whereas the Sharpe ratio is defined as excess return 10 divided by standard CTA Performance Evaluation with Data Envelopment Analysis 85 10 Return minus the risk-free rate. c05_gregoriou.qxd 7/27/04 11:07 AM Page 85 deviation, the Sortino ratio is defined as return divided by downside devia- tion. Downside deviation (DD) measures the deviations below some mini- mal accepted return (MAR). Of course, when the MAR is the average return and returns are normally distributed, the Sharpe and Sortino ratios will measure the same thing. Martin and Spurgin (1998) illustrate that even if individual asset or fund returns are skewed, the skewness tends to be diversified away at the portfolio level. However, they also illustrate that managers may choose to follow strategies that produce skewed returns as a form of signaling their skill. Note that coskewness remains irrelevant if it can be diversified away, but skewness may have some signaling value. Addi- tionally, the popularity of the related value at risk (VaR) measure 11 and the common practice of reporting drawdown 12 information for various alter- native investments suggest that skewness may be important, whether in terms of investor utility or skill signaling. Beta-Squared Coefficient The classic paper by Fama and MacBeth (1973), and several other early papers (e.g., Carroll and Wei 1988; Shanken 1992) empirically test a two-pass regression methodology for stock returns. Assuming a nonlinear relationship between stock returns, the tests include beta-squared in the second-pass regression. These tests find that the coeffi- cient for beta-squared is negative and statistically significant, providing evi- dence of a nonlinearity in stock returns. Schneeweis and Georgiev (2002, p. 7) provide evidence that CTAs have nonlinear returns with respect to the equity market: “When S&P 500 returns were ranked from low to high and divided into four thirty-three month sub-periods, managed futures offered the opportunity of obtaining positive returns in months in which the S&P 500 provided negative returns as well as in months in which the S&P 500 reported positive returns.” We include equity beta-squared in our Tobit regressions where the dependent variable is not the expected return of the CTA, but is rather the efficiency score obtained in the DEA models. Although the dependent variable is not the same as in the earlier stock studies, we might hypothe- size that CTA efficiency scores are also negatively related to beta-squared. 86 PERFORMANCE 11 See Chung (1999) for a concise review of VaR methodologies. 12 Drawdown information is generally reported as the maximum drawdown over a period and is defined as the return from a fund’s net asset value peak to trough. The Calmar ratio is a similar measure that CTA investors are often interested in and is defined as the average annual return over the past three years divided by the absolute value of the maximum drawdown during that period. c05_gregoriou.qxd 7/27/04 11:07 AM Page 86 We infer a direct correspondence between the efficiency score and expected return. The CTA returns observed by Schneeweis and Georgiev (2002), therefore, imply a positive coefficient. Finally, we note that the efficiency scores used in this study minimize variability. This leads to the hypothesis that the beta-squared coefficient is negatively correlated with the efficiency score, unless the enhanced return from high (absolute) betas is an offset- ting factor. Fund Size In his chapter “The Lure of the Small,” Jaeger (2003) describes how small firms and small portfolios are desirable features of hedge funds. Small firms satisfy hedge fund managers’ entrepreneurial spirit, and small portfolios are often necessary to enable hedge funds to implement their strategies, especially if they trade in markets that are sometimes illiquid. Gregoriou and Rouah (2002) find, however, that fund size does not matter to hedge fund performance. Being a subset class of hedge funds, CTAs are examined in this chapter to see if fund size or length of manager track record is related to the DEA efficiency scores. Determinants of Performance Based on the discussion above, we choose as bases for performance evaluation in a DEA model monthly returns, kurto- sis, minimum return, skewness, standard deviation of returns, and percent- age of negative monthly returns. We then investigate the potential of fund size, length of track record, strategy, and style to impact performance scores of funds created by the DEA model. DATA DESCRIPTION Monthly CTA return data for 216 CTAs over two periods surrounding March 2000 are obtained from the Center for International Securities and Derivatives Markets (CISDM) Alternative Investment Database. 13 The first period is an up-market period for the equity market (March 31, 1998, to February 28, 2000) and the second period is a down market environment (April 30, 2000, to March 31, 2002). The daily high for the S&P 500 occurred in March 2000, as illustrated in Figure 5.1. The mean monthly return for the S&P 500 was 1.28 percent and −1.11 percent for the first and second periods, respectively. CTA Performance Evaluation with Data Envelopment Analysis 87 13 We selected funds from the database with the most complete information on investment styles and strategies. c05_gregoriou.qxd 7/27/04 11:07 AM Page 87 Performance criteria used in the DEA model were calculated from the CTA returns for each of the two periods. The DEA approach to “estimat- ing” the efficient frontier is a nonstatistical approach. As a result, all devi- ations from the efficient frontier are measured as inefficiency (i.e., there is no allowance for statistical noise). The efficiency measures obtained from this method are, therefore, very sensitive to the effect of outliers. Hence, for each performance criterion used in the DEA model, particular effort was made to detect any outliers. CTAs with outliers in one subperiod were deleted from both subperiods so as to have the same group of CTAs. Our final sample consisted of 157 CTAs that were used for analysis in the DEA model and the subsequent Tobit regression analysis. Table 5.1 provides descriptive statistics for the DEA model criteria over both periods and for the full and final sample. Other information we use from the CISDM Alternative Investment Database includes the assets under management over time, the dates the funds were established, and information on the investment style 14 88 PERFORMANCE 750 850 950 1,050 1,150 1,250 1,350 1,450 1,550 Date 11-Mar-98 18-May-98 24-Jul-98 30-Sep-98 7-Dec-98 16-Feb-99 23-Apr-99 30-Jun-99 7-Sep-99 11-Nov-99 20-Jan-00 28-Mar-00 5-Jun-00 10-Aug-00 17-Oct-00 22-Dec-00 5-Mar-01 10-May-01 18-Jul-01 28-Sep-01 5-Dec-01 14-Feb-02 24-Apr-02 1-Jul-02 6-Sep-02 12-Nov-02 Daily Closing Value FIGURE 5.1 S&P 500 Daily Closing Values, from 1998 to 2002 14 We follow the terminology established by Sharpe (1992) and call the market focus investment style. c05_gregoriou.qxd 7/27/04 11:07 AM Page 88 [...]... . style 14 88 PERFORMANCE 750 850 950 1, 050 1, 150 1, 250 1, 350 1, 450 1 ,55 0 Date 11-Mar-98 18-May-98 24-Jul-98 30-Sep-98 7-Dec-98 16-Feb-99 23-Apr-99 30-Jun-99 7-Sep-99 11-Nov-99 20-Jan-00 28-Mar-00 5- Jun-00 10-Aug-00 17-Oct-00 22-Dec-00 5- Mar-01 10-May-01 18-Jul-01 28-Sep-01 5- Dec-01 14-Feb-02 24-Apr-02 1-Jul-02 6-Sep-02 12-Nov-02 Daily. <2 .5 23 15 2 .5 – < ;5 14 9 2 .5 – < ;5 17 11 5 – <10 13 8 5 – <10 17 11 10 – <20 25 16 10 – <20 15 10 20 – <30 8 5 20 – <30 17 11 30 – <40 14 9 30 – <40 11 7 40 – < ;50 . – < ;50 9 6 50 – <100 27 17 50 – <100 14 9 100 – < 150 7 4 100 – < 150 8 5 150 – <200 2 1 150 – <200 5 3 200 – <400 14 9 200 – <400 15 10 400+ 8 5 400+ 64 Overall 157 100