Unlike past portfolio performance, this measure of investment decision-making skill exhibits a statistically significant relationship into the future.
By Clare Flynn Levy
Clare Flynn Levy is CEO & Founder of Essentia Analytics. Prior to setting up Essentia, she spent ten years as a fund manager, in both active equity (running over $1B of pension funds for Deutsche Asset Management), and hedge (as founder and CIO of Avocet Capital Management, a specialist tech fund manager).
For the past year or so, Essentia Analytics has been proclaiming the virtues of our skills-based portfolio manager assessment methodology, the Behavioral Alpha® Benchmark, now published in the peer-reviewed Journal of Investing.
And for good reason: the Benchmark (we refer to it as EBAB) provides a far better lens on the real-world capabilities of a portfolio manager than the traditional return-based assessment metrics, because it comes at the issue from the bottom up — that is, from the actual trading decisions the manager has made. Approaching it from this direction provides significantly more data points — even for a low-turnover manager — than monthly performance figures do, and therefore a better shot at identifying a signal in the noise.
Over the last ten years, we have developed and refined this novel approach in our work with portfolio managers. Today, over 100 active equity managers around the world use it as a data-driven feedback loop on the quality of their decision-making, both to prove their skill to investors and to improve that skill on a continuous basis.
But our latest research shows why the EBAB, and its underlying Behavioral Alpha Score, is also highly valuable to those whose role is manager selection and monitoring.
It turns out that unlike past portfolio returns (which we all know bear little if any relationship to future performance), we found evidence that past demonstrated decision-making skill — as measured by the manager’s overall Behavioral Alpha Score — does exhibit a temporal relationship: a manager who exhibits good decision-making in the recent past (the last six months, for the median manager) is likely to continue to do so in the foreseeable future (just under two years, at the median).
Essentia’s Behavioral Alpha Score is a measure of portfolio manager decision-making skill that considers hit rate (the frequency of value-additive decisions) and payoff (the magnitude of the value added or destroyed by each decision). The decision-types toward the upper right — furthest away from 0,0 — represent the highest value-add.
That makes common sense: it is reasonable to expect someone who has exhibited true skill at any activity — investment-related or otherwise — in the recent past to continue to do so into the near future. But it’s also a very big deal: measures of investment skill that actually persist are rare indeed.
This is a brand-new area of inquiry that we expect will be the subject of much investigation and discussion for years to come. Our team is working on a formal paper about this initial research, but in the meantime, here are some highlights — courtesy of Essentia Data Scientist Isaac Kelleher-Unger.
The Research
In this inquiry, we set out to answer a fundamental question: does our measure of past investment skill bear any relationship to the same measure in the future? We all know that past returns do not — largely because they are subject to the random effects of luck. But what about measures of decision-making skill — not just stock-picking decisions, but sizing and timing decisions, as well?
Methods & Results
Applying the EBAB methodology to our database of daily trade data from 123 portfolios (with a median six years’ worth of trading data), we looked at each portfolio manager’s (or team’s) decisions — divided into picking, entry timing, scaling in, sizing, size adjusting, scaling out and exit timing decisions — and calculated the impact of each of those decisions on the portfolio relative to its benchmark. We split these decisions into discrete, non-overlapping batches of 240 (the number that provided a quantitatively robust measure of the sample mean), and for each batch, we calculated an overall Behavioral Alpha Score. The duration of each batch varied depending on the decision-making frequency of the manager at that time; the median batch period across all portfolios in our study was 175 calendar days (roughly six months).
Next, we looked to see if there was any relationship between one batch of decisions and the next. To do this, we fit a general linear model and found (Figure 1) that there was, indeed, a statistically significant relationship between a score at time t (St) and the score of the next batch (St+1) (β = 0.29, R2 = 0.09, p < 0.0001).
Figure 1: This linear regression shows a statistically significant relationship between the quality of decisions made by a manager from one period (X-axis: Behavioral Alpha Score at point t, or St) to the next (Y-axis: St+1).
Our next question was how long does this decision-quality relationship persist?
On the one hand, we would expect someone who has made skilled investment decisions in the past, perhaps as a matter of a strong process, to continue to do so.
On the other hand, we might expect that temporal relationship to deteriorate as time passes and environments change: for example, just because an investor had an excellent decision-making process in 2015 doesn’t mean they used it in 2020. And a manager whose decision-making has been sub-par in the past can improve, once they are aware of what they are doing that is hurting. Indeed, we have to keep in mind that the dataset here is comprised of portfolio managers who are working with Essentia Analytics — that is, they are all on a mission of continuous improvement.
To that end, we hypothesized that over time, the relationship between scores would decrease. That’s what we found: the relationship (as represented in Figure 2; green line) remained significant until four periods after an initial observed score (Pcorrected = 0.003) and fell below significance thereafter.
Figure 2: The dotted line represents the threshold below which there is no longer a statistically significant relationship with an initial observed score. That point occurs after four periods — which, for the median portfolio in our study, was about 24 months.
In other words, a portfolio manager with a good decision-making score could be expected to continue to make good decisions for the next almost 24 months (in the median-portfolio case), but after that, the relationship with the quality of the manager’s decision-making in the initial period was no longer statistically significant.
Conclusions
So what do these findings mean, in practice? The clear takeaway is that the Behavioral Alpha Score provides fund managers and their investors with a headline measure of decision-making skill that has a statistically significant probability of persisting over the next four decision-making periods. In the case of the median portfolio used in this study, for which each period was roughly six months, the quality of decision-making as measured by the Behavioral Alpha Score endured a further two years.
While we would not argue that this score should be a sole basis for making manager selection decisions, the data evidence shows that it is a powerful additional lens to apply to the analysis of a manager once the rest of due diligence has been done.
Focusing on decision-making (which is, after all, what the fund manager actually does day-to-day) — rather than on portfolio performance (which is often heavily influenced by exogenous forces the manager cannot control) — we are able to provide both the manager and the investor with previously unavailable insight into whether that manager is likely to make good, value-additive decisions (which, over time, can be expected to lead to good performance — see our EBAB methodology paper for more on this).
As I mentioned above, this marks the beginning of an important new area of inquiry into how portfolio managers are assessed. We’ll be publishing a full research paper on this initial work in due course, but in the meantime, if you wish to stay informed as this work develops, please subscribe to our mailing list — or contact us, if you’d like to discuss how to put the EBAB methodology to work in your organization.
