Abstract
Selection-bias corrections for backtested trading strategies, from the Deflated Sharpe Ratio to simple trials-adjusted performance bars, all depend on one input: the number of trials . When a large language model is the strategy-search process, is not known and cannot safely be guessed. The model emits many candidate strategies, but they are highly correlated, so the raw proposal count over-states how many independent bets were placed. I borrow the effective number of tests estimator from statistical genetics, computed from the eigenvalues of the candidates' return-correlation matrix, and use it as an effective trial count . The correction matters a great deal in some places and little in others. It is first-order in the log-style trials-adjusted bars practitioners deploy, and second-order in the Deflated Sharpe Ratio, whose benchmark grows only as the square root of the log of . My main empirical finding, on the real ÀYÒ pipeline, is that the size of the correction depends on the generation protocol. A diverse prompting protocol yields only mildly redundant proposals ( about 0.72 to 0.76 of , moving the bar by about 0.07 Sharpe), while drawing repeatedly from a single narrow angle collapses eight proposals to roughly two effective trials, inflating the selection bar by about 0.33 Sharpe and flipping real deployment decisions. Across three independent replicates per protocol the two regimes separate without overlap (1.39 ± 0.09 versus 4.54 ± 1.25 over-count; Welch , on about 2 degrees of freedom), and shows which of them a given generator is in.
1Introduction
The failure mode of amateur quantitative research is well known: try many strategies on one dataset, keep the best, and mistake the reward for searching for an edge. The standard defenses raise the bar in proportion to how many strategies were tried. Bailey and López de Prado's Deflated Sharpe Ratio (DSR) deflates the best observed Sharpe by the expected maximum of trial Sharpes. Practitioner gates, including the one in ÀYÒ, raise a required out-of-sample Sharpe as a function of .
All of these assume is known, and the assumption breaks without any warning when the search process is a language model. An LLM asked for trading strategies does not draw independent hypotheses. It produces many variations on a few themes (momentum here, mean reversion there), each dressed up again. The raw proposal count therefore over-states the independent search effort, and any correction keyed to is miscalibrated. The question for this paper is what the right is when a generative model does the searching, and where getting it wrong changes decisions.
2Background
The Deflated Sharpe Ratio of Bailey and López de Prado (2014) sets the benchmark Sharpe that a best-of- selection must beat at , where is the cross-trial variance of Sharpe estimates and is the expected maximum of standard normals, approximated by with the Euler-Mascheroni constant. The DSR is the probabilistic Sharpe of the selected strategy against this inflated benchmark.
For the probability of backtest overfitting, Bailey, Borwein, López de Prado and Zhu use combinatorially symmetric cross-validation (CSCV) to estimate the probability that the in-sample winner underperforms the median out-of-sample. The method is model-free but still treats trials as distinct.
In statistical genetics, correlated tests across linked loci are handled with an effective number of independent tests , recovered from the eigenvalues of the test correlation matrix (Cheverud 2001; Li and Ji 2005; Galwey 2009). When tests are correlated, , and the multiple-testing correction is less severe accordingly. Finance already has an effective-trials estimator of a different kind: López de Prado and Lewis (2019) cluster the return series of the strategies tried and take the number of clusters as the number of effectively uncorrelated trials, a procedure also set out in López de Prado (2020). As far as I know, applying the eigenvalue estimator from genetics to batches of LLM-proposed strategies, and measuring how its size depends on the prompting protocol, has not been done before.
3Method
Given candidate strategies with per-bar return series, form the return-correlation matrix and take its eigenvalues (clipped at zero). I use Galwey's estimator as the default because it behaves well at the endpoints:
It equals at zero correlation and tends to 1 as correlation tends to 1. then replaces the raw count wherever a trials count is required, in the DSR's expected-max term and in the practitioner log-bar. Li and Ji's estimator gives materially the same numbers throughout (Section 6). Section 5.5 tests the estimator's own stability and finds it near-unbiased and tight at realistic sample lengths, with no need for shrinkage.
4Where the correction bites
How large the correction is depends entirely on how fast the gate's penalty grows in .
- In the practitioner log-bar the effect is first-order. ÀYÒ's gate requires . Going from to moves the required out-of-sample Sharpe from 1.80 to about 1.45, a change of 0.35 that can decide whether a strategy is deployed.
- In the Deflated Sharpe Ratio it is second-order. The DSR benchmark grows as . From 40 to 8 that is roughly a 25 percent change in the benchmark, and the DSR used as a gate is conservative at realistic sample sizes anyway (Section 5.2). The count barely moves its decisions.
The correction is therefore most useful for the log-style gates practitioners run, and makes little difference inside the DSR.
5Synthetic experiments
All experiments use daily bars and proposals with a controllable equicorrelation , built from one common factor: . Every strategy is pure noise unless an edge is explicitly injected.
5.1 The correction tracks proposal correlation
falls from about 38 at to about 8 at , while the raw count stays at 40. A fixed count cannot follow the changing redundancy of LLM proposals, and can.
5.2 Null calibration, and a negative result
Under the operational rule "deploy if DSR > 0.95" (nominal 5 percent false-deploy size), the actual false-deploy rate under pure noise is:
| rho | M_eff | N=1 | N=m | N=M_eff |
|---|---|---|---|---|
| 0.0 | 38.4 | 100.0% | 0.0% | 0.0% |
| 0.3 | 31.5 | 99.8% | 0.7% | 0.9% |
| 0.6 | 21.2 | 92.0% | 2.5% | 4.2% |
| 0.9 | 8.2 | 41.1% | 5.1% | 7.6% |
Mean absolute deviation from nominal is 81.4 percent for , 3.1 percent for and 3.0 percent for . Some correction is essential, since deploys noise almost always. Beyond that, and are nearly tied here, because the DSR test is conservative at these sample sizes for reasons unrelated to the count. The selection winner fluctuates with standard deviation about 0.030 around the benchmark, but the DSR judges it against the single-strategy standard error of about 0.063, so it rarely fires for any count. I checked that the expected-max approximation is not the cause (asymptotic 0.1379 vs exact 0.1361). The DSR is the wrong place to look for the effect of the effective count.
5.3 Impact on the practitioner gate, with a decision flip
Where the penalty is first-order, the correction changes decisions. Suppose a strategy with a real edge and an out-of-sample Sharpe of 1.6 arrives:
| rho | M_eff | bar(m=40) | bar(M_eff) | m verdict | M_eff verdict |
|---|---|---|---|---|---|
| 0.0 | 38.4 | 1.80 | 1.79 | REJECT | REJECT |
| 0.3 | 31.5 | 1.80 | 1.75 | REJECT | REJECT |
| 0.6 | 21.2 | 1.80 | 1.66 | REJECT | REJECT |
| 0.85 | 10.6 | 1.80 | 1.51 | REJECT | DEPLOY |
At high proposal correlation, which is the regime LLM search produces, the raw-count bar rejects this strategy and the effective-trials bar correctly deploys it.
5.4 Why not just compare power at matched size
I also ran a matched-size power test: calibrate each count to reject noise at exactly 5 percent, then measure how often each deploys an injected edge. The three counts came out within a few points of each other, which is what one should expect. Once size is matched, the three DSRs are monotone re-thresholds of the same underlying winner statistic, so they trace nearly the same ROC curve. The count leaves the achievable power unchanged and moves where the nominal operating point lands. For that reason the effect shows up in calibration and in the position of a fixed log-bar, and not in raw power.
5.5 Stability of the estimator
A trial count is only useful if it is stable at the sample sizes practitioners have. I ran a Monte Carlo of the Galwey estimator on equicorrelated batches (, , population ) across sample lengths and shrinkage intensities:
| T | raw | shrink 0.1 | shrink 0.2 |
|---|---|---|---|
| 126 | 8.8 ± 0.4 | 9.6 ± 0.3 | 10.3 ± 0.3 |
| 252 | 8.9 ± 0.3 | 9.6 ± 0.3 | 10.3 ± 0.2 |
| 504 | 8.9 ± 0.2 | 9.7 ± 0.2 | 10.4 ± 0.1 |
The raw estimator is near-unbiased and tight even at half a year of daily data ( against a target of ), and its bias and variance both shrink with as expected. I had expected shrinkage to help, but shrinking the correlation matrix toward the identity makes things worse. It biases upward toward , because pulling correlations toward zero makes the candidates look more independent than they are. I therefore use the raw estimator with no shrinkage, so the method has no tuning knob.
5.6 Is the linearity assumption doing damage?
is computed from a Pearson correlation matrix, which sees only linear co-movement. An obvious worry is that strategy dependence is not linear. Two rules could be flat on most days and large on the same few days, and a linear measure would call them independent, inflating and under-correcting the bar. Since that is the dangerous direction of error, I tested it. Code: research/rank_correlation_check.py.
I recompute from a Spearman rank-correlation matrix across four regimes with known structure (, ):
| regime | Pearson | Spearman | delta |
|---|---|---|---|
| independent | 11.95 | 11.95 | +0.00 |
| linear, | 7.97 | 8.22 | +0.25 |
| monotone nonlinear | 3.72 | 4.38 | +0.66 |
| shared tails only | 8.01 | 11.72 | +3.71 |
Three of the four agree closely. In the monotone nonlinear regime, where a single driver reaches each strategy through a different convex transform, both give almost the same answer. That is the case rank correlation is usually recommended for, and it turns out not to be a problem for .
The fourth regime runs opposite to the worry above. When strategies are independent on ordinary days but move together on a handful of shared shocks, Pearson catches the dependence () and Spearman misses it (). Ranking discards magnitude, so a common crash becomes one more day on which everything ranked high, and 5 percent of the sample cannot move a rank correlation far. Yet shared tail behavior is the dependence that matters when the question is how many independent bets were placed. Strategies that all lose money in the same week are not independent trials, however uncorrelated their quiet days look.
A higher lowers the selection bar, so the rank variant errs in the unsafe direction in the cases where the stakes are highest. I therefore keep the linear estimator. The choice is a substantive one, since the test came out against the rank variant.
6Empirical study on ÀYÒ
I asked the ÀYÒ research brain (the on-box claude CLI) for ten strategies, one for each of the first ten angles in a list of twelve deliberately diverse ones (momentum, mean reversion, breakout, quality-and-trend, reversal, volume confirmation, low-volatility drift, and others), and backtested each on one shared ten-name universe over 910 real daily bars from Alpaca. I then measured the return-correlation matrix and its effective rank. Code: research/measure_ayo_correlation.py.
The ten proposals had an average pairwise return correlation of +0.32 and an effective count of M_eff = 7.3 (Galwey) / 7.0 (Li and Ji) against a raw count of . The redundancy is also visible in the rule vocabulary: nine of the ten proposals used the sma_20_over_sma_50 trend filter and most also used rsi_14, despite the diverse prompt angles. The model draws repeatedly from a small set of indicators, which is the behavior that makes the raw count over-state the number of independent trials.
At this batch size the effect on the log-bar is modest: versus , a 0.07 Sharpe reduction with no deployment flips at realistic out-of-sample Sharpe levels. The log-bar penalizes the redundancy ratio (here about 1.37), and that ratio belongs to the proposal generator and does not depend on batch size. The correction therefore stays near 0.07 Sharpe as grows, unless the ratio itself grows.
I also measured redundancy in signal space, expecting a different answer from the one I got. My hypothesis was that return-stream correlation understates redundancy, since strategies encoding the same idea might trade at different times, and that measuring holdings overlap (whether two strategies are in a position on the same days) would reveal a lower, truer . On a fresh 12-strategy batch I measured both on the same proposals. The hypothesis was refuted. Signal-space average correlation was +0.07 with , higher (more independent) than return-space (+0.23, ), against . The proposals share vocabulary without sharing behavior. They reuse the same indicators with opposite timing (a momentum rule holds in uptrends, a mean-reversion rule after drops, a fade-overbought rule at RSI peaks), so they are rarely in the market together.
To test whether catches redundancy when it is present, I then induced it. I repeated the measurement but drew fourteen proposals from a single narrow angle (oversold RSI mean reversion) in place of twelve diverse ones. The model converged hard, and nearly every proposal was a variant named "Oversold RSI Snapback." Average return correlation rose to +0.72 and signal-space correlation to +0.78. With the two spaces now in agreement, the redundancy is present in both idea and holdings. The result is against , a 4.6x over-count. The trials-adjusted bar drops from 1.57 to 1.24, and a candidate with out-of-sample Sharpe 1.3 that raw counting rejects (as one of 14 trials) is correctly deployed once counted as one of ~3 effective trials.
| generator | avg corr (return) | M_eff | m/M_eff | bar |
|---|---|---|---|---|
| diverse angles (ÀYÒ's protocol) | +0.23 | 9.1 | 1.3x | 1.48 |
| single narrow angle | +0.72 | 3.0 | 4.6x | 1.24 |
The size of the effective-trials correction is therefore set by the generation protocol and is not a universal constant, and both extremes appear on one real system. A diverse protocol produces mildly redundant proposals ( about 0.72 to 0.76 of ) and needs only a small correction, about 0.07 Sharpe. A redundant protocol over-counts by 4 to 5 times, inflates the required Sharpe by roughly 0.33, and wrongly rejects strategies that have an edge. identifies which regime a given generator is in and applies a correction only when one is warranted. For ÀYÒ it confirms that the diverse-angle proposal design keeps the existing raw-count bar close to correct. That is a useful negative result in its own right, because it shows prompt diversity is doing work: it is what keeps the raw-count bar close to calibrated.
6.1 Does protocol-dependence replicate?
The contrast above comes from one diverse batch and one narrow batch. With a single batch per arm, a real protocol effect cannot be told apart from two lucky draws. I therefore repeated the whole procedure three times per protocol, with independent proposals, on the same universe and the same 910 real daily bars. Code: research/measure_protocol_replicates.py; raw results in research/protocol_replicates.jsonl.
| protocol | batches | avg corr | bar reduction | ||
|---|---|---|---|---|---|
| diverse angles | 3 | +0.289 | 5.79 | 1.39 ± 0.09 | 0.070 |
| single narrow angle | 3 | +0.925 | 1.86 | 4.54 ± 1.25 | 0.322 |
Per-batch over-count ratios were 1.45, 1.29, 1.42 for the diverse protocol and 3.22, 4.70, 5.70 for the narrow one. The two sets do not overlap, since the worst diverse batch (1.45x) is still far below the best narrow batch (3.22x), and a Welch test on the ratios gives on about 2 degrees of freedom, two-sided . (The replicate script prints because it uses a normal approximation, which overstates the evidence at three batches per arm.)
The diverse protocol is very stable (1.39 ± 0.09). Asking for different strategy families produces proposals that are only mildly redundant, batch after batch. The narrow protocol is much more redundant and also much more variable (4.54 ± 1.25). When the model is pinned to one angle, how badly it collapses depends on how hard it happens to converge that time. In one narrow batch all eight proposals came back with the same name, "RSI Oversold Mean Reversion with Trend Filter".
The redundancy here is more severe than in the single batches at the start of Section 6. At the narrow protocol reaches , an average return correlation of +0.92. The protocol effect does not depend on one batch size or one run, and it is the dominant term.
7Limitations
- Return correlation is linear. I tested a Spearman variant (Section 5.6) and kept the linear estimator, because rank correlation is blind to shared tail events, which are the dependence that matters most here, and it errs toward a higher and a laxer bar. Dependence structures beyond co-moving tails and monotone transforms remain unexamined.
- The effect is concentrated where the proposal generator is redundant. A diverse generator gives of about three-quarters of and there is little to correct, as I found for ÀYÒ's own protocol across three replicates.
- Proposals arriving adaptively (bandit-style) break the fixed-batch assumption; I limit the scope to the batch case.
- The residual conservatism of the DSR used as a gate is a separate, count-independent phenomenon and is not addressed here.
- I evaluate on one strategy family space (typed technical rules over a fixed indicator vocabulary) and one proposing model. Three replicates per protocol show that the separation is not batch noise. They do not show that it holds for richer proposal spaces or other models.
8Conclusion
When a language model is the search process, the trial count that selection-bias corrections depend on is the effective number of independent proposals, which can be much smaller than the number of proposals. The effective-number-of-tests estimator from genetics gives an that is near-unbiased and stable at realistic sample sizes. Its impact is first-order in the trials-adjusted bars practitioners run and second-order in the Deflated Sharpe Ratio. On a real LLM-driven quant pipeline, the size of the correction turned out to depend on the generation protocol. A diverse prompting protocol produces mildly redundant proposals that need only a small correction, while a narrow one collapses proposals to a handful of effective trials and wrongly rejects strategies that have an edge, and this separation holds across independent batches. tells you which regime you are in, and as a negative result it confirms that prompt diversity is what keeps a generative strategy search close to its raw trial count.
References
- Bailey, D. H., & López de Prado, M. (2014). The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality. Journal of Portfolio Management.
- Bailey, D. H., Borwein, J. M., López de Prado, M., & Zhu, Q. J. (2017). The Probability of Backtest Overfitting. Journal of Computational Finance.
- Cheverud, J. M. (2001). A simple correction for multiple comparisons in interval mapping genome scans. Heredity.
- Li, J., & Ji, L. (2005). Adjusting multiple testing in multilocus analyses using the eigenvalues of a correlation matrix. Heredity.
- Galwey, N. W. (2009). A new measure of the effective number of tests. Genetic Epidemiology.
- López de Prado, M., & Lewis, M. J. (2019). Detection of false investment strategies using unsupervised learning methods. Quantitative Finance, 19(9), 1555–1565. https://doi.org/10.1080/14697688.2019.1622311
- López de Prado, M. (2020). Machine Learning for Asset Managers. Cambridge University Press (Elements in Quantitative Finance). https://doi.org/10.1017/9781108883658