Abstract
A language model can propose trading strategies faster than a human can read them. On its own that is worth little, because the binding constraint in systematic trading was never idea generation; it is avoiding ideas that only worked in the past. A strategy selected as the best of many candidates carries selection bias. Its in-sample performance is inflated by luck in proportion to how many candidates were tried, and the naive backtest that produced it cannot see the inflation. This paper describes and evaluates a validation gate designed so that a strategy of that kind cannot pass it, and reports what happened when the gate was run on real markets.
The system, ÀYÒ, separates a nondeterministic proposer (an LLM that authors strategies as typed, content-hashed data artifacts, never as executable code) from a deterministic verifier that trades only the frozen artifact. The verifier applies three defenses against backtest overfitting: strict out-of-sample evaluation with a walk-forward split, a trials-adjusted acceptance bar whose required out-of-sample Sharpe rises with the number of candidates tested, and robustness sweeps across rolling windows and parameter settings. We run all three on about four years of real U.S. equity data (Alpaca daily bars, 32 to 83 symbols, ending 2026-07-17) and report every number the gate produced. A single rule-based strategy selected for its in-sample Sharpe of 3.07 fell to an out-of-sample Sharpe of −4.90 and was rejected. A cross-sectional momentum book beat buy-and-hold out-of-sample by a Sharpe edge of +0.13 in one configuration, but the robustness sweep showed this to be a fluke: it beat the benchmark in only 2 of 5 rolling windows and in 0 of 12 parameter settings. Nothing survived out-of-sample validation. That is the expected result on efficient large-cap markets, and refusing to deploy in that situation is what the gate is for. The paper offers no alpha. What it contributes is a methodology, a reproducible gate that reliably refuses to deploy mirages and that carries over to any setting where a nondeterministic proposer must be held to a deterministic guarantee. Throughout, we say which claims the evidence supports and which remain open.
1Introduction
Suppose you can generate trading strategies at will. You describe a market intuition, and moments later you have a fully specified, backtestable strategy with entry rules, exits, and risk parameters. Large language models make this possible, and cheap. The natural reaction is to generate many strategies, backtest them all, and deploy the best one.
That is the wrong thing to do, and it is the single most common way retail systematic traders lose money. The problem is selection under multiplicity. If you test enough strategies on one dataset and keep the best, the winner's backtest performance is inflated by luck, and the inflation grows with the number of candidates tried [BacktestOverfitting; White; STW]. The backtest that selected the winner cannot see its own optimism. On efficient markets, a strategy with a beautiful in-sample track record and no real edge is the default output of an unconstrained search, far from a rare pathology.
The resulting asymmetry is the premise of the present work. Generating candidate strategies is easy and getting easier. Avoiding being fooled by them is the hard part, and a faster proposer does not make it easier. If anything it makes it harder, since a faster proposer means more trials, and more trials mean more selection bias. A system that pairs a powerful generator with a weak validator is worse than one with no generator at all, because it turns out mistakes that look confident and well dressed.
ÀYÒ is built around that asymmetry. It treats strategy generation as cheap and untrusted, and puts its engineering into a validation layer that a plausible but wrong strategy cannot pass. Concretely:
- The proposer is untrusted and quarantined. An LLM authors each strategy as a typed data artifact, a set of declarative rules, and never as code. The artifact is frozen and content-hashed. The model's output never executes; only the deterministic interpreter that reads the frozen artifact does. If the model is removed entirely, the system still trades its existing rules safely and only stops acquiring new ones.
- The verifier is deterministic and adversarial to its own inputs. It evaluates every artifact out-of-sample, raises the acceptance bar in proportion to how many candidates were tried, and requires any apparent edge to survive changes to both the time window and the parameters.
The rest of the paper concerns the verifier and its behavior on real data. It makes three contributions:
- A gate design that puts together three standard defenses against backtest overfitting that are seldom used together (walk-forward out-of-sample evaluation, a trials-adjusted acceptance bar, and robustness sweeps), behind a frozen, content-hashed artifact that keeps a nondeterministic author out of the execution path (§3 and 4).
- A real-data evaluation (§5) in which the gate rejects every candidate it was given. These include (a) a single strategy whose in-sample Sharpe of 3.07 became −4.90 out-of-sample, and (b) a momentum book whose only positive out-of-sample number (a +0.13 Sharpe edge) the robustness sweep shows to be a fluke of one configuration (2/5 windows, 0/12 settings).
- An accounting (§5.5, §7) of what "the gate rejected everything" does and does not establish, including the ways our own trial count understates the true multiplicity of the search.
We did not find a profitable strategy. On efficient large-cap equities, over these windows, with price-only signals, we did not expect to, and the backtest-overfitting literature predicts we should not [BacktestOverfitting; AFML]. What we report is that the gate behaves as designed, taking strategies that look excellent and refusing them for concrete, reproducible reasons. Methods whose main demonstration is that they kill their author's best-looking ideas are, in our view, in short supply in this field.
2Related Work
2.1 Backtest overfitting and multiple testing
Finance has known for a long time that data-snooping inflates apparent performance. White's Reality Check [White] and the data-snooping study of Sullivan, Timmermann, and White [STW] gave bootstrap procedures for asking whether the best of many trading rules beats chance. Harvey and Liu [HarveyLiu] argue that with hundreds of factors and strategies tested across the literature, conventional significance thresholds are far too lenient, and that the required hurdle should rise with the number of trials. Bailey, Borwein, López de Prado, and Zhu [BacktestOverfitting] show formally that with enough trials one can construct a strategy with an arbitrarily high in-sample Sharpe and zero true edge, and that in-sample optimization tends to select against out-of-sample performance. The Deflated Sharpe Ratio [DSR] puts this into practice by discounting an observed Sharpe for the number of trials, the length of the track record, and non-normality. López de Prado's Advances in Financial Machine Learning [AFML] gathers these ideas into walk-forward and combinatorially-purged cross-validation, and the Probability of Backtest Overfitting.
Our verifier belongs to this tradition and does not invent a new statistical test. Its trials-adjusted bar (§4.4) is a deliberately transparent, monotonic heuristic in the spirit of the Deflated Sharpe Ratio, and does not compute it. We chose that form so the penalty would be legible and hard to game, and we treat replacing it with a calibrated DSR as future work (§7).
2.2 Walk-forward and out-of-sample discipline
Out-of-sample evaluation is the standard defense: fit or select on one slice of history, then judge on a later slice the strategy never influenced. Its weakness in practice is that a single split is a single draw. A strategy can clear one out-of-sample window by luck, especially after the researcher has, consciously or not, iterated against that window. Stability should therefore be checked across many windows and parameterizations [AFML]. Our robustness sweep (§4.5) does this directly, and in §5 it turns out to decide the outcome.
2.3 Machine- and LLM-proposed strategies
Automated strategy search predates language models. Genetic programming and AutoML for trading are decades old, and both are well-known overfitting hazards because they search a large space. Recent work uses LLMs to propose or code trading strategies from natural-language intuitions. The risk profile is the same, only larger: cheaper proposals mean more trials and so more selection bias, and an LLM that emits runnable code puts a nondeterministic, potentially unsafe author directly in the execution path. ÀYÒ responds by having the proposer emit data instead of code, and by freezing and content-hashing it (§4.1), so that the deployed behavior is fully reproducible even though the author is not.
2.4 Deterministic gates around nondeterministic models
The broader pattern of wrapping a nondeterministic model in a deterministic layer, which supplies a guarantee the model cannot supply alone, recurs across the author's systems. Two examples are a governed memory engine that reconciles and audits what an agent may store [Cortex], and a citation verifier that blocks fabricated quotations in an agentic workflow. The present paper is the empirical, single-domain instance of that pattern; a cross-domain synthesis is left to future work (§6).
3The gate, formally (light)
We frame the system as a pair :
- is a proposer: a nondeterministic map from a prompt to a candidate strategy artifact . In ÀYÒ, is an LLM. Nothing in the guarantees below depends on being trustworthy, correct, or even non-adversarial.
- is a verifier: a deterministic predicate evaluated against a fixed historical dataset . Only artifacts for which accepts may be executed, and execution reads only the frozen artifact.
The design aims for four properties. We state them as design targets and prove none of them; §4 describes the mechanisms and §7 the gaps.
- Deployed behavior is reproducible. An artifact is content-addressed, so identical trading logic yields an identical hash and therefore identical behavior. The executor recomputes the hash before running and refuses a mismatch. Because the gate validates the artifact and leaves the model out of it, a nondeterministic author can still produce perfectly reproducible deployed behavior.
- Acceptance is falsifiable. is a decidable predicate over measured quantities (out-of-sample Sharpe, drawdown, trade count, decay, robustness counts), and no model's judgment enters it. Whether a given artifact passes can be checked by re-running the gate.
- Degradation is fail-safe. Removing leaves a system that still trades its already-accepted artifacts under their frozen rules. Losing the proposer costs improvement and leaves safety intact.
- Search cannot game the gate. The acceptance bar rises with the number of candidates tested and requires stability across windows and parameters, so a search cannot reach acceptance by trying more variations; each extra variation raises the bar it has to clear.
Property 4 is the one that bears on the failure mode of §1, and §5 puts the most stress on it.
4Method
4.1 Strategy artifacts: content-hashed, non-executable
A strategy is a frozen dataclass with an explicit universe, a list of entry rules (all must hold to enter), exit rules (any triggers an exit), a hard stop loss, a take profit, and a maximum holding period. Each rule is a declarative triple (indicator, op, threshold), for example (rsi_14, <, 35), which the executor evaluates against precomputed features. No text from the proposer is ever exec'd.
The artifact's content hash is a SHA-256 over its trading logic, excluding the pass/fail flag and the hash field itself. An artifact is deployable only if the gate marked it passed and its stored hash also still equals a fresh recomputation. That is the mechanical basis for reproducibility (Property 1): the object that trades is the same object that was validated.
4.2 The base backtest gate
The backtester scores every rule-based artifact against a fixed floor: at least 20 trades, Sharpe ≥ 1.0, maximum drawdown ≤ 15%, and win rate ≥ 45%. The floor gives no protection against overfitting, since a strategy can clear it by in-sample luck alone, and in the search of §5.2 it is not used as an in-sample filter either. The in-sample floor verdict is reported for each candidate, but candidates are ranked by in-sample Sharpe among those with at least 15 in-sample trades, whether or not they clear the floor. The floor's drawdown and win-rate limits are enforced on the out-of-sample slice instead (§4.3), and the full-history floor is applied only when an artifact that has already passed out-of-sample is frozen for deployment. The cross-sectional experiments (§5.3 and 5.4) do not use this floor; they are judged against a buy-and-hold benchmark.
4.3 Walk-forward out-of-sample evaluation
We split each symbol's history into an in-sample portion (the first 70%) and an out-of-sample portion (the remaining 30%, prefixed with a 50-bar warm-up so indicators are valid at the boundary but trades fire only inside the out-of-sample window). Candidate selection happens on in-sample data only, and the winner is then judged on the out-of-sample slice it never influenced. Out-of-sample, the winner must make at least 10 trades (fewer than the full floor's 20, because the slice is shorter), reach the trials-adjusted Sharpe bar of §4.4 in place of the floor's fixed 1.0, keep maximum drawdown ≤ 15% and win rate ≥ 45%, and end with a positive return. It must also not have decayed: we require the out-of-sample Sharpe to retain at least half of the in-sample Sharpe. A strategy whose performance halves out-of-sample is treated as fragile even if it clears the raw threshold.
4.4 The trials-adjusted acceptance bar
The main defense against overfitting is a required out-of-sample Sharpe that rises with the number of candidates tested:
required_oos_sharpe(N) = 1.0 + 0.15 * log2(N)
Testing 1 candidate requires an out-of-sample Sharpe of 1.0; 6 candidates requires 1.39; 64 candidates requires 1.90. The functional form is a transparent stand-in for the Deflated Sharpe Ratio's selection-bias correction [DSR]. It is monotonic in the number of trials, cheap to reason about, and impossible to satisfy by enumerating more variants. It is still a heuristic, and §7 discusses two weaknesses: the constants are not calibrated to a null distribution, and counts only the candidates in a single scripted search, which undercounts the true researcher degrees of freedom.
4.5 Robustness sweeps
For cross-sectional strategies we add two stress tests, because a single out-of-sample split is a single draw (§2.2):
- The rolling-window test partitions the post-warm-up period into 5 sequential windows and asks, in each, whether the strategy beats a buy-and-hold benchmark on a Sharpe basis. A real edge should appear in most of the windows.
- The parameter-sensitivity test varies the strategy's free parameters (here, momentum look-back over {126, 189, 252} days and portfolio breadth
top_kover {8, 12, 16, 20}, 12 combinations) and counts how many settings beat the benchmark. A real edge should hold across the grid; a single lucky point does not count.
A strategy is credited with an edge only if it beats the benchmark in most windows and most settings. Scattered signs point to a fluke.
5Experiments
5.1 Data and universe
All results use real daily bars retrieved from Alpaca on 2026-07-20. The single-strategy walk-forward (§5.2) uses a pre-declared universe of 32 large-cap U.S. equities over roughly four years. The cross-sectional experiments (§5.3 and 5.4) use a broader pre-declared universe of 83 large-cap equities; after alignment and warm-up this yields a panel of 959 trading days spanning 2022-09-20 → 2026-07-17. Universes and features were fixed before results were inspected. Both universes are lists of today's large, liquid names run backward over the period, so they carry survivorship bias: companies that shrank, were acquired, or were delisted during the window are absent, which tends to flatter long-only backtests. No language model generated the strategies tested in these runs. The six rule-based candidates of §5.2 are written into the experiment script (tests/find_strategy.py), and the momentum book of §5.3 and 5.4 is a textbook specification, so these experiments exercise the verifier and leave the proposer out. The run logs behind every number below are in the reproducibility appendix (§A), which also carries a caveat about the retrieval window.
5.2 The single-strategy overfit trap
Six rule-based "trend dip-buy" strategies were ranked on in-sample data. All looked strong in-sample; several posted in-sample Sharpes above 2 and in-sample returns in the hundreds of percent. The in-sample winner was trend_dip_30 (buy oversold names in an uptrend), with an in-sample Sharpe of 3.07 over 46 trades, a 57% win rate, and a 23.5% maximum drawdown. None of the six cleared the §4.2 floor in-sample (in-sample drawdowns ran from 23.5% to 48.8%, all above the 15% limit). As §4.2 explains, the floor was reported but not used to filter the ranking, so the winner went to the out-of-sample test on its in-sample Sharpe alone.
Judged out-of-sample against a bar of 1.39 (6 trials), its Sharpe changed sign:
IS Sharpe 3.07 -> OOS Sharpe -4.90 (need >= 1.39)
OOS: 14 trades, win 36%, maxDD 30.4%, return -25.5%, decay_ok = False
VERDICT: REJECTED — not deployed
An in-sample Sharpe of 3.07 that becomes −4.90 out-of-sample is the canonical shape of an overfit backtest. The selection process found a configuration tuned to the noise of the in-sample period, and that tuning was actively harmful once the noise changed. The gate was built to catch this case, and here it rejected the strategy.
5.3 A single out-of-sample split can still fool you
Cross-sectional momentum (hold the strongest recent performers, rebalance periodically) is one of the most documented anomalies in the literature, which makes it a fair test of whether the gate is merely pessimistic. We evaluated it in-sample and out-of-sample, with and without a market-regime filter, against a buy-and-hold benchmark of the same universe:
| Configuration | Split | Strategy Sharpe | Benchmark Sharpe | Edge vs B&H |
|---|---|---|---|---|
| With regime filter | in-sample | 0.40 | 0.91 | −0.51 |
| With regime filter | out-of-sample | 0.41 | 1.08 | −0.67 |
| No regime filter | in-sample | 0.58 | 0.91 | −0.33 |
| No regime filter | out-of-sample | 1.21 | 1.08 | +0.13 |
Three of the four cells lose to buying the universe. The fourth, momentum with no regime filter out-of-sample, beats the benchmark by a Sharpe edge of +0.13. Taken alone, that positive out-of-sample number is the kind of result a hopeful researcher publishes as an anomaly that "works out-of-sample", and a naive out-of-sample check would pass it.
5.4 The robustness sweep exposes the fluke
The +0.13 does not survive the tests of §4.5. Across 5 sequential rolling windows, momentum beat buy-and-hold in only 2 of 5, and the wins were marginal while the losses were large:
window 1 (2023-10-03 -> 2024-04-23): edge -0.40 lost
window 2 (2024-04-24 -> 2024-11-11): edge -1.76 lost
window 3 (2024-11-12 -> 2025-06-05): edge +0.05 BEAT
window 4 (2025-06-06 -> 2025-12-24): edge -1.17 lost
window 5 (2025-12-26 -> 2026-07-17): edge +0.13 BEAT
-> beat buy&hold in 2/5 windows
Across the 12-point parameter grid, the edge was positive in 0 of 12 settings. Every combination of look-back and breadth lost to buy-and-hold, by margins from −0.22 to −0.65:
lookback in {126,189,252} x top_k in {8,12,16,20}
edges ranged -0.22 to -0.65 ; positive in 0/12 settings
The +0.13 of §5.3 came from one lucky configuration on one split, and there was no edge behind it. Stability across windows and parameters is what tells the two apart, which is why an out-of-sample split, though necessary, is not sufficient on its own.
5.5 Summary, and what it does and does not establish
The gate rejected every candidate it was given: the single-strategy winner on sign inversion and decay (it also missed the out-of-sample drawdown, win-rate, and return conditions), and cross-sectional momentum on robustness. On efficient large-cap equities, over these windows, with price-only signals, out-of-sample validation predicts this outcome [BacktestOverfitting; AFML].
The results establish that the gate's mechanisms behave as designed on real data. The gate quarantines the proposer behind a frozen, hash-checked artifact; it turns a spectacular in-sample Sharpe (3.07) into a concrete rejection (−4.90 out-of-sample); and it correctly reclassifies a plausible out-of-sample "edge" (+0.13) as a fluke once stability is required (2/5, 0/12). The negative results can be reproduced from the included logs.
They do not establish that these strategies have no edge in any universe, timeframe, or cost regime, or that the specific constants in the bar are optimal. Above all, they do not show that the gate has been stress-tested against a strategy that deserves to pass. A rejecter is only fully validated once it also accepts something true. We have shown that the gate says no to mirages. We have not shown that it says yes to a real edge, because we have not yet found one to offer it.
6Discussion: the pattern generalizes
The device at the center of this paper is to quarantine a nondeterministic proposer behind a deterministic, falsifiable verifier and to validate the frozen artifact instead of the model. Nothing about it is specific to trading. The same shape appears wherever an LLM's fluency outruns its trustworthiness: a governed memory engine that admits a fact to long-term storage only after deterministic reconciliation and records why [Cortex]; a citation verifier that lets an agent's answer through only if each quoted span is found verbatim in the cited source. In each case the LLM proposes and a deterministic gate disposes, and the engineering value is concentrated in the gate. Trading is the sharpest test of the idea, because the cost of a false accept is measured directly in money and the environment is adversarial by default. A cross-domain formalization of the proposer/verifier pattern and its invariants is planned as follow-on work, and this paper is its trading instance.
7Limitations and future work
- There is no accepted strategy yet. The gate has been shown to reject, and has not yet been shown to accept correctly. The most important next experiment is to construct or discover a strategy with a real edge (likely from signals not contained in price alone) and confirm that the gate passes it. Until then, "the gate works" means "the gate reliably refuses mirages."
- The bar is a heuristic and has not been calibrated as a test.
1.0 + 0.15·log2(N)is monotonic and legible, but its constants are not derived from a null distribution. Replacing it with a properly Deflated Sharpe Ratio [DSR] and reporting the Probability of Backtest Overfitting [AFML] would put the acceptance decision on firmer statistical ground. - Our trial count undercounts researcher degrees of freedom. counts candidates in one scripted search. The true multiplicity across the whole research program (every universe, feature, and threshold the author has ever considered) is far larger, so the effective bar is if anything too lenient. We regard this as the hardest problem in the area and do not claim to have solved it.
- All results come from one retrieval window of daily U.S. equity bars. Combinatorially-purged cross-validation, multiple snapshots, other asset classes and frequencies, and an explicit transaction-cost, borrow, and capacity model would strengthen every claim.
- The reproducibility window moves. The data fetch is anchored to the retrieval date, so re-running on a later date shifts the window. We treat the committed run logs and a pinned data snapshot as the frozen reference (§A); making the fetch fully date-pinned is a small, planned change.
8Conclusion
Strategy ideas are plentiful in systematic trading. The scarce resource is not being fooled by them, and a fast proposer makes it scarcer. ÀYÒ puts its engineering where that scarcity is. It treats an LLM as a cheap, untrusted proposer of frozen data artifacts, and spends its effort on a deterministic gate that validates the artifact out-of-sample, raises its bar with the number of trials, and requires robustness across windows and parameters. On real markets the gate turned an in-sample Sharpe of 3.07 into a −4.90 rejection, showed a +0.13 out-of-sample "edge" to be a 0-of-12 fluke, and deployed nothing, which is the expected outcome on efficient markets. The paper contributes no alpha. It contributes something the field has less of: a validation layer that reliably refuses to fool itself, and a pattern for holding nondeterministic proposers to deterministic guarantees.
References
[BacktestOverfitting] D. Bailey, J. Borwein, M. López de Prado, Q. Zhu. "Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance." Notices of the American Mathematical Society, 61(5), 2014.
[DSR] D. Bailey, M. López de Prado. "The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting, and Non-Normality." Journal of Portfolio Management, 40(5), 2014.
[HarveyLiu] C. Harvey, Y. Liu. "Backtesting." Journal of Portfolio Management, 42(1), 2015. See also C. Harvey, Y. Liu, H. Zhu, "…and the Cross-Section of Expected Returns," Review of Financial Studies, 29(1), 2016.
[White] H. White. "A Reality Check for Data Snooping." Econometrica, 68(5), 2000.
[STW] R. Sullivan, A. Timmermann, H. White. "Data-Snooping, Technical Trading Rule Performance, and the Bootstrap." Journal of Finance, 54(5), 1999.
[AFML] M. López de Prado. Advances in Financial Machine Learning. Wiley, 2018.
[Cortex] S. Olagoke. "Cortex: A Governed Long-Term Memory Engine for LLM Agents." System report, 2026.
Appendix A. Reproducibility
The three experiments are reproducible from the ÀYÒ repository:
python -m tests.find_strategy # §5.2 single-strategy walk-forward
python -m tests.research_xsec # §5.3 cross-sectional momentum IS/OOS
python -m tests.robustness_xsec # §5.4 rolling-window + parameter sweep
Verbatim stdout from the runs backing this paper is committed under data/runs/
(find_strategy_run.log, research_xsec_run.log, robustness_run.log). Bars are
real Alpaca daily data retrieved 2026-07-20; the cross-sectional panel spans
2022-09-20 → 2026-07-17 (959 trading days, 83 symbols). Because get_bars/get_price_panel
anchor the fetch to the retrieval date, exact reproduction is against the committed
logs and the pinned snapshot rather than a fresh fetch on a later date.