walk-forward-validation Checked

Remove training samples whose labels overlap with the test set's time range. If a label is computed as the 24h forward return starting at time t, any training sample where t + 24h extends into the test period must be purged.

def purge_train_indices(
    train_idx: list[int],
    test_start: int,
    label_horizon: int,
    timestamps: list[int],
) -> list[int]:
    """Remove train samples whose label windows overlap test period."""
    test_start_time = timestamps[test_start]
    return [
        i for i in train_idx
        if timestamps[i] + label_horizon < test_start_time
    ]

### Embargo

Add a buffer gap between the end of training and start of testing to account for serial correlation that purging alone does not eliminate.

[===TRAIN===][--EMBARGO--][=TEST=]

Typical embargo sizes:

• **1-minute bars**: 60–240 bars (1–4 hours)
• **5-minute bars**: 12–48 bars (1–4 hours)
• **Hourly bars**: 6–24 bars (6–24 hours)
• **Daily bars**: 2–5 bars (2–5 days)
• **Crypto rule of thumb**: Embargo >= 2x the label computation horizon

Combinatorial Purged Cross-Validation (CPCV)

CPCV (Lopez de Prado, 2018) generates all possible train/test combinations from N groups while maintaining temporal ordering. This produces far more test paths than standard walk-forward, enabling statistical tests for overfitting.

**Key properties:**

• Splits data into N contiguous groups
• For each combination of k test groups, the remaining N-k groups form the training set
• Applies purging and embargo at each train/test boundary
• Produces C(N, k) backtest paths (e.g., N=6, k=2 gives 15 paths)

See references/methodology.md for the full CPCV algorithm and formulas.

Overfit Detection

### Deflated Sharpe Ratio (DSR)

The observed Sharpe ratio must be adjusted for:

• Number of strategies tested (multiple testing)
• Non-normality of returns (skewness, kurtosis)
• Length of the backtest

import numpy as np
from scipy.stats import norm

def deflated_sharpe_ratio(
    observed_sr: float,
    num_trials: int,
    backtest_length: int,
    skewness: float = 0.0,
    kurtosis: float = 3.0,
) -> float:
    """Compute the probability that observed SR > 0 after deflation.

    Args:
        observed_sr: Annualized Sharpe ratio of the selected strategy.
        num_trials: Number of strategies tested (including discarded ones).
        backtest_length: Number of return observations.
        skewness: Skewness of returns.
        kurtosis: Excess kurtosis of returns.

    Returns:
        p-value (probability SR is genuinely > 0).
    """
    sr_std = np.sqrt(
        (1 - skewness * observed_sr + (kurtosis - 1) / 4 * observed_sr**2)
        / (backtest_length - 1)
    )
    # Expected max SR under null (Euler-Mascheroni approximation)
    euler_mascheroni = 0.5772156649
    expected_max_sr = norm.ppf(1 - 1 / num_trials) * (
        1 - euler_mascheroni
    ) + euler_mascheroni * norm.ppf(1 - 1 / (num_trials * np.e))
    dsr = norm.cdf((observed_sr - expected_max_sr) / sr_std)
    return dsr

A DSR below 0.95 suggests the observed performance is likely due to overfitting across the trials tested.

### Probability of Backtest Overfitting (PBO)

PBO uses CPCV to measure the fraction of backtest paths where the in-sample optimal strategy underperforms the median out-of-sample. A PBO above 0.50 indicates more-likely-than-not overfitting.

See references/overfit_detection.md for complete derivations and implementation details.

Crypto-Specific Considerations

1. **Shorter windows**: Crypto regimes change faster than equities. A 90-day rolling window may be more appropriate than 252 days.

2. **24/7 markets**: No weekends or holidays to account for, but funding rate resets (every 8h on perps) create microstructure effects.

3. **Survivorship bias**: Many tokens delist. Validation must include delisted tokens or at minimum acknowledge this limitation.

4. **Liquidity regime shifts**: A token's liquidity profile can change dramatically (new CEX listing, liquidity mining end). Train/test splits should ideally not straddle major liquidity events.

5. **Data availability**: Many tokens have < 1 year of data. Expanding windows with small min_train_size may be necessary.

Practical Window Sizes for Crypto

| Strategy Timeframe | Train Window | Test Window | Embargo |

|---|---|---|---|

| Scalping (1-5min) | 3-7 days | 1 day | 2-4 hours |

| Intraday (15min-1h) | 14-30 days | 3-7 days | 12-24 hours |

| Swing (4h-daily) | 30-90 days | 7-14 days | 2-5 days |

| Position (daily-weekly) | 90-180 days | 30 days | 5-10 days |

Quick Start

from walk_forward import WalkForwardValidator, WalkForwardConfig

config = WalkForwardConfig(
    train_size=90,
    test_size=14,
    step_size=14,
    window_type="rolling",
    embargo_size=3,
    purge_horizon=1,
)

validator = WalkForwardValidator(config)
for fold in validator.split(price_data):
    model.fit(fold.train_X, fold.train_y)
    predictions = model.predict(fold.test_X)
    fold.record_performance(predictions, fold.test_y)

results = validator.aggregate_results()
print(f"OOS Sharpe: {results.oos_sharpe:.3f}")
print(f"Train/Test Sharpe ratio: {results.sharpe_ratio_ratio:.2f}")

Files

### References

• references/methodology.md — Walk-forward theory, window types, purging, embargo, CPCV algor