signal-classification Checked

from sklearn.calibration import CalibratedClassifierCV

calibrated = CalibratedClassifierCV(base_model, cv=5, method="isotonic")
calibrated.fit(X_train, y_train)
probs = calibrated.predict_proba(X_test)[:, 1]

Isotonic calibration works better than Platt scaling for tree models.

Walk-Forward Validation

**This is the single most important concept in trading ML.** Standard cross-validation randomly shuffles data, which creates lookahead bias. Walk-forward validation respects time ordering.

### How It Works

Window 1: [===TRAIN===][GAP][=TEST=]
Window 2:    [===TRAIN===][GAP][=TEST=]
Window 3:       [===TRAIN===][GAP][=TEST=]
Window 4:          [===TRAIN===][GAP][=TEST=]

Each window:

1. Train on past N bars

2. Skip a gap (embargo) equal to the forward return horizon

3. Predict on next M bars

4. Record out-of-sample predictions

5. Slide forward and repeat

### Typical Parameters

| Parameter | Value | Rationale |

|-----------|-------|-----------|

| Train window | 30 days (720 hourly bars) | Enough data to learn, recent enough to be relevant |

| Test window | 7 days (168 hourly bars) | Enough predictions for statistical significance |

| Step size | 1 day (24 bars) | Overlap test windows for more data points |

| Gap (embargo) | Same as forward horizon | Prevents label leakage |

### Walk-Forward Implementation

from typing import Iterator

def walk_forward_splits(
    n_samples: int,
    train_size: int = 720,
    test_size: int = 168,
    step_size: int = 24,
    gap: int = 24,
) -> Iterator[tuple[np.ndarray, np.ndarray]]:
    """Generate walk-forward train/test index splits.

    Args:
        n_samples: Total number of samples.
        train_size: Number of training samples per window.
        test_size: Number of test samples per window.
        step_size: Step between successive windows.
        gap: Gap between train end and test start.

    Yields:
        Tuples of (train_indices, test_indices).
    """
    start = 0
    while start + train_size + gap + test_size <= n_samples:
        train_idx = np.arange(start, start + train_size)
        test_start = start + train_size + gap
        test_idx = np.arange(test_start, test_start + test_size)
        yield train_idx, test_idx
        start += step_size

See references/validation_methods.md for purged CV, CPCV, and evaluation metrics.

Model Training Pipeline

### Full Pipeline Overview

1. **Feature engineering** — compute technical indicators, on-chain metrics, volume features (see feature-engineering skill)

2. **Label creation** — forward returns with threshold, drop neutral zone

3. **Walk-forward split** — time-ordered train/test windows with gap

4. **Train model** — XGBoost or LightGBM on each training window

5. **Predict on test** — generate out-of-sample probability predictions

6. **Aggregate predictions** — concatenate all out-of-sample results

7. **Evaluate** — accuracy, precision, recall, F1, AUC, profit factor

### Quick Training Example

from xgboost import XGBClassifier

model = XGBClassifier(
    n_estimators=200,
    max_depth=4,
    learning_rate=0.05,
    subsample=0.8,
    colsample_bytree=0.8,
    eval_metric="logloss",
    use_label_encoder=False,
    random_state=42,
)

model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
    verbose=False,
)

probabilities = model.predict_proba(X_test)[:, 1]

See references/model_guide.md for parameter recommendations and tuning.

SHAP Feature Importance

SHAP (SHapley Additive exPlanations) provides the gold standard for understanding model predictions.

### Global Feature Importance

Which features matter most across all predictions:

import shap

explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)

# Summary plot (top 15 features)
shap.summary_plot(shap_values, X_test, max_display=15)

### Local Explanations

Why a specific prediction was made:

# Explain a single prediction
shap.force_plot(explainer.expected_value, shap_values[0], X_test.iloc[0])

### Temporal Feature Importance

Track how feature importance drifts over walk-forward windows. If a feature's importance drops significantly, the market regime may have shifted.

Threshold Optimization

The default 0.5 probability threshold is almost never optimal for trading.

### Why Not 0.5?

• Class imbalance: if 60% of labels are "up", a 0.5 threshold is too aggressive
• Trading costs: marginal signals (0.51 probability) rarely cover transaction costs
• Asymmetric payoffs: precision matters more than recall for trading

### Optimize for Profit Factor

def optimize_threshold(
    probabilities: np.ndarray,
    returns: np.ndarray,
    thresholds: np.ndarray | None = None,
) -> tuple[float, float]:
    """Find threshold that maximizes profit factor.

    Args:
        probabilities: Model predicted probabilities.
        returns: Actual forward returns.
        thresholds: Thresholds to search over.

    Returns:
        Tuple of (best_threshold, best_profit_factor).
    """
    if thresholds is None:
        thresholds = np.arange(0.50, 0.85, 0.01)
    best_threshold, best_pf = 0.5, 0.0
    for t in