Backtest and gate¶
Rolling-origin evaluation without lookahead, its metrics, the holdout gate that decides GO or NO_GO, and the expectation a promotion records.
rolling_origin_eval(
panel: DataFrame,
model: _ModelProtocol,
min_train_rows: int = 20,
stride: int = 10,
flat_stake: float = 1.0,
taker_bps: float | None = None,
costs: CostModel | None = None,
policy: TradePolicy | None = None,
*,
embargo_days: int | None = None,
) -> RollingOriginResult
Walk-forward evaluation on a panel dataset.
panel columns required
market_id, decision_date (str YYYY-MM-DD), outcome_label, winning_label, market_price, + feature_* columns
For each eval step
- Train on the rows decided before
eval_datewhose label was already known when the eval decision was taken (see below) - Predict on rows at eval_date
- Decide the trade of each (market_id, decision_date) under
policy - Compute PnL using settle_position
market_price is the label's YES price at the row's decision_date (the plugin
contract, see MarketPlugin.build_training_row). A decision uses only its own row's
price and forecast, so nothing later than the decision date reaches it.
Labels are known later than decisions. A decision on day D can ask about D+1 (the
weather_tmax previous_evening horizon), so the row decided on the day before an
eval date carries the eval day's own outcome. Training on it is lookahead. A row therefore
trains a fold only when its label was known at that fold's decision:
market_close_ts(the plugin writes it on every row: the instant the market ends, which is when the day it asks about is over) must be at or before the fold's decision time, the earliestdecision_tsof the fold's rows (the start ofeval_dateUTC when the panel has nodecision_ts);- a row with no usable
market_close_ts, or a panel without the column, is assumed to resolve the day after its decision, so it trains a fold only from two days later (decision_date + 1 day < eval_date). If your labels arrive later than that, put the time the label became known inmarket_close_ts.
The rule trusts market_close_ts to be no earlier than the moment the label can be computed.
The weather plugin takes the earlier of the market's own close and the end of its target day,
so a market that closes at noon on its target day counts as known from noon although the
day's maximum is final only at midnight: that matters to a fold that decides later that same
day (never to one that decides in the morning, as the sample's does).
A fold therefore starts one decision day later. Backtests run before this rule (release 1.0.2 and earlier) carried a day of lookahead.
A longer hold-back, on request. embargo_days=N adds a second condition to the rule
above: a row trains a fold only if its decision_date is at least N days before the
fold's. Use it when you want a margin the labels do not give you (a model that must not see
the last week, say). The two conditions are AND-ed: a row must have its label known and be at
least N days old, so the embargo can only take rows out of a fold, never put one in. It
cannot make a label that is not yet known trainable, whatever N, and 0 or 1 change
nothing (a fold already trains only on rows decided on an earlier day). None, the default,
is the rule above exactly: the results of release 1.0.3 are reproduced to the last bit.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
panel
|
DataFrame
|
Training/eval panel (see column contract above). |
required |
model
|
_ModelProtocol
|
Anything with sklearn-style |
required |
min_train_rows
|
int
|
Skip an eval date until this many training rows exist. |
20
|
stride
|
int
|
Evaluate every Nth distinct decision_date. |
10
|
flat_stake
|
float
|
Notional USDC risked per trade. |
1.0
|
taker_bps
|
float | None
|
Legacy fee-only knob. Mutually exclusive with |
None
|
costs
|
CostModel | None
|
Full |
None
|
policy
|
TradePolicy | None
|
The decision rule. Omit it for the historical backtest: YES on the bin
with the highest predicted probability, every market, no threshold (the
numbers published before policies existed reproduce). Pass
|
None
|
embargo_days
|
int | None
|
Keyword-only. |
None
|
Returns:
| Type | Description |
|---|---|
RollingOriginResult
|
RollingOriginResult with all trade records and step metadata. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If both |
TypeError
|
If |
dataclass
¶
RollingOriginResult(
trades: DataFrame, steps: list[dict[str, Any]] = list()
)
compute_metrics(trades: DataFrame) -> BacktestMetrics
Compute backtest metrics from a trades DataFrame.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
trades
|
DataFrame
|
DataFrame with column |
required |
Returns:
| Type | Description |
|---|---|
BacktestMetrics
|
BacktestMetrics with aggregated stats. All zeros if trades is empty. |
dataclass
¶
BacktestMetrics(
num_trades: int,
total_pnl: float,
hit_rate: float,
avg_pnl_per_trade: float,
avg_edge: float,
)
dataclass
¶
HoldoutGateResult(
decision: str,
segment_results: list[SegmentGateResult],
aggregate_pnl: float,
aggregate_trades: int,
)
classmethod
¶
evaluate(
trades: DataFrame,
required_segments: list[str] | None = None,
min_trades_per_segment: int = 40,
min_pnl_per_segment: float = 0.0,
) -> HoldoutGateResult
Evaluate the holdout gate.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
trades
|
DataFrame
|
Backtest trade log (as produced by |
required |
required_segments
|
list[str] | None
|
Segments that must each pass. When |
None
|
min_trades_per_segment
|
int
|
Minimum trades a segment needs to pass. |
40
|
min_pnl_per_segment
|
float
|
Minimum total PnL a segment needs to pass. |
0.0
|
dataclass
¶
SegmentGateResult(
segment: str,
num_trades: int,
total_pnl: float,
passes: bool,
reason: str,
)
dataclass
¶
BacktestExpectation(
n_trades: int,
flat_stake: float,
mean_pnl_per_stake: float,
std_pnl_per_stake: float,
brier_score: float | None = None,
brier_std: float | None = None,
by_direction: Mapping[str, GroupExpectation]
| None = None,
policy: TradePolicy | None = None,
by_segment: Mapping[str, SegmentExpectation | None]
| None = None,
)
Mean, spread and size of the backtest trades a champion was promoted on.
brier_score / brier_std are the mean and standard deviation of the
per-trade squared forecast error. They are None when the backtest trades did
not carry both predicted_prob and label_occurred.
by_direction holds the same numbers for the YES trades and for the NO trades on
their own: a NO trade is a buy of a different token, picked where the model is
bearish, and its PnL and forecast error are not draws from the YES distribution.
None means the backtest did not record a split (a champion published before
0.9.1, or a frame without a direction column): the pooled numbers then describe
the historical backtest, which only bought YES. A side the backtest traded fewer
than MIN_GROUP_TRADES times is absent from the mapping, unless it traded nothing
else (then it is the pooled expectation); an empty mapping means neither side traded
enough to be promised.
policy is the decision rule the backtest ran (TradePolicy), recorded so live
trading can use the same threshold. None when not recorded.
by_segment holds the same numbers for each segment on its own, each with its own
split by side. A segment the backtest traded fewer than MIN_GROUP_TRADES times is
present with None: recorded as absent, so "this segment could not be promised
anything" stays distinguishable from "nobody looked". None for the whole mapping
means no per-segment record was made (a champion published before it existed).
An expectation is only worth recording if a live run can be measured against it, so the constructor refuses one that cannot: every number finite, a spread that is not negative, a stake above zero, a Brier score in [0, 1] and at least two trades (one trade has no spread, none divides by zero). NaN compares false with everything, so it would not fail a test, it would silently switch the check off. The mean may be zero or negative; that is a finding the watchdog reports, not a malformed record.
class-attribute
instance-attribute
¶
by_direction: Mapping[str, GroupExpectation] | None = None
class-attribute
instance-attribute
¶
by_segment: (
Mapping[str, SegmentExpectation | None] | None
) = None
classmethod
¶
from_trades(
trades: DataFrame,
*,
flat_stake: float = 1.0,
policy: TradePolicy | None = None,
min_group_trades: int = MIN_GROUP_TRADES,
) -> BacktestExpectation
Summarise a rolling_origin_eval(...).trades frame.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
trades
|
DataFrame
|
Needs |
required |
flat_stake
|
float
|
The notional each backtest trade risked — the
|
1.0
|
policy
|
TradePolicy | None
|
The trade policy the backtest ran, to record with the numbers. |
None
|
min_group_trades
|
int
|
Fewest trades a side needs to be recorded (at least 2). |
MIN_GROUP_TRADES
|
Raises:
| Type | Description |
|---|---|
ValueError
|
On fewer than two trades (no spread to measure), a
non-finite PnL, a non-positive stake, forecasts outside [0, 1] or a
|
classmethod
¶
from_dict(data: Mapping[str, Any]) -> BacktestExpectation
Restore a record; raises ValueError (KeyError if a key is absent) if unusable.
A record written before 0.9.1 has neither by_direction nor policy; both load
as None. One that has them but cannot be used raises: it must never read as
"not recorded" and quietly weaken the watchdog.
dataclass
¶
GroupExpectation(
n_trades: int,
mean_pnl_per_stake: float,
std_pnl_per_stake: float,
brier_score: float | None = None,
brier_std: float | None = None,
)
Mean, spread and size of one group of backtest trades: a side, or a segment.
The same numbers BacktestExpectation keeps for the whole backtest, per unit
stake. A group is held to the same rules as the whole: every number finite, a
spread that is not negative, at least two trades, a Brier score in [0, 1].
classmethod
¶
from_dict(data: Mapping[str, Any]) -> GroupExpectation
Restore a record; raises ValueError (KeyError if a key is absent) if unusable.
dataclass
¶
SegmentExpectation(
n_trades: int,
mean_pnl_per_stake: float,
std_pnl_per_stake: float,
brier_score: float | None = None,
brier_std: float | None = None,
by_direction: Mapping[str, GroupExpectation]
| None = None,
)
Bases: GroupExpectation
One segment's numbers, with that segment's own split by side.
A segment trades YES and NO in its own proportion, so a live segment is held to its own per-side means, weighted by the live mix, exactly as the whole book is.
purged_kfold(
samples: NDArray[int_] | ndarray,
n_splits: int = 5,
embargo: int = 0,
) -> Iterator[tuple[NDArray[np.int_], NDArray[np.int_]]]
Yield (train_idx, test_idx) for purged k-fold CV.
The samples are split into n_splits contiguous test blocks (preserving
time order). For each, the training set is every other sample minus an
embargo band immediately before and after the test block.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
samples
|
NDArray[int_] | ndarray
|
1-D array whose length defines the sample count (values unused;
pass |
required |
n_splits
|
int
|
Number of folds (>= 2). |
5
|
embargo
|
int
|
Number of samples to purge on each side of the test block. |
0
|
Yields:
| Type | Description |
|---|---|
tuple[NDArray[int_], NDArray[int_]]
|
(train_idx, test_idx) as positional-index arrays into |
embargoed_split(
samples: NDArray[int_] | ndarray,
n_splits: int = 5,
embargo: int = 0,
) -> Iterator[tuple[NDArray[np.int_], NDArray[np.int_]]]
Yield (train_idx, test_idx) for walk-forward CV with an embargo gap.
Expanding-origin: fold f's training set is everything up to the start of
its test block minus an embargo gap, and the test block is the next
contiguous slice. Train is always strictly earlier than test.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
samples
|
NDArray[int_] | ndarray
|
1-D array whose length defines the sample count. |
required |
n_splits
|
int
|
Number of forward test blocks (>= 2). |
5
|
embargo
|
int
|
Gap (in samples) between the end of train and the start of test. |
0
|
Yields:
| Type | Description |
|---|---|
NDArray[int_]
|
(train_idx, test_idx) positional-index arrays. Early folds may yield an |
NDArray[int_]
|
empty train_idx when the embargo consumes the available history. |