Evaluation

Core evaluation orchestration for the DNA segmentation benchmark.

The public entry point benchmark_from_arrays() (built on the internal per-sequence _benchmark_gt_vs_pred_single()) compares ground-truth nucleotide-level annotations with predictions and dispatches to the per-metric modules:

• INDEL — indel_metrics

• REGION_DISCOVERY / BOUNDARY_EXACTNESS — section_metrics

• NUCLEOTIDE_CLASSIFICATION — local confusion counts

• PHASE_DRIFT — frame_shift

• STRUCTURAL_COHERENCE — chain_comparison, structure, splice_sites

• DIAGNOSTIC_DEPTH — structural_summary

The state-transition diagnostics (transition_failures / false_transitions) are opt-in via EvalMetrics.STATE_TRANSITIONS. They remain in the default metric set (_DEFAULT_METRICS) so the transition-matrix plots that frame every other metric still render by default; pass an explicit metrics list without STATE_TRANSITIONS to skip the pass. Every metric group is opt-in via metrics.

Input preprocessing (intron inference, mask splitting) lives in preprocessing; cross-sequence accumulation and reduction live in accumulators. All functions accept a LabelConfig that maps integer tokens to names and declares semantic roles (background, coding, …).

class gene_calling_benchmark.eval.evaluate_predictors.StreamingBenchmark(label_config, metrics=None, infer_introns=False)

Bases: object

Incrementally aggregate per-sequence benchmarks without materialising them.

Wraps a single BenchmarkAccumulator. Feed paired GT/pred arrays one at a time with add() — each array is reduced to a per-sequence fragment, accumulated, and may then be discarded by the caller — and call result() once to obtain the aggregated summary. This lets callers build label arrays lazily (one transcript pair at a time) instead of holding every array in memory at once.

benchmark_from_arrays() (aggregate mode) is implemented on top of this class, so streaming and list-based aggregation return identical numbers for the same sequence of pairs.

Parameters:

• label_config (LabelConfig)

• metrics (list[EvalMetrics] | None)

• infer_introns (bool)

property count: int

Number of sequences added so far.

add(gt_labels, pred_labels, mask_labels=None)

Accumulate one GT/pred pair (same per-sequence path as the list API).

Atomic: all fragments for the sequence are computed before any is committed, so a mid-sequence failure leaves the accumulator untouched (nothing partially added) and can be safely isolated by the caller.

Return type:

None

Parameters:

• gt_labels (ndarray)

• pred_labels (ndarray)

• mask_labels (ndarray | None)

result()

Return the aggregated, metadata-annotated summary of all added pairs.

Return type:

dict

gene_calling_benchmark.eval.evaluate_predictors.benchmark_from_arrays(gt_labels, pred_labels, label_config, metrics=None, return_individual_results=False, mask_labels=None, infer_introns=False)

Benchmark paired GT/prediction label-array lists (the array entry point).

Runs the internal per-sequence _benchmark_gt_vs_pred_single() over each pair and, by default, aggregates the fragments into precision/recall and distribution statistics.

Per-sequence isolation: a pair that raises during evaluation is logged and skipped so a single pathological sequence never aborts the whole batch — important when the benchmark runs inside a training loop. In aggregate mode the failing pair is simply omitted; in return_individual_results mode its slot is None (indices stay aligned with the inputs).

Parameters:

• gt_labels (list[ndarray]) – Equally-sized lists of 1-D integer token arrays.

• pred_labels (list[ndarray]) – Equally-sized lists of 1-D integer token arrays.

• label_config (LabelConfig) – Token-to-name mapping and semantic roles.

• metrics (list[EvalMetrics] | None) – Metric groups to compute. Defaults to _DEFAULT_METRICS (REGION_DISCOVERY, BOUNDARY_EXACTNESS, NUCLEOTIDE_CLASSIFICATION, STATE_TRANSITIONS).

• return_individual_results (bool) – If True, return per-sequence results as a list instead of aggregating.

• mask_labels (list[ndarray] | None) – Optional boolean masks (True = exclude). Must match length of GT.

• infer_introns (bool) – If True, background gaps between adjacent coding segments are relabelled as introns before each sequence is evaluated.

Returns:

Aggregated (default) or per-sequence results. In return_individual_results mode the list may contain None for any sequence whose evaluation failed.

Return type:

dict | list[dict]