Labels And Metrics

Label semantics for the DNA segmentation benchmark.

This module defines the public LabelConfig model and the annotation mode/scope enums used by the array-based benchmark core.

Stage 1 introduces two explicit annotation modes:

• EXON_INTRON — one exonic label, optional intron and splice-site labels

• UTR_CDS_INTRON — distinct 5' UTR, CDS, and 3' UTR labels, plus optional intron and splice-site labels

class gene_calling_benchmark.label_definition.AnnotationMode(*values)

Bases: str, Enum

Supported annotation semantics for array-based benchmarking.

class gene_calling_benchmark.label_definition.BenchmarkScope(*values)

Bases: str, Enum

Metric scopes available in Stage 1 array benchmarking.

class gene_calling_benchmark.label_definition.LabelConfig(**data)

Bases: BaseModel

Complete label definition for one benchmarking run.

Required fields

annotation_modeAnnotationMode

Declares whether arrays represent a single exon-like class or a transcript-anatomy label set with explicit UTR/CDS semantics.

background_labelint

Token for non-coding / intergenic / background regions.

Mode-specific required fields

EXON_INTRON

exon_label

UTR_CDS_INTRON

cds_label, five_prime_utr_label, three_prime_utr_label

Optional semantic roles

intron_labelint | None

Token for intronic nucleotides.

splice_donor_labelint | None

Token for the 5’ donor splice site.

splice_acceptor_labelint | None

Token for the 3’ acceptor splice site.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

classmethod default_exon_intron()

Canonical EXON_INTRON config so GFF users need not pick tokens.

Tokens: background=8, exon=0, intron=2, splice_donor=1, splice_acceptor=3 (the same scheme as BEND_LABEL_CONFIG). The integers are an internal array-encoding detail; only their distinctness matters.

Return type:

LabelConfig

classmethod default_utr_cds_intron()

Canonical UTR_CDS_INTRON config with explicit UTR/CDS/intron tokens.

Tokens: background=8, cds=0, five_prime_utr=4, three_prime_utr=5, intron=2, splice_donor=1, splice_acceptor=3. Mirrors the anatomy example in the README; the integers are an internal encoding detail.

Return type:

LabelConfig

property supports_phase_drift: bool

Whether CDS-only phase-drift evaluation is well-defined.

scope_tokens(scope)

Return the positive-token set for one benchmark scope.

Return type:

frozenset[int]

Parameters:

scope (BenchmarkScope | str)

available_scopes()

Return the metric scopes available for this label configuration.

Return type:

tuple[BenchmarkScope, ...]

out_of_scope_labels()

Exonic-content tokens demoted to background under evaluation_scope.

These are the labels STATE_TRANSITIONS collapses before analysis (via eval.preprocessing.collapse_out_of_scope_content), so transition plots omit them too. 5'/3' UTR under cds scope; empty under the default transcript_exon scope and in EXON_INTRON mode. INTRON and splice labels are never in scope_tokens and so are always kept.

Return type:

frozenset[int]

property labels: dict[int, str]

Full {token: name} mapping for all defined labels.

property evaluation_labels: dict[int, str]

Labels to compute per-class metrics for, excluding background.

name_of(token)

Return the human-readable name for token.

Return type:

str

Parameters:

token (int)

Parameters:

• annotation_mode (AnnotationMode)

• evaluation_scope (BenchmarkScope)

• background_label (int)

• exon_label (int | None)

• cds_label (int | None)

• five_prime_utr_label (int | None)

• three_prime_utr_label (int | None)

• intron_label (int | None)

• splice_donor_label (int | None)

• splice_acceptor_label (int | None)

class gene_calling_benchmark.label_definition.EvalMetrics(*values)

Bases: Enum

Available evaluation metric groups.

Each value answers one clear question about prediction quality:

• INDEL – “What structural errors exist?” 5’/3’ extensions/deletions, whole insertions/deletions, splits/joins.

• REGION_DISCOVERY – “Did we find the right regions?” Four coherent detection P/R tiers nested by strictness: neighborhood_hit (any overlap) ⊇ internal_hit (pred ⊆ GT) and full_coverage_hit (pred ⊇ GT) ⊇ perfect_boundary_hit (exact boundaries). Each hardens its FP so a matched-but-wrong-shape pair is booked as both FP and FN; internal_hit vs full_coverage_hit give the direction of the boundary error (under- vs over-extension).

• BOUNDARY_EXACTNESS – “How precise are the boundaries?” Scope-aware IoU stats, bias/reliability landscape, and boundary flags.

• NUCLEOTIDE_CLASSIFICATION – “Per-base, how accurate is it?” Binary per-scope outputs; scope depends on annotation mode.

• PHASE_DRIFT – “Is the CDS reading phase preserved?” CDS-only per-position reading-phase deviation. Measures relative CDS-base count drift between GT and prediction; it is a structural comparison signal, not a biological frameshift-mutation detector.

• STRUCTURAL_COHERENCE – “Is the segment chain correct as a whole?” Scope-aware chain comparison, transcript classification, and segment count diagnostics.

• DIAGNOSTIC_DEPTH – “Why is the prediction structurally wrong?” Scope-aware segment-length EMD and position-bias summaries.

• STATE_TRANSITIONS – “Where do label transitions go wrong?” GT transition confusion matrices plus classified false transitions (late-catchup / premature / spurious). Honours evaluation_scope: out-of-scope exonic content is demoted to background before analysis, so under cds scope 5’/3’ UTR read as NONCODING (a UTR-aware prediction is not penalised against a CDS-only GT). INTRON and splice labels are kept; the default transcript_exon scope demotes nothing.

gene_calling_benchmark.label_definition.SEMANTIC_BOUNDARY_ORDER: tuple[str, ...] = ('five_prime_terminal_exon', 'internal_exon', 'three_prime_terminal_exon', 'single_exon_gene')

Canonical display order for the four semantic exon-position categories.

gene_calling_benchmark.label_definition.semantic_boundary_label(left, right)

Map a GT flank pair to a biological exon-position category.

A flank is terminal when it is outside the intron chain: "NONCODING" (intergenic), "none" (sequence edge), or a UTR label ("FIVE_PRIME_UTR" / "THREE_PRIME_UTR" — marks the CDS start/stop-codon boundary in UTR_CDS_INTRON mode). "INTRON" is the only internal flank.

Returns one of the four values in SEMANTIC_BOUNDARY_ORDER.

Return type:

str

Parameters:

• left (str)

• right (str)