Global Metrics

Global annotation-level metrics for the DNA segmentation benchmark.

Computes metrics over the full set of reference and predicted transcripts, comparable to gffcompare’s nucleotide and exon sensitivity/precision.

Unlike the per-transcript metrics (which evaluate matched pairs in isolation), global metrics aggregate over all transcripts — including unmatched ones — so false-positive predictions and missed reference transcripts both contribute to the final numbers.

Eight metric groups are computed, each answering a distinct question:

  • nucleotide — “At the base level, how accurate is the exon coverage?” Union-based: each genomic base is counted once regardless of isoform count. Equivalent to gffcompare’s nucleotide sensitivity/precision.

  • exon — “How many exons are exactly reconstructed?” De-duplicated counting: each unique (seqid, strand, start, end) interval is counted once across all transcripts, regardless of how many isoforms share it. This diverges from gffcompare’s per-isoform counting; see _compute_exon_and_structure_metrics() for the rationale.

  • exon_lenient — “Exon recovery with TSS/TES boundary leniency.” Terminal-exon outer boundaries are not required to match (gffcompare default =); only internal splice-site boundaries must be exact.

  • intron_chain — “How many multi-exon intron chains match exactly?” Reproduces gffcompare’s intron-chain Sn/Sp by coordinate-exact structure matching over the full reference, independent of the locus-matching mode.

  • transcript_exact — “How many transcripts match exactly (coordinate-exact)?” Reproduces gffcompare’s transcript Sn/Sp by coordinate-exact structure matching over the full reference, independent of the locus-matching mode.

  • transcript — “How many transcripts are recovered?” Sensitivity = matched ref transcripts / total ref transcripts. Precision = matched pred transcripts / total pred transcripts.

  • gene — “How many gene loci are detected?” Transcripts are clustered into loci by coordinate overlap (same algorithm as map_transcripts). A locus is matched if any of its transcripts was assigned a counterpart on the other side.

  • locus_isoform — “How completely are multi-isoform loci recovered?” For each GT locus, counts the fraction of its isoforms that received a prediction match. Unlike the gene metric (which only checks for ANY match), this distinguishes a locus where 1 of 5 isoforms was found from one where all 5 were found. Most meaningful with FULL_DISCOVERY matching, where every GT isoform participates in Hungarian assignment.

gene_calling_benchmark.eval.global_metrics.compute_global_metrics(gt_df, pred_df, mappings, predictor_name, label_config, transcript_types, gt_feature_role_map=None, pred_feature_role_map=None, locus_matching_mode=LocusMatchingMode.FULL_DISCOVERY, ignore_novel_predictions=False, ignore_missed_reference=False)[source]

Compute global annotation-level metrics for one predictor.

Parameters:

  • gt_df (DataFrame) – Pre-collected ground-truth GFF DataFrame (from collect_gff).

  • pred_df (DataFrame) – Pre-collected prediction GFF DataFrame for this predictor.

  • mappings (list[TranscriptMapping]) – Transcript mapping result from map_transcripts.

  • predictor_name (str) – Name of the predictor; must match the name used in map_transcripts.

  • label_config (LabelConfig) – Label configuration.

  • transcript_types (list[str]) – GFF feature types that define transcript boundaries (e.g. ["mRNA", "transcript"]).

  • gt_feature_role_map (dict[str, str] | None) – Optional mapping of GT GFF feature types to label roles; normalized against label_config when omitted.

  • pred_feature_role_map (dict[str, str] | None) – Optional mapping of prediction GFF feature types to label roles; normalized against label_config when omitted.

  • locus_matching_mode (LocusMatchingMode) – How predictions are assigned to GT loci (FULL_DISCOVERY or BEST_PER_LOCUS); affects the assignment-based "transcript" and "locus_isoform" families.

  • ignore_novel_predictions (bool) – gffcompare -Q: drop prediction transcripts that overlap no GT transcript from the precision side (incomplete-ground-truth correction).

  • ignore_missed_reference (bool) – gffcompare -R: drop GT transcripts that overlap no prediction from the sensitivity side (incomplete-prediction correction).

Returns:

Eight keys: "nucleotide", "exon", "exon_lenient", "intron_chain", "transcript_exact", "transcript", "gene", "locus_isoform". Each value is a flat dict of counts and derived P/R/F1 scores. "exon" uses exact boundary matching; "exon_lenient" relaxes the outer boundary of terminal exons (gffcompare style). "intron_chain" and "transcript_exact" reproduce gffcompare’s intron-chain and transcript Sn/Sp by coordinate-exact structure matching over the full reference, independent of the locus-matching mode (see the respective functions). "transcript" instead reports assignment-based transcript recall from mappings (mode-dependent) and is retained for backward compatibility. "locus_isoform" reports per-locus isoform recall — the fraction of GT isoforms per locus that received a match, addressing multi-isoform caller fairness. Most meaningful with FULL_DISCOVERY matching.

Return type:

dict

gene_calling_benchmark.eval.global_metrics.compute_overlap_keepsets(gt_df, pred_df, transcript_types)[source]

Coordinate-overlap keep-sets for gffcompare-style -R / -Q.

Returns (ref_keep, pred_keep) where

  • ref_keep — GT transcripts whose span overlaps ≥1 prediction transcript span on the same (seqid, strand) (drives -R).

  • pred_keep — prediction transcripts whose span overlaps ≥1 GT transcript span on the same (seqid, strand) (drives -Q).

Transcripts are identified by (seqid, gff_id), not by gff_id alone: some callers (e.g. Tiberius) restart transcript ids per sequence, so a bare id keeps every same-named transcript on every other seqid alive once one copy overlaps — silently under-pruning the correction.

Overlap is transcript-span based (any shared base) and strand-aware — the same notion map_transcripts() uses to assign predictions to loci — and is computed on the original, unfiltered frames so applying one correction never changes the other’s keep-set.

Return type:

tuple[set[tuple[str, str]], set[tuple[str, str]]]

Parameters: