Online W&B Logging

The W&B integration is built around a simple contract: run the benchmark yourself, then hand the aggregated result dict to the logging helpers.

Relevant public functions:

Minimal Loop

from gene_calling_benchmark import (
    BEND_LABEL_CONFIG,
    EvalMetrics,
    benchmark_from_arrays,
    init_wandb_with_presets,
    log_benchmark_scalars,
    log_benchmark_media,
    log_benchmark_media_videos,
)

run = init_wandb_with_presets(
    project="gene-benchmark",
    run_name="validation",
)

results = benchmark_from_arrays(
    gt_labels=gt_arrays,
    pred_labels=pred_arrays,
    label_config=BEND_LABEL_CONFIG,
    metrics=[
        EvalMetrics.REGION_DISCOVERY,
        EvalMetrics.BOUNDARY_EXACTNESS,
        EvalMetrics.STRUCTURAL_COHERENCE,
    ],
    infer_introns=True,
)

log_benchmark_scalars(
    results,
    step=12,
    method_prefix="val",
)
log_benchmark_media(
    results,
    BEND_LABEL_CONFIG,
    step=12,
    method_prefix="val"
)

# Optional, usually after several calls to log_benchmark_media(...)
log_benchmark_media_videos()

run.finish()

Note

benchmark_from_arrays(...) tops out at roughly 1000 transcript pairs per second, so on a large validation set it can take a while. Run it on a separate thread (or process) off the critical GPU training path so the benchmark doesn’t stall training — hand it the arrays and let it log asynchronously.

Note

benchmark_from_arrays(...) needs every GT/prediction array in memory at once. If your validation set is too large for that, use gene_calling_benchmark.StreamingBenchmark instead — feed pairs one at a time with .add(gt, pred) and call .result() at the end. It returns the same numbers; it just never holds all the arrays simultaneously.

What Gets Logged

log_benchmark_scalars(...) logs a curated, high-signal online subset. Keys follow [<method_prefix>/]<group>/<leaf>, and a group only appears if its metric was in the result:

  • boundary_exactness/iou_mean

  • region_discovery/{neighborhood_hit,internal_hit,full_coverage_hit,perfect_boundary_hit}/{precision,recall}

  • nucleotide_classification/nucleotide/{precision,recall,f1}

  • struct_coherence/intron_chain/match_rate, struct_coherence/exon_chain/match_rate (all transcripts), struct_coherence/exon_chain_multi/match_rate (multi-exon only), struct_coherence/exon_chain_single/match_rate (single-exon only) (whole-chain tiers are all-or-nothing, so precision = recall = F1 — a single rate is logged), struct_coherence/segment_count_delta/{mean,mae}, struct_coherence/exon_recall_per_transcript/mean, struct_coherence/exon_precision_per_transcript/mean, struct_coherence/false_exon_count_per_transcript/mean, struct_coherence/exact_match_rate, struct_coherence/splice_site_results/{donor,acceptor}_{precision,recall}

  • diagnostic_depth/length_emd/{mean,mae}

To log every numeric scalar instead of this subset, use gene_calling_benchmark.log_benchmark_all_scalars().

log_benchmark_media(...) renders and logs every figure the benchmark plotting layer produces for the metric groups present in the result — not a fixed shortlist. Requesting STRUCTURAL_COHERENCE also emits transcript_match, segment_count_delta, boundary_shift_dist, per_transcript_exon_recovery, splice_site_confusion, splice_site_pr; INDEL adds indel_counts, indel_rates_by_boundary, indel_lengths_*; BOUNDARY_EXACTNESS adds iou_* and the boundary landscapes; and so on.

The subset that is also buffered for GIF videos (see below) is: position_bias, transition_matrices, false_transitions, boundary_bias_landscape, boundary_recall_landscape, transcript_match, segment_count_delta, phase_drift.

Media History vs GIF Videos

Each call to gene_calling_benchmark.log_benchmark_media() logs regular W&B images and also stores the rendered RGB frames in an internal buffer. gene_calling_benchmark.log_benchmark_media_videos() converts the buffered history into GIF videos and clears the buffer.

This design keeps the user-facing API simple:

  • call log_benchmark_media(...) whenever you want

  • call log_benchmark_media_videos() later if you want videos

  • call clear_benchmark_media_video_buffer() if you want to discard buffered frames instead

Dependency Note

Raw-array video logging through wandb.Video(...) requires the W&B media extras. In pyproject.toml that should be declared as:

[project.optional-dependencies]
wandb = ["wandb[media]>=0.26"]

Then install with:

uv sync --extra wandb

Caveats

  • The logging helpers expect one aggregated benchmark result dict, not raw per-sequence outputs.

  • method_prefix="val" is only a naming convention. The helper does not decide when validation happens.

  • Video generation happens only when gene_calling_benchmark.log_benchmark_media_videos() is called.