CAFA Evaluation Protocol¶

PROTEA implements the evaluation protocol used in the CAFA5 (Critical Assessment of protein Function Annotation) challenge. This page explains the protocol, the NK/LK/PK classification, and how to run an evaluation end-to-end within PROTEA.

Background: the CAFA temporal holdout ¶

CAFA evaluates protein function prediction by exploiting the growth of experimental GO annotations over time:

t0. An older annotation snapshot (the reference set). Methods may use these annotations as training signal.
t1. A newer annotation snapshot (the ground truth). Proteins that gained new experimental GO annotations between t0 and t1 form the test set (the delta).

Only annotations with experimental evidence codes are considered (EXP, IDA, IMP, IGI, IEP, IPI, and their ECO equivalents). Annotations with a NOT qualifier (meaning the protein is not associated with that term) are excluded, and their exclusion is propagated to all GO descendants through the is_a and part_of relationships.

Formal definition ¶

This section gives a rigorous statement of the NK/LK/PK partitioning. The notation follows [Radivojac et al., 2013] and matches the reference implementation in protea.core.evaluation.

Preliminaries ¶

Let \(\mathcal{P}\) denote the universe of UniProt proteins present in PROTEA and let \(\mathcal{N} = \{\mathrm{F}, \mathrm{P}, \mathrm{C}\}\) denote the three GO namespaces (molecular function, biological process and cellular component). A fixed GO snapshot \(\sigma\) determines a set of terms \(\mathcal{G}_\sigma\) together with an aspect function \(\alpha_\sigma: \mathcal{G}_\sigma \to \mathcal{N}\) that assigns each term to exactly one namespace.

Define the GO children relation under the subset of edges used by the evaluation protocol (the is_a and part_of relationships only):

\[D_\sigma \;=\; \{(u, v) \in \mathcal{G}_\sigma \times \mathcal{G}_\sigma \;:\; v \text{ is a direct } \mathtt{is\_a} \text{ or } \mathtt{part\_of} \text{ child of } u \}.\]

The reflexive transitive closure of \(D_\sigma\) yields, for every term \(t\), its set of descendants

\[\mathrm{desc}_\sigma(t) \;=\; \{ v \in \mathcal{G}_\sigma \;:\; (t, v) \in D_\sigma^{+} \}.\]

Let \(\mathsf{Exp}\) denote the set of experimental evidence codes

\[\mathsf{Exp} \;=\; \{\mathrm{EXP},\, \mathrm{IDA},\, \mathrm{IPI},\, \mathrm{IMP},\, \mathrm{IGI},\, \mathrm{IEP},\, \mathrm{TAS},\, \mathrm{IC}\}\]

together with their ECO equivalents (the mapping is enumerated in protea.core.evidence_codes).

Annotation sets and NOT-propagation ¶

A raw annotation set \(s\) loaded into PROTEA is a set of tuples \((p, t, q, e) \in \mathcal{P} \times \mathcal{G}_\sigma \times Q \times E\) where \(q\) is a qualifier (possibly containing NOT) and \(e\) an evidence code. The negative closure of \(s\) over two annotation sets \(s_0, s_1\) sharing the same snapshot \(\sigma\) is

\[\begin{split}\mathsf{neg}(s_0, s_1) \;=\; \bigcup_{\substack{(p, t, q, e)\, \in\, s_0\, \cup\, s_1 \\ \texttt{NOT} \,\in\, q}} \; \{p\} \times \bigl(\{t\} \cup \mathrm{desc}_\sigma(t)\bigr).\end{split}\]

That is: every NOT-qualified annotation for a protein propagates to all descendants of the negated term through \(D_\sigma\), and the resulting (protein, term) pairs are excluded from both sides of the delta. This matches the function protea.core.evaluation._build_negative_keys().

The set of experimental, non-negated annotations of protein \(p\) in namespace \(n\) at time \(t_i\) is then

\[A_i(p, n) \;=\; \bigl\{\, g \in \mathcal{G}_\sigma \;:\; (p, g, q, e) \in s_i, \; e \in \mathsf{Exp}, \; \texttt{NOT} \notin q, \; \alpha_\sigma(g) = n, \; (p, g) \notin \mathsf{neg}(s_0, s_1) \bigr\}.\]

The per-namespace delta is the standard set difference

\[\Delta(p, n) \;=\; A_1(p, n) \setminus A_0(p, n),\]

and a protein belongs to the delta set \(\mathcal{P}_\Delta \subseteq \mathcal{P}\) iff it gained at least one annotation in at least one namespace:

\[\mathcal{P}_\Delta \;=\; \Bigl\{ p \in \mathcal{P} \;:\; \bigcup_{n \in \mathcal{N}} \Delta(p, n) \neq \emptyset \Bigr\}.\]

Partitioning ¶

The NK, LK and PK subsets are defined over \((p, n)\) pairs with \(p \in \mathcal{P}_\Delta\) and \(\Delta(p, n) \neq \emptyset\). Let the indicator \(\kappa(p) \in \{0, 1\}\) record whether the protein had any experimental annotation at \(t_0\):

\[\kappa(p) \;=\; \mathbb{1}\!\Bigl[ \bigcup_{n' \in \mathcal{N}} A_0(p, n') \neq \emptyset \Bigr].\]

No-Knowledge (NK): a single per-protein category:

\[\mathrm{NK} \;=\; \bigl\{ (p, g) \;:\; p \in \mathcal{P}_\Delta,\; \kappa(p) = 0,\; g \in \textstyle\bigcup_{n \in \mathcal{N}} \Delta(p, n) \bigr\}.\]

Limited-Knowledge (LK): a per-(protein, namespace) category:

\[\mathrm{LK} \;=\; \bigl\{ (p, g) \;:\; \kappa(p) = 1,\; \exists\, n \in \mathcal{N},\; A_0(p, n) = \emptyset,\; g \in \Delta(p, n) \bigr\}.\]

Partial-Knowledge (PK): also per-(protein, namespace):

\[\mathrm{PK} \;=\; \bigl\{ (p, g) \;:\; \kappa(p) = 1,\; \exists\, n \in \mathcal{N},\; A_0(p, n) \neq \emptyset,\; g \in \Delta(p, n) \bigr\}.\]

The associated known-term exclusion file used by cafaeval via the -known flag on the PK pass is

\[\mathrm{PK}_{\text{known}} \;=\; \bigl\{ (p, g) \;:\; (p, n) \in \mathrm{PK}_\text{keys},\; g \in A_0(p, n) \bigr\},\]

where \(\mathrm{PK}_\text{keys} = \{(p, n) : \exists\, g,\, (p, g) \in \mathrm{PK} \wedge \alpha_\sigma(g) = n\}\).

Properties ¶

The partition has three properties that make the protocol well-defined and directly testable against the implementation.

Disjointness of NK from LK and PK. A protein classified as NK satisfies \(\kappa(p) = 0\), which means \(A_0(p, n) = \emptyset\) for every namespace \(n\). The LK and PK conditions both require \(\kappa(p) = 1\), so no protein appears in both NK and LK ∪ PK. Formally:

\[\{p : (p, g) \in \mathrm{NK}\} \;\cap\; \{p : (p, g) \in \mathrm{LK} \cup \mathrm{PK}\} \;=\; \emptyset.\]

LK / PK mutual exclusion per namespace. For a fixed namespace \(n\), the conditions \(A_0(p, n) = \emptyset\) and \(A_0(p, n) \neq \emptyset\) are mutually exclusive. Therefore no (protein, namespace) pair can be simultaneously LK and PK.

LK and PK are *not* mutually exclusive across namespaces. A protein with \(A_0(p, \mathrm{F}) \neq \emptyset\) and \(A_0(p, \mathrm{P}) = \emptyset\) can gain new annotations in both namespaces at \(t_1\); it then contributes PK pairs in \(\mathrm{F}\) and LK pairs in \(\mathrm{P}\) simultaneously. This is not a bug of the protocol: it reflects the per-(protein, namespace) granularity that distinguishes CAFA5 from earlier rounds.

Relation to the implementation. The definitions above correspond line-by-line to the classification loop of protea.core.evaluation.compute_evaluation_data():

\(A_i(p, n)\) ← _load_experimental_annotations_by_ns;
\(\mathsf{neg}(s_0, s_1)\) ← _build_negative_keys;
\(\kappa(p)\) ← had_anything_old = bool(old_ns_map);
the per-namespace delta_ns = new_ns - old_ns computes \(\Delta(p, n)\);
the branches if not old_ns and else realise the LK / PK separation;
\(\mathrm{PK}_{\text{known}}\) is accumulated in the pk_known dict.

NK / LK / PK classification ¶

A key feature of CAFA5 is that test proteins are not treated uniformly. Classification is determined per (protein, namespace), where namespace is one of Molecular Function (MFO), Biological Process (BPO), or Cellular Component (CCO).

NK. No-Knowledge.: The protein had no experimental annotations in any namespace at t0. All its new annotations across all namespaces form the NK ground truth. Evaluating NK targets tests a method’s ability to make predictions from sequence alone, without any prior functional signal.
LK. Limited-Knowledge.: The protein had experimental annotations in some namespaces at t0, but not in namespace S. It gained new annotations in S at t1. Those new annotations in S are the LK ground truth for that (protein, S) pair. Evaluating LK tests transfer across namespaces.
PK. Partial-Knowledge.: The protein already had experimental annotations in namespace S at t0, and gained additional annotations in S at t1. Only the novel terms are ground truth; the old terms are collected in a pk_known_terms.tsv file and passed to cafaeval with the -known flag, which excludes them from scoring. This prevents credit for simply repeating prior annotations.

Important

A single protein can be LK in one namespace and PK in another simultaneously. For example, a protein with MFO and BPO annotations at t0 that gains new CCO and BPO annotations at t1 will be LK for CCO and PK for BPO.

Toy example ¶

Protein P1 at t0:  MFO={GO:0003674}   BPO={}       CCO={}
Protein P1 at t1:  MFO={GO:0003674}   BPO={GO:0008150}  CCO={GO:0005575}

had_anything_old = True (had MFO)

Namespace BPO: old_BPO={}  → LK (empty at t0, gained GO:0008150)
Namespace CCO: old_CCO={}  → LK (empty at t0, gained GO:0005575)
Namespace MFO: no new terms → not in test set for this namespace

Protein P2 at t0:  BPO={GO:0006355}   (all others empty)
Protein P2 at t1:  BPO={GO:0006355, GO:0045893}

Namespace BPO: old_BPO={GO:0006355}  delta={GO:0045893}
  → PK ground truth = {GO:0045893}
  → pk_known = {GO:0006355}  (passed as -known)

Protein P3 at t0:  (no annotations in any namespace)
Protein P3 at t1:  MFO={GO:0003674}   BPO={GO:0008150}

had_anything_old = False → NK
NK ground truth = {GO:0003674, GO:0008150} (all new terms)

Evaluation flow in PROTEA ¶

1. Load two GOA annotation sets (old = t0, new = t1).
2. POST /annotations/evaluation-sets/generate
   → queues generate_evaluation_set job
   → computes delta and creates EvaluationSet row with stats
3. Download delta-proteins.fasta (all NK+LK+PK sequences).
4. POST /jobs  (compute_embeddings, query_set_id=...)
   → compute ESM-2 embeddings for delta proteins
5. POST /embeddings/predict  (predict_go_terms, query_set_id=...)
   → run KNN GO transfer; creates PredictionSet
6. POST /annotations/evaluation-sets/{id}/run
   → queues run_cafa_evaluation job
   → runs cafaeval for NK, LK, PK; creates EvaluationResult
7. View results in the Evaluation UI or download artifacts.zip.

The cafaeval command equivalent (for manual inspection):

python -m cafaeval go-basic.obo predictions/ ground_truth_NK.tsv -out_dir results/NK
python -m cafaeval go-basic.obo predictions/ ground_truth_LK.tsv -out_dir results/LK
python -m cafaeval go-basic.obo predictions/ ground_truth_PK.tsv \
  -known pk_known_terms.tsv -out_dir results/PK

Data model ¶

EvaluationSet: Stores the (old_annotation_set_id, new_annotation_set_id) pair and a JSONB stats dict with delta/NK/LK/PK protein and annotation counts. Created by generate_evaluation_set.
EvaluationResult: Stores per-setting (NK/LK/PK) and per-namespace (MFO/BPO/CCO) metrics: Fmax, precision, recall, τ (threshold), and coverage. Created by run_cafa_evaluation. Multiple EvaluationResult rows can exist per EvaluationSet, one per (prediction_set, run).

See Infrastructure for the full ORM schema.

Benchmark: PROTEA vs external tools ¶

Provisional results, pending final recompute

The Fmax values in the tables below were produced before the 2026-04-10 unification of the embedding-backend slicing convention (see Operations, section Residue-tensor convention). They will be regenerated end-to-end for the Zenodo deposit accompanying the thesis. The experimental protocol is stable; only the numbers will change. See Results for the full provisional notice.

PROTEA was benchmarked against two widely used function annotation tools using the temporal holdout GOA 220 → GOA 229 (NK: 2831, LK: 3410, PK: 15313 proteins). All evaluations use cafaeval with Information Accretion (IA) weighting from the CAFA6 benchmark.

Fmax (IA-weighted): GOA 220 → 229¶
Method	NK-BPO	NK-MFO	NK-CCO	LK-BPO	LK-MFO	LK-CCO	PK-BPO	PK-MFO	PK-CCO
Pannzer2 ^†	0.656	0.717	0.791	0.681	0.729	0.813	0.391	0.574	0.618
PROTEA (re-ranker iteration 3, full features)	0.431	0.620	0.692	0.478	0.607	0.697	0.201	0.297	0.339
InterProScan 6 ^†	0.312	0.551	0.476	0.479	0.488	0.491	0.208	0.269	0.250
eggNOG-mapper 2.1.13 ^†	0.247	0.359	0.386	0.382	0.334	0.450	0.190	0.199	0.325

^† Subject to temporal data leakage; see below.

Temporal data leakage ¶

Both Pannzer2 and eggNOG-mapper were executed in March 2026 against their current reference databases, which contain annotations published well after GOA 220 (the t0 snapshot). This means they have access to functional knowledge that is part of the ground truth.

To quantify this leakage, we measured exact (protein, GO term) matches between each tool’s predictions and the ground truth:

Exact match with ground truth¶
Category	GT pairs	Pannzer2 match	eggNOG match
NK	6,953	4,339 (62.4%)	1,025 (14.7%)
LK	5,520	3,624 (65.7%)	1,087 (19.7%)
PK	27,541	12,410 (45.1%)	8,196 (29.8%)
Total	40,014	20,373 (50.9%)	10,308 (25.8%)

Pannzer2 exactly matches 62.4% of NK annotations (proteins that by definition had no experimental annotations at t0). This confirms that its reference database already contains the experimental evidence that appeared between GOA 220 and GOA 229.

PROTEA is the only tool in this benchmark that enforces temporal integrity by design: the reference set is frozen at t0, the ground truth is computed as the delta, and all versions are tracked in the database. Pannzer2 and eggNOG-mapper numbers should be interpreted as an optimistic upper bound under data leakage, not as a fair comparison.

Note

Running Pannzer2 or eggNOG-mapper against a frozen historical database is not possible: the Pannzer2 web server does not offer version selection, and eggNOG does not publish historical orthology snapshots.

Evaluating external tools ¶

External tools can be evaluated against the same ground truth using scripts/evaluate_external_tool.py:

poetry run python scripts/evaluate_external_tool.py \
    --evaluation-set-id <uuid> \
    --tool emapper \
    --input /path/to/annotations.emapper.annotations

poetry run python scripts/evaluate_external_tool.py \
    --evaluation-set-id <uuid> \
    --tool pannzer2 \
    --input /path/to/anno.out

Supported formats: emapper, pannzer2, interproscan, blast.

Per-band canonical (ontology snapshot, IA) registry

A band is a GOA evaluation window (for example v226 or v227). Each band binds two derived artefacts that are pinned rather than free-floated:

An OntologySnapshot identified by its obo_version. The snapshot governs True-Path propagation, the term universe, and orphan handling. Every cell in the band scores against the same snapshot.
An Information Accretion (IA) artefact identified by a stable file token. The IA weights each GO term by its information content on the t0 corpus of that band.

The registry is protea.core.band_registry.BANDS. Authoritative pairs:

Band	GOA t0	Canonical `obo_version`	Canonical IA token
v226	goa 226 (2025-05-03)	`releases/2025-03-16`	`IA_cafa6.tsv`
v227	goa 227 (2025-09-04)	`releases/2025-07-22`	`IA.tsv` / `IA-swissprot-exp-v227.txt`

Adding a new band requires one new Band row in BANDS. No obo_version and no IA token may be shared by two bands; the CI guard (scripts/check_band_registry.py, wired into lint.yml) enforces this.

Why a snapshot/IA mismatch inflates a phantom gap. If a cell declared for one band is scored with the snapshot or IA of another band, the comparison measures artefact drift rather than prediction quality. A cross-band snapshot changes the propagation closure and the term universe; a cross-band IA reweights f_micro_w against a foreign corpus. The v226 IA_cafa6.tsv and the v227 IA.tsv disagree by up to 14.6 on shared terms. The registry fix binds both artefacts to the band and rejects any mix.

Dispatching a banded evaluation. Pass the band name in the run_cafa_evaluation payload and supply the canonical IA for that band:

{
  "evaluation_set_id": "<eval set>",
  "prediction_set_id": "<pred set>",
  "band": "v227",
  "ia_file": "/path/to/lafa_t0_Sep_2025/IA.tsv"
}

The operation resolves the pivot snapshot from the EvaluationSet, verifies its obo_version is canonical for the declared band, verifies the IA token, emits run_cafa_evaluation.band_verified, and only then calls cafaeval. A cross-band cell raises BandMismatchError and the job fails immediately.

LAFA evaluation parity

PROTEA’s run_cafa_evaluation operation uses the same cafaeval fork binary as LAFA (CAFA_forever). When the same prediction is scored on both sides, the headline metric must agree within LAFA’s 3-decimal rounding (epsilon about 5e-4). This section documents the alignment.

Headline metric. Both pipelines report f_micro_w: the IA-weighted micro-averaged F-measure, taken per namespace (BPO / CCO / MFO) at the threshold that maximises it. LAFA reads it from column 31 of evaluation_best_f_micro_w.tsv; PROTEA reads it from dfs_best["f_micro_w"] in parse_results.

cafaeval flag alignment. The table below lists every flag where LAFA and PROTEA were previously misaligned and the current state after the fix (PRs #599, #601):

Flag	LAFA	PROTEA (after fix)	Impact
`th_step`	`0.01` (default)	`0.01` (was `0.001`)	Dominant gap: finer grid inflated `f_micro_w` by up to +0.0144
`max_terms`	unlimited (default)	`None` (was `500`)	Inert for KNN-style predictions; removed for mechanical identity
`ia`	t0 `IA.tsv`	payload `ia_file`	Must point at the same IA artefact as LAFA used for the band
`toi`	`groundtruth_terms_of_interest.txt`	payload `toi_file`, else snapshot terms	Residual up to 0.004 on MFO without the exact file; pass it for strict parity
`prop`	`fill`	`fill`	Already matched
`norm`	`cafa`	`cafa`	Already matched
`no_orphans`	on	on	Already matched

How to run a LAFA-comparable evaluation. In the run_cafa_evaluation payload, leave th_step and max_terms at their defaults (0.01 and None), pass ia_file pointing at the same IA artefact LAFA used for the band, and pass toi_file pointing at LAFA’s release groundtruth_terms_of_interest.txt for strict MFO parity.

Persisted metrics. run_cafa_evaluation persists, per aspect, the IA-weighted f_micro_w (headline), fmax_w, f_micro, and the weighted micro precision_w / recall_w / coverage_w. The _w keys appear only when a real IA file was supplied; under the uniform IC=1 fallback they are omitted. The /v1/benchmark/matrix endpoint ranks every cell by f_micro_w (falling back to fmax for legacy IC=1 rows, flagged via primary_metric) and exposes a per_task mean and 95% CI.

Obsolete metrics. Unweighted fmax / f_micro (equal weight on all GO terms) are superseded by IA-weighted f_micro_w; they are kept for history but are not LAFA-comparable. Numbers from the v226 evaluation window are also kept for history but should not be compared head-to-head with LAFA v227 results (different band, different ground truth shape).

Implementation reference ¶

Core logic: protea.core.evaluation (EvaluationData, compute_evaluation_data)
Band registry: protea/core/band_registry.py (BANDS, assert_band_consistency); CI guard scripts/check_band_registry.py
Operations: protea.core.operations.generate_evaluation_set, protea.core.operations.run_cafa_evaluation
API router: protea/api/routers/annotations.py (download endpoints, generate and run routes)