Catastrophic Forgetting Detector
Detect catastrophic forgetting by comparing model performance before and after fine-tuning.
Given performance scores on N tasks before and after fine-tuning, compute:
- Forgetting score per task: max(0, score_before - score_after)
- Average forgetting: mean of all forgetting scores
- Backward transfer: mean of (score_after - score_before) for all tasks
A model suffers catastrophic forgetting if average forgetting > threshold.
Input:
- Line 1: N threshold
- Line 2: N floats (scores before fine-tuning)
- Line 3: N floats (scores after fine-tuning)
Output:
- Line 1: Average forgetting, rounded to 4 decimal places
- Line 2: Backward transfer, rounded to 4 decimal places
- Line 3: "FORGETTING" if avg forgetting > threshold, else "OK"
Example
Input
4 0.1 0.9 0.8 0.7 0.6 0.5 0.85 0.75 0.4
Output
0.1500 -0.0500 FORGETTING
Reasoning
- The input is processed to calculate the forgetting score per task: , (so ), (so ),
- Then, we calculate the average forgetting:
- The backward transfer is calculated as: , which rounds to when rounded to 4 decimal places in the context of the output format, but the actual calculation before rounding is
- Since the average forgetting () is greater than the given threshold (), the output is "FORGETTING"
Constraints
- 1 <= N <= 20
- Scores are in [0, 1]
- Threshold is in [0, 1]
- Round to 4 decimal places