Catastrophic Forgetting Detector
Detect catastrophic forgetting by comparing model performance before and after fine-tuning.
Given performance scores on N tasks before and after fine-tuning, compute:
- Forgetting score per task: max(0, score_before - score_after)
- Average forgetting: mean of all forgetting scores
- Backward transfer: mean of (score_after - score_before) for all tasks
A model suffers catastrophic forgetting if average forgetting > threshold.
Input:
- Line 1: N threshold
- Line 2: N floats (scores before fine-tuning)
- Line 3: N floats (scores after fine-tuning)
Output:
- Line 1: Average forgetting, rounded to 4 decimal places
- Line 2: Backward transfer, rounded to 4 decimal places
- Line 3: "FORGETTING" if avg forgetting > threshold, else "OK"
Example:
4 0.1 0.9 0.8 0.7 0.6 0.5 0.85 0.75 0.4
0.1500 -0.0500 FORGETTING
- The input is processed to calculate the forgetting score per task: 0.9−0.5=0.4, 0.8−0.85=−0.05 (so 0), 0.7−0.75=−0.05 (so 0), 0.6−0.4=0.2
- Then, we calculate the average forgetting: 40.4+0+0+0.2​=40.6​=0.15
- The backward transfer is calculated as: 4(0.5−0.9)+(0.85−0.8)+(0.75−0.7)+(0.4−0.6)​=4−0.4+0.05+0.05−0.2​=4−0.5​=−0.125, which rounds to −0.0500 when rounded to 4 decimal places in the context of the output format, but the actual calculation before rounding is −0.125
- Since the average forgetting (0.15) is greater than the given threshold (0.1), the output is "FORGETTING"
Constraints:
- 1 <= N <= 20
- Scores are in [0, 1]
- Threshold is in [0, 1]
- Round to 4 decimal places
More from LLM 2: Training & Alignment
Background Knowledge
The concept of catastrophic forgetting is crucial in understanding this problem. Catastrophic forgetting occurs when a model forgets previously learned information after being fine-tuned on new data. This is a common issue in transfer learning, where a pre-trained model is adapted to a new task. The goal of detecting catastrophic forgetting is to determine if the model's performance on previous tasks has significantly degraded after fine-tuning.
In the context of this problem, we are given performance scores on N tasks before and after fine-tuning. We need to compute the forgetting score per task, which represents the amount of performance degradation for each task. The average forgetting score is then calculated as the mean of all forgetting scores. Additionally, we need to compute the backward transfer, which represents the mean change in performance after fine-tuning.
The problem also introduces a threshold value, which is used to determine if the model has suffered from catastrophic forgetting. If the average forgetting score exceeds this threshold, the model is considered to have suffered from catastrophic forgetting. Understanding these concepts is essential to developing an effective solution to this problem.
Algorithm/Approach
The approach to solving this problem involves the following general steps:
- Calculate the forgetting score for each task by comparing the performance scores before and after fine-tuning.
- Compute the average forgetting score by taking the mean of all forgetting scores.
- Calculate the backward transfer by taking the mean of the changes in performance after fine-tuning.
- Compare the average forgetting score to the given threshold to determine if the model has suffered from catastrophic forgetting.
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.