📘

Gradient Boosting Residual Step

Implement one step of Gradient Boosting for regression.

Given current predictions $\hat{y}$ and true values $y$ :

Compute residuals: $r_i = y_i - \hat{y}_i$
Fit a simple model to the residuals: compute the mean residual for each group defined by a feature threshold. Given the split threshold, predict the mean residual for left ( $\leq$ threshold) and right ( $>$ threshold) groups.
Update predictions: $\hat{y}_i' = \hat{y}_i + \eta \cdot h(x_i)$ where $\eta$ is the learning rate and $h(x_i)$ is the predicted residual.

Return the updated predictions, rounded to 4 decimal places.

Input:

y_true = [3, 6, 4, 8]
y_pred = [2, 5, 3, 7]
feature_values = [1, 3, 2, 4]
threshold = 2.5
learning_rate = 0.1

Output:

[2.1, 5.1, 3.1, 7.1]

Reasoning:

Compute residuals: $r_i = y_i - \hat{y}_i$ which results in $r = [3-2, 6-5, 4-3, 8-7] = [1, 1, 1, 1]$
Fit a simple model to the residuals: since the feature threshold is 2.5, the mean residual for the left group ( $\leq$ 2.5) is $(1+1)/2 = 1$ and for the right group ( $>$ 2.5) is $(1+1)/2 = 1$ .
Update predictions: $\hat{y}_i' = \hat{y}_i + \eta \cdot h(x_i)$ where $\eta = 0.1$ and $h(x_i)$ is the predicted residual, which is 1 for all samples, resulting in $\hat{y}_i' = \hat{y}_i + 0.1 \cdot 1 = \hat{y}_i + 0.1$
The final output is obtained by applying the update to each prediction and rounding to 4 decimal places: $[2+0.1, 5+0.1, 3+0.1, 7+0.1] = [2.1, 5.1, 3.1, 7.1]$

Editor

Python 3.13.1

0/0

Run code to see test results.