📘

Logistic Regression Prediction

MediumClassification 📚 Machine Learning 1

Implement the prediction step of logistic regression.

Given a feature matrix $X$ (without bias), a weight vector $w$ , and a bias $b$ , compute the probability for each sample using:

$P(y=1|x) = \sigma(x \cdot w + b) = \frac{1}{1 + e^{-(x \cdot w + b)}}$

Then classify each sample: if $P \geq 0.5$ , predict 1; otherwise predict 0.

Return a list of tuples [(probability, prediction), ...] with probabilities rounded to 4 decimal places.

Example:

Input:

X = [[1, 2], [3, 4], [-1, -2]]
w = [0.5, -0.3]
b = 0.1

Output:

[(0.5, 1), (0.5987, 1), (0.5498, 1)]

Reasoning:

We compute the dot product of each sample in $X$ $X$ with the weight vector $w$ $w$ and add the bias $b$ $b$ :
- For the first sample: $1 \cdot 0.5 + 2 \cdot -0.3 + 0.1 = 0.5 - 0.6 + 0.1 = 0$
- For the second sample: $3 \cdot 0.5 + 4 \cdot -0.3 + 0.1 = 1.5 - 1.2 + 0.1 = 0.4$
- For the third sample: $-1 \cdot 0.5 + -2 \cdot -0.3 + 0.1 = -0.5 + 0.6 + 0.1 = 0.2$
Then, we apply the sigmoid function $\sigma(x) = \frac{1}{1 + e^{-x}}$ $σ (x) = \frac{1}{1 + e ^{- x}}$ to each result:
- For the first sample: $\sigma(0) = \frac{1}{1 + e^{0}} = \frac{1}{1 + 1} = 0.5$
- For the second sample: $\sigma(0.4) = \frac{1}{1 + e^{-0.4}} \approx 0.5987$
- For the third sample: $\sigma(0.2) = \frac{1}{1 + e^{-0.2}} \approx 0.5498$
We classify each sample based on the predicted probability, with $P \geq 0.5$ resulting in a prediction of 1, and $P < 0.5$ resulting in a prediction of 0.
The final output is a list of tuples containing the predicted probabilities rounded to 4 decimal places and the corresponding predictions: $(0.5, 1), (0.5987, 1), (0.5498, 1)$

Constraints:

X is a 2D list (n_samples x n_features)
w is a list of n_features weights
b is a scalar bias
Return list of (probability, prediction) tuples
Probabilities rounded to 4 decimal places

More from Machine Learning 1

RBF Kernel MatrixHard Polynomial KernelMedium Hinge LossEasy Linear Decision BoundaryEasy Gradient Boosting Residual StepHard

Background Knowledge

Logistic Regression is a fundamental algorithm in Machine Learning used for binary classification problems. It's based on the idea of predicting the probability of an event occurring by fitting data to a logistic curve, also known as the sigmoid function. The sigmoid function, denoted as $\sigma(x)$ , maps any real-valued number to a value between 0 and 1, making it suitable for modeling probabilities.

The prediction step in logistic regression involves calculating the probability of a sample belonging to a particular class (in this case, $y=1$ ) given its features $x$ . This is done using the formula $P(y=1|x) = \sigma(x \cdot w + b)$ , where $w$ is the weight vector, $b$ is the bias, and $x \cdot w$ represents the dot product of the feature vector $x$ and the weight vector $w$ . The result of this calculation gives us the probability, which can then be used to make a prediction by comparing it to a threshold, typically 0.5.

Understanding the sigmoid function is crucial for logistic regression. The sigmoid function is defined as $\sigma(x) = \frac{1}{1 + e^{-x}}$ , where $e$ is the base of the natural logarithm. This function has an S-shaped curve, where values of $x$ less than 0 approach 0, and values greater than 0 approach 1. This property makes the sigmoid function ideal for modeling binary outcomes.

Algorithm/Approach

The general approach to solving this problem involves:

Computing the dot product of each feature vector $x$ in the feature matrix $X$ with the weight vector $w$ .
Adding the bias $b$ to each result from the previous step.
Applying the sigmoid function to each result to obtain the probabilities.
Comparing each probability to the threshold (0.5) to make predictions.

Step-by-Step Strategy

Initialize an empty list to store the results.
Iterate over each feature vector $x$ in the feature matrix $X$ .
For each $x$ , compute the dot product $x \cdot w$ .
Add the bias $b$ to the result from step 3.
Apply the sigmoid function to the result from step 4 to get the probability.
Round the probability to 4 decimal places.
Compare the probability to 0.5 and make a prediction (1 if $P \geq 0.5$ , 0 otherwise).
Append a tuple containing the rounded probability and the prediction to the results list.
Return the list of results.

Common Pitfalls

Incorrectly implementing the sigmoid function or the dot product.
Forgetting to add the bias $b$ .
Not rounding the probabilities to the specified number of decimal places.
Incorrectly comparing the probabilities to the threshold for making predictions.

Time & Space Complexity

The time complexity of this solution is $O(n \cdot m)$ , where $n$ is the number of samples in the feature matrix $X$ and $m$ is the number of features. This is because for each sample, we are performing a dot product operation that involves iterating over all its features. The space complexity is $O(n)$ , as we need to store the results for each sample.

solution.py

Python 3.13

Test Results

0/0

Run code to see test results.