Logistic Regression Prediction

mediumFREEClassificationMachine Learning 1

Implement the prediction step of logistic regression.

Given a feature matrix $X$ (without bias), a weight vector $w$ , and a bias $b$ , compute the probability for each sample using:

$P(y=1|x) = \sigma(x \cdot w + b) = \frac{1}{1 + e^{-(x \cdot w + b)}}$

Then classify each sample: if $P \geq 0.5$ , predict 1; otherwise predict 0.

Return a list of tuples [(probability, prediction), ...] with probabilities rounded to 4 decimal places.

Example

Input

X = [[1, 2], [3, 4], [-1, -2]]
w = [0.5, -0.3]
b = 0.1

Output

[(0.5, 1), (0.5987, 1), (0.5498, 1)]

Reasoning

Inputs: X holds three samples of two features; w weights each feature and b = 0.1 shifts every score equally.
Linear score $z = x \cdot w + b$ $z = x \cdot w + b$ for each row:
- $(1)(0.5) + (2)(-0.3) + 0.1 = 0.5 - 0.6 + 0.1 = 0$
- $(3)(0.5) + (4)(-0.3) + 0.1 = 1.5 - 1.2 + 0.1 = 0.4$
- $(-1)(0.5) + (-2)(-0.3) + 0.1 = -0.5 + 0.6 + 0.1 = 0.2$
Squash with the sigmoid: $\sigma(0) = \frac{1}{1+1} = 0.5$ , $\sigma(0.4) = \frac{1}{1 + 0.6703} \approx 0.5987$ , $\sigma(0.2) = \frac{1}{1 + 0.8187} \approx 0.5498$ , each rounded to 4 decimal places.
Threshold: all three probabilities are $\geq 0.5$ , so every label is 1, giving [(0.5, 1), (0.5987, 1), (0.5498, 1)].
Notice the first sample lands exactly on the decision boundary ( $z = 0$ , $P = 0.5$ ) and still counts as class 1 because the rule is $\geq$ . The third row negates the first, yet scores $0.2$ , not $0$ : the bias does not flip sign with $x$ .

Constraints

X is a 2D list (n_samples x n_features)
w is a list of n_features weights
b is a scalar bias
Return list of (probability, prediction) tuples
Probabilities rounded to 4 decimal places

Learn this in

ClassificationML Study Plan

More from Machine Learning 1

RBF Kernel Matrixhard Polynomial Kernelmedium Hinge Losseasy Linear Decision Boundaryeasy Gradient Boosting Residual Stephard

Concept primer

Background Knowledge

Logistic Regression is a fundamental algorithm in Machine Learning used for binary classification problems. It's based on the idea of predicting the probability of an event occurring by fitting data to a logistic curve, also known as the sigmoid function. The sigmoid function, denoted as $\sigma(x)$ , maps any real-valued number to a value between 0 and 1, making it suitable for modeling probabilities.

The prediction step in logistic regression involves calculating the probability of a sample belonging to a particular class (in this case, $y=1$ ) given its features $x$ . This is done using the formula $P(y=1|x) = \sigma(x \cdot w + b)$ , where $w$ is the weight vector, $b$ is the bias, and $x \cdot w$ represents the dot product of the feature vector $x$ and the weight vector $w$ . The result of this calculation gives us the probability, which can then be used to make a prediction by comparing it to a threshold, typically 0.5.

Understanding the sigmoid function is crucial for logistic regression. The sigmoid function is defined as $\sigma(x) = \frac{1}{1 + e^{-x}}$ , where $e$ is the base of the natural logarithm. This function has an S-shaped curve, where values of $x$ less than 0 approach 0, and values greater than 0 approach 1. This property makes the sigmoid function ideal for modeling binary outcomes.

Algorithm/Approach

The general approach to solving this problem involves:

Computing the dot product of each feature vector $x$ in the feature matrix $X$ with the weight vector $w$ .
Adding the bias $b$ to each result from the previous step.
Applying the sigmoid function to each result to obtain the probabilities.
Comparing each probability to the threshold (0.5) to make predictions.

Step-by-Step Strategy

Initialize an empty list to store the results.
Iterate over each feature vector $x$ in the feature matrix $X$ .
For each $x$ , compute the dot product $x \cdot w$ .
Add the bias $b$ to the result from step 3.
Apply the sigmoid function to the result from step 4 to get the probability.
Round the probability to 4 decimal places.
Compare the probability to 0.5 and make a prediction (1 if $P \geq 0.5$ , 0 otherwise).
Append a tuple containing the rounded probability and the prediction to the results list.
Return the list of results.

Common Pitfalls

Incorrectly implementing the sigmoid function or the dot product.
Forgetting to add the bias $b$ .
Not rounding the probabilities to the specified number of decimal places.
Incorrectly comparing the probabilities to the threshold for making predictions.

Time & Space Complexity

The time complexity of this solution is $O(n \cdot m)$ , where $n$ is the number of samples in the feature matrix $X$ and $m$ is the number of features. This is because for each sample, we are performing a dot product operation that involves iterating over all its features. The space complexity is $O(n)$ , as we need to store the results for each sample.

solution.py

Python 3.13

Test results