Chapter 2: NumPy & Data Wrangling

Master NumPy arrays for numerical computing, Pandas DataFrames for data manipulation, and essential data preprocessing techniques including cleaning, scaling, and encoding.

Topics: 9
Videos: 9
Min read: 83

Chapter Overview

NumPy is the foundation of scientific computing in Python. Every image processing, machine learning, and data science library builds on NumPy arrays. Whether you're processing images pixel by pixel, training neural networks, or analyzing tabular datasets, NumPy arrays are at the core.

Why NumPy matters:

Universal data structure: Images are 2D/3D arrays, feature matrices are 2D arrays, neural network weights are multi-dimensional arrays
Vectorized operations: Process millions of elements without explicit loops -- 10-100x faster than pure Python
Memory efficiency: Contiguous storage, typed arrays, views instead of copies
Interoperability: OpenCV, PyTorch, TensorFlow, scikit-learn all use NumPy-compatible arrays

Pandas builds on NumPy to provide labeled, tabular data structures. A DataFrame is like a spreadsheet in Python: rows represent observations, columns represent features, and you can refer to data by name rather than position. Pandas excels at loading data from various sources, handling missing values, merging datasets, and computing aggregations.

Real-world data is messy. You'll encounter missing values, outliers, inconsistent formats, and features on vastly different scales. Data preprocessing often takes 80% of a data scientist's time, and this chapter teaches the essential techniques.

What you'll learn:

Array Creation & Operations -- Building arrays, understanding shape/dtype, vectorized computation
Indexing & Slicing -- Extracting regions, channels, and subsets efficiently
Broadcasting -- Writing concise code for element-wise operations across different shapes
Reshaping -- Converting between image formats (HWC, CHW, batched) and data layouts
Statistics -- Computing histograms, means, distributions, and aggregations
Linear Algebra -- Matrix operations for transforms, decompositions, and dimensionality reduction
Pandas DataFrames -- Loading, exploring, filtering, and transforming tabular data
Data Cleaning -- Handling missing values, outliers, and data quality issues
Feature Preparation -- Scaling, encoding, and transforming features for ML algorithms

These skills directly apply to every CV and ML task: loading images, preprocessing data, applying filters, preparing features, and feeding data into models.

Chapter Roadmap

Click any topic to jump in

Explainer video

Array Creation

Creating N-dimensional arrays, dtypes, and vectorized operations — the foundation of numerical computing in Python.

Creating ArraysArray Shape and DimensionsData Types (dtype)Special Array ConstructorsVectorizationUniversal Functions (ufuncs)

Accessing and combining

Selection and shape alignment

Indexing & Slicing

Basic, advanced, and boolean indexing with views vs copies — efficiently selecting data from arrays.

Basic IndexingSlicing SyntaxAdvanced IndexingViews vs CopiesBoolean Indexing

Broadcasting

How NumPy aligns arrays of different shapes for element-wise operations — eliminating explicit loops.

Element-wise OperationsBroadcasting RulesCommon Broadcasting PatternsUniversal Functions (ufuncs)

Transforming structure and content

Geometry and summary statistics

Reshaping

Reshape, transpose, stack, and concatenate — manipulating array geometry without copying data.

ReshapeFlatten and RavelTranspose and AxesStacking and Concatenating

Statistics

Mean, std, percentiles along any axis — summarizing data in one vectorized call.

Basic StatisticsAxis ParameterPercentiles and HistogramsCumulative Operations

Core math operations

Linear Algebra

Matrix multiply, SVD, eigendecomposition — the core operations behind PCA, regression, and neural networks.

Matrix MultiplicationTranspose and InverseEigendecompositionSVD (Singular Value Decomposition)

Real-world data workflows

DataFrames, cleaning, and features

Pandas DataFrames

Labeled tabular data with method chaining — the standard for data wrangling in ML.

Series vs DataFrameMethod Chaining

Data Cleaning

Missing values, outlier detection, and imputation — preparing real-world data for models.

Missing Value StrategiesOutlier Detection

Feature Prep

Scaling, normalization, encoding — transforming raw features into model-ready inputs.

Feature ScalingOne-Hot Encoding

Topics

9 topics in this chapter

In the Python chapter you worked with ordinary lists and loops, but lists are a poor fit for numerical work: a list is a chain of pointers to separate objects, list + list concatenates instead of adding, and looping over a million numbers element by element is painfully slow. Yet every image, table, or signal in this plan is a grid of numbers you want to add, scale, and transform as one unit. That need is what NumPy arrays solve, and this topic is the foundation every later one -- indexing, broadcasting, reshaping, statistics -- builds on. You will first construct arrays from lists and from built-in constructors like zeros and arange, then learn to read an array's three defining properties: shape (the tuple of axis lengths), ndim (how many dimensions), and dtype (the type of each element). Finally you meet vectorization and universal functions, which apply arithmetic or math to every element in one operation instead of a Python loop, often 10-100x faster. Getting shape and dtype right up front prevents the most common bugs in image and ML code.

Definition

A NumPy array is a dense n-dimensional grid of values of a single element type, stored in one contiguous block of memory and addressed by an integer tuple called its shape. Its three defining properties are shape (the length of every axis), ndim (the number of axes, equal to len(shape)), and dtype (the element type). All operations are vectorized, each applying element-wise across the grid in one compiled pass.

In this topic

1Creating Arrays
2Array Shape and Dimensions
3Data Types (dtype)
4Special Array Constructors
5Vectorization
6Universal Functions (ufuncs)

1 of 6

Creating Arrays

$\texttt{np.array}([\text{data}], \texttt{dtype})$

Arrays are the core data structure of scientific Python: unlike a list of pointers, an ndarray stores elements of one type in one contiguous block of memory. np.array([data]) builds an array from a Python list, a nested list such as [[0,128],[255,64]] becoming a 2D array with one row per inner list. NumPy infers the dtype (element type) unless you pass dtype=..., e.g. dtype=np.float32. data may be any nested sequence, but it must be rectangular -- ragged rows raise an error. Constructors like zeros, ones, and randint build such arrays directly. This inherits Python's lists yet adds shape, dtype, and ndim, read in the next concept.

Mathematical Intuition

np.array allocates a single contiguous block of typed memory, unlike a list-of-lists, which stores pointers scattered across the heap; iterating the latter costs O(n) pointer-chases and cache misses. A contiguous float64 buffer lets the CPU prefetcher stream data at near-DRAM bandwidth, typically 10-50 GB/s. Construction from a list still copies every element, so np.array is O(n) in size and pays a conversion cost that constructors like zeros avoid by writing a known value directly.

Example:

Create a 640x480 grayscale image initialized to gray (128). Then create same-sized random noise image.

2 of 6

Array Shape and Dimensions

$\texttt{shape} = (d_0, d_1, ..., d_n)$

shape is a tuple (d_0,...,d_n) listing each axis's length; for an RGB image (480, 640, 3) the axes are height, width, and channels. ndim counts the axes -- 3 here -- and equals len(shape). size is the total element count, the product 4806403 = 921,600. dtype is the element type, e.g. uint8 or float32. Convention matters: numpy's image shape is (height, width, channels), the opposite of some graphics APIs, and shape[0] is the row (y) count, not the width. These properties are read in nearly every image or ML computation. This reads off what the Creating Arrays constructors attached.

Mathematical Intuition

shape is stored alongside a strides tuple giving the byte offset between consecutive elements along each axis. Accessing (i,j,k) computes the byte offset is_0 + js_1 + k*s_2 in O(1) -- pure pointer arithmetic on the flat buffer, no search or hash. A contiguous array has a fixed stride per axis (row stride equals row length times dtype size), so the address is exact even though no per-element index list exists. This bounded layout is why random access is as cheap as a contiguous scan, and it foreshadows how the reshaping concept later reinterprets the same bytes.

Example:

cv2.imread() returns shape (720, 1280, 3). What are height, width, channels? How many total pixels?

3 of 6

Data Types (dtype)

$\texttt{uint8, float32, float64, ...}$

dtype names the element type an array stores: uint8 (unsigned 8-bit integer, values 0-255, one byte per element), float32 (4-byte single precision, typical for neural networks normalized to 0-1), and float64 (8-byte double, numpy's default for precise calculation). Choosing dtype controls both memory and precision: uint8 is standard for images, float32 for model tensors, float64 for careful math. astype converts, e.g. img.astype(np.float32)/255.0 maps an image into 0-1. The failure mode is overflow: arithmetic on uint8 wraps modulo 256, so 250+10 becomes 4 instead of raising an error, silently corrupting brightened images. dtype is fixed at creation but readable anytime, connecting back to the dtype the Creating Arrays concept set.

Mathematical Intuition

uint8 uses 1 byte per element, float32 4, float64 8. A 1080p RGB image is 108019203 about 6M elements -- 6 MB in uint8 versus 24 MB in float32, a 4x memory saving at the price of integer-only precision. Smaller dtypes also raise cache hit rates because more elements fit in the L1/L2 lines, so a float32 pipeline can be several times faster than float64 purely from memory traffic, not fewer flops. The tradeoff is range: uint8 is capped at 255, which is the overflow failure mode.

Example:

Brighten uint8 image by adding 50. Pixel at 220 becomes 270 -- but uint8 max is 255! What happens? How to fix?

4 of 6

Special Array Constructors

$\texttt{zeros, ones, arange, linspace, eye}$

zeros and ones build arrays filled with 0 or 1, e.g. np.ones((3,3))*5 gives a 3x3 of fives, while np.full((3,3),7) fills any constant directly. arange produces a range such as np.arange(0,10,2) = [0,2,4,6,8] -- start, stop-exclusive, step -- ideal for coordinate indices. linspace returns evenly spaced points, np.linspace(0,1,5) = [0,0.25,0.5,0.75,1], guaranteeing the count including both endpoints, which arange cannot: arange(0,1,0.25) stops before 1. eye gives an identity matrix np.eye(3) and diag places a vector on the diagonal. empty allocates uninitialized memory fastest when every element will be overwritten. These are the fast, typed relatives of the list-building you met in Creating Arrays.

Mathematical Intuition

np.zeros and np.ones perform one memset-style fill over a contiguous buffer, so they run in O(n) at near-peak memory bandwidth; their cost is mostly the bytes written. np.arange emits values by computing start + i*step in compiled code, never materializing a Python int per element, unlike a list comprehension that allocates one object per value. np.linspace precomputes the step as (stop-start)/(num-1) and hits the endpoints exactly. This is why these constructors outpace equivalent Python loops by orders of magnitude even though the logical work looks identical.

Example:

Create gradient image: 256x256, where pixel value = column index (0-255 left to right). Horizontal gradient.

5 of 6

Vectorization

$\mathbf{c} = \mathbf{a} + \mathbf{b}$ where $c_i = a_i + b_i$

Vectorization means expressing an operation over an entire array at once rather than looping in Python. Here a and b are same-shaped operands, c the output, and the operation is element-wise: c_i = a_i + b_i for each index i. This is the payoff promised by Creating Arrays -- one dispatch over the whole buffer instead of a Python-level iteration per element, typically 10-100x faster. It fails when shapes do not align, or when you expect matrix multiplication, which is @, not this element-wise plus. Universal functions in the next concept generalize the very same idea to exp and log.

Mathematical Intuition

A Python for-loop pays roughly 100 ns of interpreter overhead per iteration. NumPy dispatches one C function over the whole buffer, processing several elements per SIMD instruction (AVX2 can handle 4-8 float64s at once). For n = 10^6 elements, loop overhead alone is about 100 ms versus roughly 1 ms for the vectorized call -- a factor of order O(n) in overhead that grows with array size, so vectorization matters more exactly when the data is large.

Example:

Compute squared L2 norm: sum(x_i^2) for x with 1 million elements. Compare: Python loop vs NumPy.

6 of 6

Universal Functions (ufuncs)

A ufunc (universal function) is an element-wise mathematical function numpy applies to every element of an array in one optimized C pass: np.exp, np.log, and np.sin are the common ones. np.exp(x) computes e^{x_i} for each element, the engine of softmax and logistic-sigmoid; np.log undoes exponentials; np.sin applies trigonometry. Unlike a Python loop that calls math.exp per value, a ufunc dispatches once over the whole buffer with a dtype-selected implementation, so it scales to millions of elements with no per-item overhead. They fail by overflow: np.exp(800.0) returns inf in float64. This generalizes the element-wise view from Vectorization to arbitrary functions.

Mathematical Intuition

Ufuncs are compiled C loops with a type-dispatch table that selects an implementation per dtype, so the same np.exp covers int, float32, and float64 without Python-level branching. np.exp uses a polynomial and rational minimax approximation from the Cephes library, accurate to about one ulp at float64. The practical boundary is exponential overflow: e^{710} is about 1.5e308, already near float64's maximum, and e^{800} returns inf. Softmax subtracts max(x) to keep every exponent at most 0, so the result stays finite and, being a scale-invariant shift, yields identical probabilities.

Example:

Compute softmax: exp(x_i) / sum(exp(x_j)) for array x = [1, 2, 3].

Code Examples

Building a gray image, a gradient, and reading shape

import numpy as np

gray = np.full((480, 640), 128, dtype=np.uint8)
grad = np.tile(np.arange(256), (256, 1)).astype(np.uint8)

print("shape", gray.shape, "| ndim", gray.ndim, "| size", gray.size, "| dtype", gray.dtype)
print("gray pixel", int(gray[0, 0]))
print("linspace", np.linspace(0, 1, 5))
print("grad corners", int(grad[0, 0]), int(grad[0, 255]), int(grad[255, 0]), int(grad[255, 255]))

Prints shape (480, 640), ndim 2, size 307200, dtype uint8; gray pixel 128; linspace [0. 0.25 0.5 0.75 1. ] and gradient corners 0 255 0 255, confirming that grad[r, c] equals the column index c and every row is the identical horizontal ramp.

Vectorized sum, uint8 overflow, and a stable softmax

import numpy as np

x = np.arange(1_000_000, dtype=np.float64)
print("sum of squares", f"{float(np.sum(x * x)):.5e}", "| dot", f"{float(x @ x):.5e}")

a = np.array([250], dtype=np.uint8)
print("uint8 250 + 10 wraps to", int((a + 10)[0]))

v = np.array([1.0, 2.0, 3.0])
e = np.exp(v)
print("exp", np.round(e, 2))
print("softmax", np.round(e / e.sum(), 2))

big = np.array([900.0, 905.0, 910.0])
z = big - big.max()
print("stable softmax", np.round(np.exp(z) / np.sum(np.exp(z)), 3))

The vectorized sum of squares and the dot product both equal 3.33333e+17, one compiled C pass over a million elements. uint8 250 + 10 wraps modulo 256 to 4 instead of 260. exp([1,2,3]) = [2.72 7.39 20.09] and the softmax = [0.09 0.24 0.67]. Finally, subtracting the max lets softmax of [900,905,910] evaluate stably to [0. 0.007 0.993] instead of overflowing to inf.

Theory Exercise

Problem:

You have an RGB image stored as a NumPy array. What are its shape, ndim, and total number of pixels if the image is 1920x1080?

Hints:

RGB images have 3 channels
Shape is (height, width, channels) in NumPy/OpenCV
ndim is the number of dimensions in the shape tuple

Papers

Array programming with NumPy (Harris et al., Nature 2020)

Blogs

NumPy Quickstart NumPy quickstart for absolute beginners — ndarray, shape, dtype 3Blue1Brown — But what is a neural network? (vectorized computation intuition)

Coding Exercise

Problem:

Create two 2D arrays of shape (3, 3) and compute their matrix multiplication using np.dot().

Hints:

Use np.array() or np.random.randn()
Matrix multiplication: C = A @ B

Definition

Indexing, slicing, and masking let you read or write selected parts of an ndarray by position. arr[i, j] indexes one element; arr[start:stop:step] slices a share-memory view of a contiguous region; arr[indices] with an integer or boolean array gathers arbitrary scattered elements into a fresh copy. The three modes trade flexibility against speed and memory sharing: basic and slice indexing return views, advanced indexing returns copies.

In this topic

1Basic Indexing
2Slicing Syntax
3Advanced Indexing
4Views vs Copies
5Boolean Indexing

1 of 5

Basic Indexing

$\texttt{arr[row, col]} = \text{single element}$

Basic indexing pulls a single element from an array. In arr[row, col], row counts down the first axis and col across the second, both zero-based, so arr[0, 0] is top-left and arr[-1, -1] bottom-right (negative indices count from the end). In NumPy's row-major layout each row's values sit contiguously in one flat buffer, so arr[row, col] is the element at flat position row * num_cols + col — found with no search. The only assumption: indices stay in range, else IndexError. Indexing an image img[y, x, c] addresses pixel $(x, y)$ , channel $c$ , building on the shapes met in the Array Creation & Operations topic.

Mathematical Intuition

Because arrays live in contiguous row-major memory, arr[row, col] is pure pointer arithmetic: the element sits at byte offset $\sum_k i_k \cdot \text{stride}_k$ from the buffer start, so a 100 MP image and a $2 \times 2$ array cost the same $O(1)$ to address one element. Positive and negative indices map to the same physical offset, which is why arr[0, 0] and arr[-1, -1] are both direct lookups rather than scans. This is the limiting case of slicing: a single element is just a slice of length one with the stride left unchanged.

Example:

RGB image shape (480, 640, 3). Get the green channel value at position x=100, y=50. Why is it img[50, 100, 1]?

2 of 5

Slicing Syntax

$\texttt{arr[start:stop:step]}$

A slice extracts a subarray along an axis. In arr[start:stop:step], start is the first index included, stop is the first index excluded, and step is the stride — the gap between kept positions, so ::2 keeps every other element. All parts are optional, defaulting to start=0, stop=end, step=1: arr[:3] is the first three elements, arr[-2:] the last two. Because a slice records a start position, per-axis strides, and a shape rather than copying values, it returns a view that shares the parent's memory — the subject of the later Views vs Copies concept. Its silent failure is forgetting that stop is exclusive, so arr[1:3] holds two elements, not three.

Mathematical Intuition

A slice builds a new ndarray header — shape, an updated data pointer, and per-axis strides — while still pointing into the parent's buffer, so its memory cost is $O(1)$ no matter how many elements it spans, and a megabyte-sized view allocates nothing at all. A non-unit step multiplies the axis stride: arr[::2] simply doubles it, which is why subsampling by a step is nearly free and why negative steps mirror the axis, reversing the elements without moving data. Strikingly, a slice can even be non-contiguous (like arr[:, ::2]) yet still share memory.

Example:

Crop center 100x100 region from 640x480 image. What slice expression?

3 of 5

Advanced Indexing

$\texttt{arr[indices]} \text{ or } \texttt{arr[bool\_mask]}$

Advanced indexing selects arbitrary elements no plain slice can describe. Fancy indexing passes positions: arr[[0, 2, 4]] gathers rows 0, 2, and 4 in one call though not contiguous. Boolean indexing passes a mask — a True/False array from a comparison like img > 128 — so img[mask] collects every element whose mask entry is True into a flattened 1D result; img[img < 50] = 0 writes only those slots. Because the chosen positions are scattered, advanced indexing returns a copy, not a view. It assumes valid, in-range indices that match or broadcast; else IndexError. It reuses Slice-topic coordinates, trading views for arbitrary selection.

Mathematical Intuition

Fancy indexing gathers values at arbitrary positions, so each of the $k$ requested indices is an independent random memory access rather than a walk along a stride. Random accesses miss the CPU cache far more often than sequential ones, so on large arrays fancy indexing is slower per element than a contiguous slice by roughly the cache-miss penalty — tens to hundreds of cycles. The output is a brand-new buffer sized $k$ , so the cost is $O(k)$ time and $O(k)$ memory. Boolean indexing is the data-dependent special case where $k$ equals the number of True entries, discovered only after a full scan of the mask.

Example:

Threshold image: set all pixels > 200 to white (255), all others to black (0). One line with boolean indexing?

4 of 5

Views vs Copies

$\text{slice} \rightarrow \text{view} \quad \text{fancy} \rightarrow \text{copy}$

A slice and an advanced index both return an ndarray, but differ in who owns the data. A view is a lightweight header — shape, strides, pointer to the parent's buffer — sharing the actual elements, so mutating the view mutates the original. A copy is an independent buffer, made with .copy() or produced by fancy and boolean indexing. The failure mode is nasty: roi = img[10:20]; roi[:] = 255 silently rewrites the image. The rule: basic slicing gives views, advanced indexing gives copies, and .copy() forces independence either way. This mirrors the buffer-sharing idea from the reshaping concept, where an array either owns its buffer or points into another's.

Mathematical Intuition

The view-versus-copy choice is an asymptotic memory trade: a view is built in $O(1)$ — one header object pointing into the existing buffer — while a copy allocates $O(n)$ fresh bytes and performs a memcpy over the whole extent. For a 4K RGB float32 image of roughly 24 MB, a view costs tens of nanoseconds versus a copy of a few milliseconds, a difference of about five orders of magnitude. The scalable intuition: when a downstream routine only reads, prefer the view; materialize a copy only when you must own or mutate data without side effects on the source.

Example:

You extract ROI, blur it, paste back: roi = img[10:50, 10:50]; roi = cv2.blur(roi, (5,5)). Original unchanged! Why?

5 of 5

Boolean Indexing

$\text{X[X > 0]}$ selects all positive elements

Boolean indexing filters an array through a parallel mask of True/False values, keeping only elements whose mask entry is True. X[X > 0] builds the mask by evaluating the comparison element-wise, then gathers the winners. It is the standard tool for data cleaning — dropping missing or negative values — and for feature selection. It accepts any dtype and broadcasts like arithmetic, but the mask must match the array's shape, else IndexError. Because the kept positions are scattered in memory, the result is always a copy. This reuses the mask idea from the Advanced Indexing concept and previews np.where logic (ReLU via np.maximum(0, X)).

Mathematical Intuition

Boolean indexing is a two-pass algorithm: it first walks the whole array in $O(n)$ to evaluate the mask and count the matches $k$ , then walks again to gather exactly those $k$ values into a fresh $k$ -element buffer. The result's shape is fully data-dependent — masked selection flattens the output to one dimension regardless of the source's shape, collapsing all axis structure. Because the kept positions are generally non-contiguous, the gathered output cannot share a stride with the source, which forces a copy. Selectivity $k/n$ therefore governs both cost and output structure: sparse masks yield tiny arrays, dense masks nearly the full data.

Example:

X = [[1, -2], [3, -4]]. Replace all negative values with 0.

Code Examples

Proving slice views share memory while fancy and boolean indexing copy

import numpy as np

# 4x4 row-major array of values 0..15
a = np.arange(16).reshape(4, 4)

# (1) A basic slice is a VIEW: it shares the parent's buffer.
view = a[1:3, 1:3]
view[0, 0] = -1
print("a[1,1] after slice mutation:", a[1, 1])   # -1

# (2) Fancy indexing returns a COPY: mutations do not reach a.
rows = a[[0, 3]]
rows[:] = 99
print("original row 0:", a[0].tolist())          # [0, 1, 2, 3]

# (3) Boolean indexing returns a flattened COPY of the matches.
mask = a % 3 == 0
print("values divisible by 3:", a[mask].tolist())  # [0, 3, 6, 9, 12, 15]

The first block shows the slice is a view: after setting view[0, 0] to -1, printing a[1, 1] yields -1 because the view and the array share one buffer. The second block shows fancy indexing copies: assigning 99 to every element of rows leaves row 0 as [0, 1, 2, 3], unchanged. The third shows boolean indexing returns a flattened copy holding exactly the values divisible by 3 — [0, 3, 6, 9, 12, 15] — with a[1, 1] already replaced by -1, which is not divisible by 3.

Theory Exercise

Problem:

Given a 100x100 grayscale image, write expressions to: (a) Get the top-left 10x10 region (b) Get every 10th pixel (subsample) (c) Set the border (outer 5 pixels) to white (255)

Hints:

Use [start:stop] for regions
Use [::step] for subsampling
Border = first/last rows/cols

Papers

Array programming with NumPy (Harris et al., Nature 2020)

Blogs

Indexing on ndarrays — NumPy Copies and Views — NumPy user guide A Visual Intro to NumPy and Data Representation (Jay Alammar)

Coding Exercise

Problem:

Create a 6x5 integer array. Use slicing, fancy indexing, and boolean indexing to select data, then prove that a basic slice is a view (not a copy) by mutating it and watching the original change.

Hints:

Build the array with np.arange(30).reshape(6, 5) so the values are easy to track.
A slice like arr[1:4, ::2] returns a view; assigning into it (slice[:] = -1) writes back to the parent array.
Compare with fancy indexing arr[[0, 2, 5]] and boolean indexing arr[arr % 2 == 0], both of which return COPIES.

Definition

Broadcasting is the set of rules NumPy uses to apply element-wise operations like + or * to arrays of different shapes. Two dimensions are compatible if they are equal or one of them is 1. Comparing shapes aligned from the right, each result dimension takes the larger of the pair, and a size-1 dimension stretches to match its neighbor as a view — so no data is copied.

In this topic

1Element-wise Operations
2Broadcasting Rules
3Common Broadcasting Patterns
4Universal Functions (ufuncs)

1 of 4

Element-wise Operations

$C_{ij} = A_{ij} \circ B_{ij}$

NumPy's default is element-wise: an operator applies between corresponding entries. With $A$ , $B$ equal-shaped, the result entry $C_{ij}$ depends only on $A_{ij}$ and $B_{ij}$ — no Python loop, one C call over millions of elements.

a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
a + b  # [5, 7, 9], a * b # [4, 10, 18], a ** 2 # [1, 4, 9]

This is not matrix multiplication: a * b pairs same-index entries, a @ b sums a row against a column. Failure mode: uint8 wraps silently, 270 becoming 14, so promote the dtype first — extending the vectorization from Array Creation & Operations.

Mathematical Intuition

Element-wise ops are memory-bandwidth-bound, not compute-bound. Adding two float32 arrays needs 1 FLOP but 12 bytes of memory traffic per element (read A, read B, write C). Peak throughput is $\text{bandwidth} / 12$ elements/sec, typically billions per second on modern hardware, so wall-clock time is set by main-memory speed rather than by the arithmetic itself: doubling the array length doubles the bytes moved and roughly doubles the time.

Example:

Blend two images: result = 0.7img1 + 0.3img2 (weighted average). Both are uint8. What's the issue?

2 of 4

Broadcasting Rules

$\text{shapes aligned from right}$

Broadcasting stretches smaller arrays to match larger ones without copying data. Think of it as automatic repetition: arr + 5 treats 5 as [5, 5, ...], and a (3, 1) column added to a (3, 4) array repeats rightwards. Rule: align shapes from the right; dimensions are compatible if equal, or if one is 1. A size-1 dimension stretches to its neighbour's size; two differing non-1 sizes raise ValueError. No copy is materialized — that dimension becomes a view with stride 0, so the CPU re-reads the same memory and a (1, n) row feeds an $m \times n$ result in $O(n)$ memory, reusing the strides from Indexing & Slicing.

Mathematical Intuition

Broadcasting avoids materializing the expanded array. For shapes $(m,1)$ and $(1,n)$ , the result is $(m,n)$ using $O(mn)$ compute but only $O(m+n)$ memory. The stride for the broadcast dimension is set to 0, so the CPU re-reads the same cache line repeatedly — very cache-friendly. A broadcast row against a tall matrix therefore streams the short operand from L1 cache at every output position, so the total work scales with the size of the result, not with any materialized intermediate.

Example:

Can you add: A shape (5, 3) + B shape (3,)? What about (5, 3) + (5,)?

3 of 4

Common Broadcasting Patterns

$\text{normalize, center, scale}$

Broadcasting turns common preprocessing into one-liners. Normalize to 0–1: img / 255.0 broadcasts one constant over every pixel. Per-channel centering: mean = img.mean(axis=(0, 1)) collapses height and width, leaving one value per color channel — shape (3,) — and img - mean broadcasts that vector against every pixel; the axis argument picks which dimensions to sum away. Feature standardization repeats the idea on a sample matrix X: X.mean(axis=0) and X.std(axis=0) give each column's average and its typical spread, and (X - mean) / std rescales every sample so each feature ends near mean 0, spread 1. Each is nested loops in plain Python; broadcasting makes it one expression.

Mathematical Intuition

Per-channel normalization subtracts a $(3,)$ mean from an $(H,W,3)$ image. The mean vector fits in a single cache line (24 bytes for float64). The operation is $O(H \times W \times 3)$ FLOPs with negligible extra memory, and it runs as a single pass over the buffer, so the cost scales linearly with the number of pixels — a few dozen milliseconds for a 4K frame, microseconds for a thumbnail.

Example:

ImageNet normalization: subtract mean [0.485, 0.456, 0.406] and divide by std [0.229, 0.224, 0.225] per channel. One line?

4 of 4

Universal Functions (ufuncs)

$\texttt{np.sin, np.exp, np.clip, ...}$

Universal functions — ufuncs like np.exp, np.log, np.cos, np.clip — are element-wise operations implemented in C, so they loop in C rather than Python and broadcast like the operators above. np.exp(x) raises $e \approx 2.718$ to each element, as softmax needs.

np.clip(img + 50, 0, 255)   # brighten, keep in [0,255]
np.where(img > 128, 255, 0) # threshold -> binary

np.clip bounds every element, a safe handoff to uint8; np.where picks per element like an array-valued if-else. The axis argument steers reductions: img.sum(axis=0) sums down the height, img.mean(axis=-1) averages the color channels. Failure mode: np.exp overflows to inf past about 709 in float64, so subtract the max first.

Mathematical Intuition

Clip compiles to a branch-free min/max using SIMD instructions: vminps + vmaxps on AVX2. For $n$ elements, np.clip is $O(n)$ with no branch mispredictions, making it ~5x faster than a Python loop with if-else per element; a mispredicted branch stalls the CPU pipeline each time the comparison flips, whereas vectorized min/max has no branch at all, so worst-case cost equals best-case cost regardless of where the values fall.

Example:

Apply gamma correction: output = input^gamma. For gamma=2.2, input in [0,255]. How to handle the math correctly?

Code Examples

Broadcast shapes, stride-0 views, and column centering

import numpy as np

col = np.array([10.0, 20.0, 30.0]).reshape(3, 1)   # (3, 1)
row = np.array([1.0, 2.0, 3.0, 4.0])               # (4,)
outer = col + row
print("(3,1) + (4,) ->", outer.shape)              # (3, 4)
print(outer)

big = np.broadcast_to(row, (3, 4))                 # stride-0 view
print("broadcast_to shares memory with row?", np.shares_memory(big, row))

data = np.array([[1., 2., 3.],
                 [4., 5., 6.],
                 [7., 8., 9.],
                 [10., 11., 12.]])
centered = data - data.mean(axis=0)                # (4,3) - (3,)
print("centered column means ~0:", np.allclose(centered.mean(axis=0), 0))

col, of shape (3,1), plus row, of shape (4,), broadcasts to (3,4): each row is one column value added onto the whole row, so row 0 is [11, 12, 13, 14], row 1 is [21, 22, 23, 24], row 2 is [31, 32, 33, 34]. Then np.broadcast_to(row, (3, 4)) builds a stride-0 view and np.shares_memory prints True, proving no copy was made. Finally a (4,3) data matrix is centered with data - data.mean(axis=0), and np.allclose on the per-column means prints True.

Theory Exercise

Problem:

Explain what happens when you add arrays with shapes (3, 1) and (1, 4). What is the result shape?

Hints:

Compare dimensions from right to left
A dimension of 1 can stretch to match any size
The result has the maximum size in each dimension

Papers

Array programming with NumPy (Harris et al., Nature 2020)The NumPy array: a structure for efficient numerical computation (van der Walt et al., 2011)

Blogs

Broadcasting — NumPy Eric's Broadcasting Document (visualizer) — SciPy wiki

Coding Exercise

Problem:

Center a 4x3 data matrix by subtracting the per-column mean using broadcasting, then scale each row by a 1D weight vector. Show how a (4,3) array combines with (3,) and (4,1) shapes without explicit loops.

Hints:

Column means come from data.mean(axis=0), giving shape (3,) which broadcasts across all rows.
To scale per row, reshape the weights to (4, 1) so they broadcast across the 3 columns.
Broadcasting aligns shapes from the right; dimensions must be equal or one of them must be 1.

Definition

Array manipulation re-labels or re-directs the memory buffer behind an ndarray without inventing new content: reshape groups the same elements under a new shape tuple, transpose permutes which axis is which, flatten/ravel collapse to one dimension, and stack/concatenate combine separate arrays into one. Every operation preserves all original values; only the layout description and possibly the allocation change.

In this topic

1Reshape
2Flatten and Ravel
3Transpose and Axes
4Stacking and Concatenating

1 of 4

Reshape

$\texttt{arr.reshape(new\_shape)}$

reshape changes only the shape tuple describing an array; the underlying values stay in the same order and memory. For 12 elements [0,1,...,11], arr.reshape(3, 4) labels rows of 4 and arr.reshape(2, 2, 3) wraps them into a 3D box — same data, new geometry. The new dimensions must multiply to 12 or NumPy raises a ValueError. Pass -1 in one slot and NumPy infers that dimension: arr.reshape(-1, 4) yields (3, 4). reshape avoids copying only when the array is contiguous — elements packed into one uninterrupted memory block, like the arange output in Array Creation. A transposed view is non-contiguous and forces a copy.

Mathematical Intuition

reshape is $O(1)$ when contiguous: NumPy writes a new shape tuple and recomputes the strides — the byte offsets between neighbours along each axis — without touching the buffer. A contiguous 12-float (3,4) array gets strides (32,8) bytes; that the first stride equals the second dimension times the element size is the signature of row-major layout. When the array is non-contiguous (as after a transpose), no stride tuple can describe the old memory order under the new shape, so reshape falls back to an $O(n)$ copy. .flags['C\_CONTIGUOUS'] tells you which path will run before you commit.

Example:

Flatten image (480, 640, 3) to feed into fully-connected layer. What's the flat shape? How to reshape?

2 of 4

Flatten and Ravel

$\text{ND} \rightarrow \text{1D}$

Both flatten() and ravel() collapse any array into a 1D vector; they differ in view versus copy. A view shares the original's memory buffer; a copy allocates fresh data. ravel() returns a view when the array is contiguous — elements packed into one uninterrupted block — so it costs $O(1)$ and mutating it changes the source; flatten() always copies, safe to modify. Traversal order matters. Row-major (C) order reads left-to-right, top-to-bottom, turning [[1,2],[3,4]] into [1,2,3,4]; this is NumPy's default, the arange fill order from Array Creation. Column-major (Fortran) order reads one column at a time, [1,3,2,4]; request it with 'F'. Non-contiguous input forces ravel() to copy.

Mathematical Intuition

ravel() makes a zero-copy attempt, pointing a fresh 1D header at the existing buffer, so its cost is a microsecond when strides already describe a linear traversal — true for contiguous arrays. flatten() always allocates and memcpys, $O(n)$ . On a ten-million-element array that is roughly 30 ns versus 20 ms, a factor of $10^5$ . Both accept an order flag, so ravel('F') streams a column-major walk; that is how data gets reshuffled into the storage layout that Fortran-written linear algebra kernels or BLAS expect, without ever leaving NumPy's C-level speed.

Example:

Compute histogram of image values. Why use ravel() instead of flatten() when passing to np.histogram()?

3 of 4

Transpose and Axes

$\texttt{arr.T} \text{ or } \texttt{transpose(axes)}$

Transpose reorders an array's axes by swapping the strides, the byte steps between neighbours along each axis. For a 2D image img = np.zeros((480, 640, 3)), img.T reverses all axes to (3, 640, 480), while img.transpose(2, 0, 1) names the new axis order, channels, height, width — the (C, H, W) layout PyTorch demands. In 2D this is the matrix transpose you will reuse in the Linear Algebra topic: rows become columns. Because no data moves, transpose returns a view (a concept from Indexing & Slicing), but the swapped strides make it non-contiguous, so row-wise reads jump across cache lines; call .copy() to restore contiguity before heavy computation.

Mathematical Intuition

A transpose is a stride permutation, $O(1)$ in time and memory: the $(3,4)$ row-major array with strides (32,8) bytes becomes (4,3) with strides (8,32). Reading a row afterward strides 8 bytes per element instead of scanning a contiguous line, so the CPU fetches many cache lines — a real slowdown on large arrays. Geometrically, transposing (2,3) yields (3,2), and the operation generalizes to any axis permutation; every permutation has an inverse, so transposing again with the inverse permutation restores the original layout. If the next kernel assumes contiguous data, materialize it with .copy() to let the prefetcher stream at memory bandwidth again.

Example:

OpenCV loads as (H, W, C) BGR. PyTorch expects (C, H, W) RGB. Convert in one line?

4 of 4

Stacking and Concatenating

$\texttt{stack, concatenate, hstack, vstack}$

concatenate joins arrays along an existing axis: two 1D arrays of length 3 become one of length 6, [1,2,3,4,5,6], copying every element. stack instead creates a new dimension, so two length-3 arrays become shape (2, 3) — it adds an axis and places each input along it, how you build a batch, the leading (B, ...) axis a network expects. axis=1 produces (3, 2). The 2D helpers vstack/hstack stack along axes 0 and 1. Every axis not being joined must match size between inputs (recall the shape-alignment rules from Broadcasting), or NumPy raises a ValueError. A common trick: np.stack([gray, gray, gray], axis=-1) attaches a channel axis.

Mathematical Intuition

concatenate allocates one output buffer of size $\sum_i n_i$ and memcpys each input in — $O(N)$ for $k$ arrays of total size $N$ , a single streaming pass. stack generalizes by inserting a unit axis first: $k$ arrays of shape $(n,)$ become $(k,n)$ , an $O(kn)$ copy in which every row is a contiguous chunk. The key geometric difference is rank: concatenate never changes dimensionality, while stack always increases it by one. Pass all inputs as a single list so NumPy sizes the output in one allocation; appending in a loop forces repeated reallocation and copying, turning an $O(N)$ job into something quadratic.

Example:

Create batch from list of 10 images, each (224, 224, 3). stack vs concatenate?

Code Examples

Which operations copy: views versus copies

import numpy as np

a = np.arange(12)
m = a.reshape(3, 4)          # contiguous

t = m.transpose(1, 0)        # non-contiguous view
print("flags C_CONTIGUOUS:", t.flags["C_CONTIGUOUS"])   # False

r_contig = m.ravel(); r_contig[0] = -1
print("ravel view shares memory:", m[0, 0] == -1)       # True

r_nc = t.ravel(); r_nc[0] = -2
print("ravel on transposed copied:", t[0, 0] != -2)     # True

f = m.flatten(); f[0] = -3
print("flatten independent:", m[0, 0] != -3)            # True

t2 = m.transpose(1, 0); t2[0, 0] = -4
print("transpose view writes through:", m[0, 0] == -4)  # True

This snippet proves which operations are views and which copy. A contiguous (3,4) array survives reshape as a view, but after transpose the C_CONTIGUOUS flag reads False. ravel() then stays a view on the contiguous input — writing -1 shows up at m[0,0], printing True — yet copies on the transposed input, where writing -2 leaves t[0,0] untouched, also printing True. flatten() always copies, while transpose writes through to the original, confirming it is a view. Every check prints True; only the first line prints False.

Stacking a batch, adding channels, and transposing

import numpy as np

# one grayscale 3x3 image
img = np.arange(9).reshape(3, 3)

# add a channel axis -> (H, W, C)
rgb = np.stack([img, img, img], axis=-1)
print("rgb shape:", rgb.shape)          # (3, 3, 3)
print("rgb value at (1,1):", rgb[1, 1])  # [4 4 4]

# batch: new leading axis -> (N, H, W)
batch = np.stack([img, img + 10], axis=0)
print("batch shape:", batch.shape)      # (2, 3, 3)
print("first row of sample 0:", batch[0, 0])   # [0 1 2]

# concatenate on existing axis 0 -> more rows, same rank
rows = np.concatenate([img, img + 10], axis=0)
print("concat rows shape:", rows.shape)  # (6, 3)

# transpose to channels-first
chw = batch.transpose(0, 2, 1)
print("channels-first shape:", chw.shape)  # (2, 3, 3)

Deep-learning array construction. Stacking three identical grayscale (3,3) arrays along axis=-1 builds a (3, 3, 3) RGB image, and rgb[1, 1] prints [4 4 4], three copies of the value at row 1, column 1. Stacking two (3,3) images along axis 0 makes a batch of shape (2, 3, 3) whose first row prints [0 1 2]. concatenate along axis 0 instead joins the existing rows to give (6,3) — 'new axis' versus 'existing axis' in one comparison. Finally transpose(0, 2, 1) reorders the batch, printing (2, 3, 3); the shape only coincides with the input because the last two axes were exchanged.

Theory Exercise

Problem:

You have a batch of 32 RGB images in PyTorch format: shape (32, 3, 224, 224). Convert to NumPy/OpenCV format (32, 224, 224, 3). Then extract just the red channel as a (32, 224, 224) array.

Hints:

Use transpose() to reorder dimensions
Channel is now the last dimension
Index into the channel axis to extract one channel

Papers

Array programming with NumPy (Harris et al., Nature 2020)The NumPy array: a structure for efficient numerical computation (van der Walt et al., 2011)

Blogs

Array Manipulation — NumPy The N-dimensional array (ndarray): strides and memory layout — NumPy

Coding Exercise

Problem:

Start from a 1D array of 24 elements. Reshape it to (4, 6) and (2, 3, 4), flatten it back with ravel vs flatten, transpose a 2D view, and stack two arrays both vertically and horizontally.

Hints:

Use reshape(2, 3, 4) and let one dimension be -1 (e.g. reshape(4, -1)) to have NumPy infer it.
ravel() returns a view when possible; flatten() always returns a copy — mutate one to see the difference.
np.vstack / np.hstack (or np.concatenate with axis=) combine arrays along an axis.

Definition

Statistical functions reduce an array — or each slice along a chosen axis — to descriptive numbers: the mean $\bar{x}$ (center), standard deviation $\sigma$ (spread), percentiles (robust quantiles), and cumulative sums. They compress thousands of values into a few interpretable numbers while the axis parameter controls exactly which dimension is collapsed.

In this topic

1Basic Statistics
2Axis Parameter
3Percentiles and Histograms
4Cumulative Operations

1 of 4

Basic Statistics

$\bar{x} = \frac{1}{n}\sum x_i \quad \sigma = \sqrt{\frac{1}{n}\sum(x_i - \bar{x})^2}$

A raw array holds thousands of numbers, too many to read; statistics compresses them into a few meaningful values. The mean, $\bar{x}=\frac{1}{n}\sum x_i$ , adds the $n$ elements and divides by $n$ — the data's center, e.g. average image brightness. The standard deviation, $\sigma=\sqrt{\frac{1}{n}\sum (x_i-\bar{x})^2}$ , measures spread: average the squared distances from the mean (that average is the variance), then take the square root. Low $\sigma$ means consistent values, high $\sigma$ wide variation, and bell-shaped normal data is described by these two: $\bar{x}$ sets the peak, $\sigma$ the width. .min()/.max() give extremes, .argmin()/.argmax() their flattened indices — handy for finding the brightest pixel.

Mathematical Intuition

np.mean sums in a single $O(n)$ pass using pairwise summation, cutting floating-point error from $O(n\epsilon)$ to about $O(\log n\,\epsilon)$ . np.std is two-pass — mean first, then squared deviations — which is stable but reads the array twice, and it uses the population divisor $n$ (a biased estimate), not the sample divisor $n-1$ you would want when estimating a population standard deviation from a sample.

Example:

Detect if image is mostly dark or bright. What statistics tell you this? What thresholds?

2 of 4

Axis Parameter

$\texttt{func(axis=k)} \rightarrow \text{reduce along axis k}$

A reduction must know which dimension to collapse, and axis supplies it: reducing along axis $k$ replaces that dimension with a single summary, keeping every other coordinate. For an RGB image of shape $(H,W,3)$ , img.mean(axis=(0,1)) averages over rows, then columns, per channel, yielding one mean per color (shape $(3,)$ ) — the per-channel normalization an ML pipeline needs. axis=-1 reduces the trailing dimension, collapsing the color axis into a per-pixel $(H,W)$ mean, while omitting axis flattens everything to one scalar. Choosing the wrong axis silently yields a different shape and value, so check the output. This extends the axis ordering used during reshaping into control over which data you summarize.

Mathematical Intuition

Reducing along axis $k$ of a shape $(d_0,\dots,d_{N-1})$ array yields shape $(d_0,\dots,d_{k-1},d_{k+1},\dots,d_{N-1})$ : every output element aggregates $d_k$ inputs, and the reduction reads all $\prod d_i$ entries, so cost is always $O(n)$ . Locality differs though: reducing the last axis streams contiguous rows in row-major memory and is fastest, while reducing the first axis jumps across strides and allocates per-position accumulators.

Example:

Batch of images (32, 224, 224, 3). Compute mean and std per channel across entire batch for normalization.

3 of 4

Percentiles and Histograms

$\texttt{percentile, histogram}$

The mean alone hides how values cluster, so percentiles and histograms reveal the distribution's shape. A percentile $q$ is the value below which $q$ percent of the data falls: the 50th, the median, splits the sample in half and, unlike the mean, is immune to outliers. np.percentile(arr, [25, 50, 75]) returns the quartiles and np.percentile(arr, 99) the near-maximum used for outlier detection. A histogram counts values per bin: np.histogram(arr, bins=256) gives counts[i], how many elements fall in bin $i$ ; two peaks (bimodal) reveal foreground and background, a flat histogram signals even contrast. Together they capture a shape that the mean and standard deviation of Basic Statistics cannot express.

Mathematical Intuition

np.percentile sorts a copy of the data in $O(n\log n)$ by default; requesting a single quantile can use introselect to reach $O(n)$ average time. np.histogram, by contrast, never sorts: it buckets each value with $\lfloor (x-\min)/w \rfloor$ in $O(n)$ once the bin width $w$ follows from the requested count and range. That is why histograms scale gracefully to millions of elements while many percentile queries do not.

Example:

Stretch image contrast: map the 5th percentile to 0 and 95th percentile to 255. Why 5th/95th not min/max?

4 of 4

Cumulative Operations

$\texttt{cumsum, cumprod, diff}$

Some statistics describe running prefixes rather than single values. np.cumsum computes prefix sums, so cumsum([1, 2, 3, 4]) gives [1, 3, 6, 10], where entry $k$ is the total of all elements up to $k$ . The cumulative distribution is a normalized cumsum of a histogram and is the lookup table histogram equalization applies. np.cumprod does the same with products; np.diff returns consecutive differences, diff([1, 4, 6]) → [3, 2], the change per step. A two-dimensional cumsum builds an integral image, letting you average any rectangular region in constant time. All of this builds on the histograms of the previous concept, turning per-bin counts into running totals.

Mathematical Intuition

np.cumsum builds prefix sums in one pass, $S_k=S_{k-1}+a_k$ , so along the reduced axis it is inherently sequential — each output depends on the previous — unlike element-wise ops that parallelize freely. Rounding accumulates as $O(n\epsilon)$ for a naive pass; using np.add.accumulate with a wider dtype, or cascaded summation, bounds the drift. A two-dimensional cumulative sum is exactly the integral image behind constant-time box filters of any size.

Example:

Implement histogram equalization using NumPy. Given grayscale image, output should have uniform histogram.

Code Examples

Reduction, axis, percentiles, histogram, and cumsum on one array

import numpy as np

arr = np.arange(10, 130, 10).reshape(3, 4).astype(float)
print("mean          :", arr.mean())
print("std           :", round(arr.std(), 3))
print("column means  :", arr.mean(axis=0))
print("row medians   :", np.median(arr, axis=1))
counts, edges = np.histogram(arr, bins=4)
print("hist counts   :", counts)
print("cumsum row 0  :", np.cumsum(arr[0]))

The array is the reshaped sequence 10, 20, ..., 120. Its overall mean is 65.0 (the middle of an arithmetic progression), its population standard deviation is about 34.52, and its column means are [50, 60, 70, 80] because each column steps up by 10. Row medians are the midpoints [25, 65, 105]. With bins=4 the 12 evenly spaced values split 3-3-3-3 into the histogram counts [3 3 3 3], and the prefix sum [10 30 60 100] of the first row shows how cumsum accumulates each new element.

A normal distribution: mean, standard deviation, and the empirical rule

import numpy as np

rng = np.random.default_rng(7)
data = rng.normal(loc=50.0, scale=8.0, size=10000)
mean = data.mean()
sd = data.std()
print("mean        :", round(float(mean), 2))
print("std         :", round(float(sd), 2))
print("frac in 1sd :", round(float((np.abs(data - mean) < sd).mean()), 3))
print("frac in 2sd :", round(float((np.abs(data - mean) < 2 * sd).mean()), 3))
print("median      :", round(float(np.percentile(data, 50)), 2))

rng.normal draws 10,000 values from a bell-shaped (Gaussian) distribution centered at 50 with standard deviation 8. The sample mean prints 49.9 and the standard deviation 7.95, both close to the values asked for, and the median prints 49.89 — nearly the mean, because a normal curve is symmetric. The 'empirical rule' holds too: the printed fractions are 0.679 within one standard deviation and 0.953 within two, against the textbook 0.683 and 0.954. That is what a standard deviation means in practice — roughly 68 percent of normal data lies within one of them of the center.

Theory Exercise

Problem:

Given an RGB image, compute: (a) overall mean brightness, (b) per-channel means, (c) the brightest pixel location.

Hints:

Convert to grayscale for overall brightness
axis=(0,1) for per-channel
np.unravel_index converts flat index to coordinates

Papers

Array programming with NumPy (Harris et al., Nature 2020)

Blogs

Statistics — NumPy Khan Academy — Statistics and probability (mean, variance, the normal distribution)3Blue1Brown — But what is the Central Limit Theorem?StatQuest — Mean, Variance and Standard Deviation, Clearly Explained

Coding Exercise

Problem:

Given a 5x4 array of exam-like scores, compute overall and axis-wise mean/std, the 25th/50th/75th percentiles, a small histogram, and a cumulative sum along an axis.

Hints:

Pass axis=0 to aggregate down columns and axis=1 to aggregate across rows.
np.percentile(arr, [25, 50, 75]) returns the three quartiles; the 50th equals the median.
np.histogram returns (counts, bin_edges); np.cumsum(arr, axis=1) accumulates along each row.

Definition

Linear algebra is the study of vectors and of the matrices that map vectors to vectors. A matrix multiplication applies such a transformation, its inverse undoes it, and eigendecomposition and the singular value decomposition factor a matrix into simpler rotation-and-scaling pieces whose diagonal scalars reveal the directions and amounts of stretching. NumPy exposes compact one-call routines for each.

In this topic

1Matrix Multiplication
2Transpose and Inverse
3Eigendecomposition
4SVD (Singular Value Decomposition)

1 of 4

Matrix Multiplication

$C = AB \quad C_{ij} = \sum_k A_{ik} B_{kj}$

Matrix multiplication applies transformations. Think of a matrix as a machine that transforms vectors: it can rotate them, scale them, shear them, or combine these effects.

A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])

C = A @ B        # Matrix multiply (apply B, then A)
C = np.dot(A, B) # Same thing

# DIFFERENT from element-wise multiply!
D = A * B  # Just multiplies corresponding elements

In CV/ML: Geometric transformations (rotation, scaling, projection) are all matrix multiplications. Neural network layers are matrix multiplies followed by non-linearities.

Mathematical Intuition

Matrix multiply of $(m,k) \times (k,n)$ costs $O(mkn)$ FLOPs. NumPy delegates to BLAS (OpenBLAS / MKL), which tiles the computation to fit in L1 cache and uses SIMD to approach the theoretical peak of ~100 GFLOPS on a modern CPU. For n = 1000, that is roughly two billion FLOPs in about 20 ms. That same product, recomputed at every layer of a network, is why matrix multiply dominates the runtime of deep learning.

Example:

Rotate point (x, y) by angle theta degrees. Build 2x2 rotation matrix and apply it.

2 of 4

Transpose and Inverse

$A^{-1} A = I$

Transpose (A.T) flips a matrix across its diagonal -- rows become columns. Think of it as mirroring across the main diagonal.

Inverse (A^-1) is the "undo" matrix. If A transforms a vector, A^-1 transforms it back. Not all matrices have inverses -- only square matrices with non-zero determinant.

A.T                  # Flip rows and columns
np.linalg.inv(A)     # Inverse (undo transformation)
np.linalg.pinv(A)    # Pseudo-inverse (works even when no true inverse)

# Solve Ax = b for x (better than x = inv(A) @ b)
x = np.linalg.solve(A, b)  # More numerically stable

In CV/ML: Pseudo-inverse is used when fitting transformations from noisy data (least-squares problems).

Mathematical Intuition

Matrix inversion via LU decomposition is $O(n^3)$ -- the same cost as multiplying two $n \times n$ matrices. Numerically, inverting a matrix with condition number $\kappa$ amplifies input error by a factor of $\kappa$ . When $\kappa$ exceeds 1e8 (float32) or 1e15 (float64), the result is unreliable; use np.linalg.solve instead, which computes the answer in the same $O(n^3)$ time without forming the inverse at all, so an ill-conditioned system keeps far more of its digits.

Example:

Find homography from 4 point pairs. You have 8 equations (2 per point), 8 unknowns. How to solve Ah=0?

3 of 4

Eigendecomposition

$A v = \lambda v$

Eigenvalues and eigenvectors reveal a matrix's "natural directions." An eigenvector is a non-zero vector whose direction the matrix leaves unchanged — multiplication only stretches, squeezes or flips it. The eigenvalue says how much: 2 doubles it, 0.5 halves it, -1 flips it.

A = np.array([[3, 1], [1, 3]])
eigenvalues, eigenvectors = np.linalg.eigh(A)  # For symmetric matrices
# eigenvalues: how much each direction is stretched
# eigenvectors: the special directions themselves

In CV/ML: PCA reads the eigenvectors of the covariance matrix, whose entry (i, j) records how features i and j vary together, to find the main directions of variation.

Mathematical Intuition

Eigendecomposition of an $n \times n$ symmetric matrix costs $O(n^3)$ via the QR algorithm (LAPACK dsyev). Because the matrix is symmetric the eigenvectors are orthonormal -- mutually perpendicular (each pair's dot product is zero) and each of unit length -- so they form a clean coordinate frame. Writing $A = V \Lambda V^T$ puts those eigenvectors as columns of $V$ and the eigenvalues on the main diagonal of $\Lambda$ , a diagonal matrix with zeros everywhere off the diagonal; multiplying by $\Lambda$ just scales each coordinate by its eigenvalue. Then $A^k = V \Lambda^k V^T$ turns a matrix power into powering its diagonal entries, which is why repeated applications in ML become cheap. PCA keeps the top $k$ eigenvectors, reducing dimension from $n$ to $k$ while retaining the most variance.

Example:

PCA for dimensionality reduction: 1000 images of 100x100 pixels. How to find principal components?

4 of 4

SVD (Singular Value Decomposition)

$A = U \Sigma V^T$

SVD decomposes any matrix into simpler factors — square or rectangular, $A = U \Sigma V^T$ . $U$ and $V$ are orthogonal, acting like rotations; $\Sigma$ is diagonal, its entries the singular values $s_1 \ge s_2 \ge \cdots \ge 0$ , each the stretch along one direction.

U, s, Vt = np.linalg.svd(A)
# U: what to rotate the input by; s: scale per direction, large to small
# Vt: what to rotate the output by
A_rebuilt = U @ np.diag(s) @ Vt   # equals A up to rounding

In CV/ML: keeping only the largest singular values rebuilds $A$ almost exactly — image compression, denoising, and reducing the covariance matrix above.

Mathematical Intuition

Full SVD of an $(m,n)$ matrix costs $O(\min(m,n)^2 \cdot \max(m,n))$ ; a truncated SVD keeping $k$ singular values costs only $O(mnk)$ with randomized algorithms (Halko et al.), so huge matrices become tractable. The squared singular values $\sigma_i^2$ sum to the matrix's squared length, so the ratio $\sum_{i=1}^k \sigma_i^2 / \sum_i \sigma_i^2$ is the fraction of "energy" the top $k$ directions capture. The Eckart-Young theorem says the rank- $k$ truncation $U_k \Sigma_k V_k^T$ is the closest rank- $k$ matrix to $A$ in any orthonormally-invariant norm, which is why discarding small singular values loses so little. Fast decay of $\sigma_i$ means the matrix is effectively low-rank and compresses well.

Example:

Denoise image using SVD: remove small singular values (assumed to be noise). How many to keep?

Code Examples

Eigendecomposition of a symmetric matrix

import numpy as np

A = np.array([[3.0, 1.0], [1.0, 3.0]])
vals, vecs = np.linalg.eigh(A)
print("eigenvalues:", vals)
print("recon A:", np.allclose(vecs @ np.diag(vals) @ vecs.T, A))
print("orthonormal:", np.allclose(vecs @ vecs.T, np.eye(2)))
print("A cubed:")
print(A @ A @ A)
print("A cubed via eigen:")
print(vecs @ np.diag(vals ** 3) @ vecs.T)

Build the symmetric matrix A = [[3.0, 1.0], [1.0, 3.0]] and call np.linalg.eigh(A). The eigenvalues print in ascending order as [2. 4.] (they sum to the trace 6 and multiply to the determinant 8). Because A is symmetric, the eigenvectors are orthonormal, so vecs @ vecs.T equals the identity (prints True), and vecs @ diag(vals) @ vecs.T reconstructs A (True). Direct cubing gives [[36. 28.] [28. 36.]], and the eigendecomposition route vecs @ diag(vals**3) @ vecs.T reproduces that same matrix (lambda 2 and 4 cubed are 8 and 64), demonstrating the identity A**3 = V Lambda**3 V.T.

SVD, energy, and a low-rank approximation

import numpy as np

rng = np.random.default_rng(7)
A = rng.normal(size=(8, 6))
U, s, Vt = np.linalg.svd(A, full_matrices=False)
print("singular values:", np.round(s, 3))
print("descending:", np.all(np.diff(s) <= 0))
print("recon A:", np.allclose((U * s) @ Vt, A))

energy_ratio = np.cumsum(s ** 2) / np.sum(s ** 2)
k = int(np.sum(energy_ratio < 0.95)) + 1
A_k = (U[:, :k] * s[:k]) @ Vt[:k, :]
print("k keeps >=95% energy:", k)
print("rel error:", round(float(np.linalg.norm(A - A_k) / np.linalg.norm(A)), 4))
print("energy ratio:", np.round(energy_ratio, 4))

Generate an 8-by-6 random matrix from numpy's default_rng(7) and compute its thin SVD with np.linalg.svd(..., full_matrices=False). The six singular values print as [4.058 3.259 2.712 1.475 1.325 0.814] in descending order (True), and (U * s) @ Vt reproduces A (True). The cumulative energy ratio cumsum(s**2)/sum(s**2) is [0.4218 0.694 0.8824 0.9381 0.983 1.]; the smallest k whose ratio reaches 0.95 is k=5, and the relative error of that rank-5 approximation is 0.1303. Dropping just the weakest singular value (0.814) already loses under 2% of the energy.

Theory Exercise

Problem:

Explain why we use SVD for image compression. If an image is 1000x1000, how much space does keeping the top 50 singular values use compared to the original?

Hints:

Original: 1000 x 1000 values
SVD: U is 1000x50, s is 50, Vt is 50x1000
Calculate total storage for each

Papers

The approximation of one matrix by another of lower rank (Eckart & Young, 1936)Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions (Halko, Martinsson, Tropp, 2011)

Blogs

Linear Algebra — NumPy 3Blue1Brown — Eigenvectors and eigenvalues (Essence of Linear Algebra)StatQuest — Covariance, Clearly Explained

Coding Exercise

Problem:

Build a small square matrix and demonstrate matrix multiply, transpose, inverse (verify A @ A_inv ≈ I), eigendecomposition, and SVD; confirm SVD reconstructs the original matrix.

Hints:

Use the @ operator (or np.matmul) for matrix products, and A.T for the transpose.
np.linalg.inv(A) gives the inverse; A @ inv should be close to np.eye — check with np.allclose.
np.linalg.eig returns (eigenvalues, eigenvectors); np.linalg.svd returns U, s, Vt that recombine as U @ diag(s) @ Vt.

Definition

A pandas DataFrame is a two-dimensional, labeled table: each column is an ordered Series — a one-dimensional, single-dtype array with an index of row labels — and all columns share that same index, so rows are addressed by label rather than by position. A Series is one column, whose values and labels you can select, filter, aggregate, and transform directly.

In this topic

1Series vs DataFrame
2Method Chaining

1 of 2

Series vs DataFrame

DataFrames let you name data instead of counting positions. A Series is a one-dimensional labeled array: single-dtype values plus an index, the row labels like 'Alice', 'Bob', 'Carol' identifying each value. A DataFrame is a collection of aligned Series (columns like name, age, city) sharing that index, each column stored as its own contiguous NumPy array — a columnar layout. Pull out a column with df['age'] and you hold a Series; keep the whole frame and you have the DataFrame. It fails when labels disagree: aligning two Series of unequal length fills gaps with NaN. Compare with Indexing — there position is everything; here the label carries the meaning.

Mathematical Intuition

A pandas Series wraps a single NumPy array plus an index. A DataFrame stores each column as a separate contiguous array, so column access is $O(1)$ (return the array), but row access requires gathering one element from each column — $O(\text{num\_cols})$ with potential cache misses. This columnar layout mirrors analytical databases, which is why whole-column reductions like sum or mean stay nearly as fast as pure NumPy.

Example:

You have ages = [25, 30, 35] and names = ['Alice', 'Bob', 'Carol']. When to use Series vs DataFrame?

2 of 2

Method Chaining

Every pandas operation that returns data returns a fresh object exposing the same methods, so you can string calls into one expression — method chaining: df.dropna().groupby('city')['signups'].mean().sort_values(), read left to right with no intermediate variables. Here groupby('city') partitions rows by the city key, ['signups'] selects the column, .mean() aggregates each city's rows into one number, and sort_values orders them — a labeled cousin of the per-column reductions from the Statistics topic, but split by category first. It breaks when a stage returns an object lacking the next method, so track each intermediate type. For custom logic, insert any function with .pipe().

Mathematical Intuition

Each chained method returns a new DataFrame, allocating $O(n \cdot m)$ memory for $n$ rows and $m$ columns. For large datasets, long chains can peak at several copies in flight, so drop columns you no longer need and let the temporaries be garbage-collected. The .pipe() method does not reduce copies but improves readability. For memory-critical work, use inplace operations or work directly with the underlying NumPy arrays.

Example:

Calculate average salary by department, excluding missing values, sorted descending. Do it in one chain.

Code Examples

Series vs DataFrame and columnar grouping

import pandas as pd

df = pd.DataFrame({
    "city": ["NY", "SF", "NY", "SF", "NY"],
    "age": [23, 35, 31, 29, 42],
    "signups": [5, 12, 8, 3, 20],
})

col = df["signups"]                       # extracting one column
print("column type:", type(col).__name__)   # Series
print("frame type:", type(df).__name__)     # DataFrame
print("mean signups:", col.mean())          # 9.6
print("index label 0:", df.index[0])        # 0

# columnar layout: reduce a whole column, then group by category
overall = df["signups"].mean()
by_city = df.groupby("city")["signups"].mean()
print("overall:", overall)                    # 9.6
print("NY ->", by_city["NY"], "| SF ->", by_city["SF"])   # 11.0 | 7.5
print("NY above overall:", bool(by_city["NY"] > overall))  # True

Extracting df['signups'] returns a Series (prints 'Series') while df itself is a DataFrame. col.mean() reduces the whole column in one vectorized pass to 9.6. df.index[0] shows the raw row label 0. Reducing overall then again per group shows the columnar layout: the groupby splits on the city key and averages signups, printing NY -> 11.0 and SF -> 7.5, and 'NY above overall' prints True because 11.0 exceeds the global 9.6.

Papers

Data Structures for Statistical Computing in Python

Blogs

10 Minutes to pandas DataFrames and Series — pandas user guide

Coding Exercise

Problem:

Create a DataFrame of users with name, age, city, and signups. Show that a single column is a Series, then use method chaining (filter -> assign a derived column -> sort -> group/aggregate) to summarize average signups per city.

Hints:

df['age'] returns a Series; type(df['age']) is pandas.Series while df is a DataFrame.
Chain methods on one expression: df.query(...).assign(...).sort_values(...) without intermediate variables.
Use .groupby('city')['signups'].mean() to aggregate, then .reset_index() to get a tidy DataFrame back.

Before you load a DataFrame into a model, you have to trust the numbers in it — and real-world data rarely earns that trust. Columns arrive with holes where a survey respondent skipped a question or a sensor went dark, and with a few extreme values that are either genuine rare events or plain typos. If you feed raw missing cells and stray outliers into the statistics and vectorized tools from the Pandas DataFrames topic, the damage is silent: a single ten-million-dollar salary can drag a column mean across a threshold that matters, and one NaN can turn an entire mean() into another NaN. This topic repairs the data before any analysis. We first handle missing values, deciding when to drop a row or column and when to fill (impute) the gap with a sensible substitute. Then we hunt outliers with the z-score and the interquartile range. Both techniques share one goal that every later topic depends on: a clean, defensible array. Because cleaning choices change results, you make them deliberately — never accidentally.

Definition

Data cleaning is the process of detecting and repairing defects in a dataset before analysis: replacing or removing missing values, and identifying or adjusting values that sit implausibly far from the rest of the distribution, so that computed statistics and fitted models describe the underlying signal rather than the errors poking through the gaps.

In this topic

1Missing Value Strategies
2Outlier Detection

1 of 2

Missing Value Strategies

Real data arrives with empty cells, which pandas represents as NaN (Not a Number), a floating-point placeholder. It is dangerous: most NumPy and pandas reductions propagate it, so one gap turns mean(), sum(), or a fitted model into NaN. The first strategy is to drop rows (dropna()) or whole columns when too much is missing. Otherwise we impute: fill each gap with an estimate. The median (the middle value after sorting) is usually safer than the mean (the arithmetic average) because outliers drag the mean while they skip past the median; SimpleImputer(strategy='median') codifies that choice. Every imputed value is a guess, so treat the repaired column as an estimate.

Mathematical Intuition

Mean imputation replaces each NaN with the column mean $\bar{x}$ , preserving the column's mean exactly (that is its appeal) but shrinking the observed variance by a factor of $(1 - f)$ , where $f$ is the fraction missing: with fewer deviations from $\bar{x}$ surviving, the measured spread looks smaller than reality, and that bias leaks into every covariance and correlation you later compute. KNN imputation is $O(n^2 d)$ for $n$ samples and $d$ features — quadratic in rows, so it is expensive on large tables, but in exchange it borrows from actual neighboring rows and preserves local structure.

Example:

Column 'income' has 5% NaN. Column 'rare_feature' has 80% NaN. What strategy for each?

2 of 2

Outlier Detection

$\text{outlier if } |x - \mu| > 3\sigma$

Clean arrays still hide outliers, values that sit implausibly far from their neighbors. The z-score counts how many standard deviations a value sits from the mean: for value $x$ , mean $\mu$ , and standard deviation $\sigma$ (the typical spread), $z=(x-\mu)/\sigma$ , so the rule $|x-\mu|>3\sigma$ means $|z|>3$ . In a normal (bell-curve) distribution about 99.7% of values lie within $3\sigma$ of $\mu$ , so only rare points are flagged. Because $\mu$ and $\sigma$ bend toward the extreme value, the IQR method is more robust: sort, take quartiles $Q_1$ (25th percentile) and $Q_3$ (75th), and fence at $Q_1-1.5\cdot\text{IQR}$ and $Q_3+1.5\cdot\text{IQR}$ , with $\text{IQR}=Q_3-Q_1$ . Flagged points get capped, removed, or investigated — often a typing error.

Mathematical Intuition

The IQR method flags points outside $[Q_1 - 1.5\cdot\text{IQR},\; Q_3 + 1.5\cdot\text{IQR}]$ ; for a Gaussian this interval captures 99.3% of observations, while the $|z|>3$ rule captures 99.7%, so at sample size $n$ both expect fewer than roughly $n/100$ false alarms. Each method is $O(n)$ per feature once the quartiles are found by sorting. Multivariate data is subtler: Mahalanobis distance accounts for correlations between features but requires inverting the covariance matrix, an $O(d^3)$ cost for $d$ features — worth it only when features move together.

Example:

Salary data: most values 50K-150K, but one value is 10M. Z-score method vs IQR method?

Code Examples

Imputation changes the column statistics

import numpy as np
import pandas as pd

df = pd.DataFrame({"salary": [50, 52, 49, np.nan, 51, 530]})

mean_imp = df["salary"].fillna(df["salary"].mean())
med_imp = df["salary"].fillna(df["salary"].median())

raw = df["salary"].dropna()
print("raw            mean %.2f  median %.2f  std %.2f" % (raw.mean(), raw.median(), raw.std(ddof=0)))
print("mean-imputed   mean %.2f  std %.2f" % (mean_imp.mean(), mean_imp.std(ddof=0)))
print("median-imputed mean %.2f  std %.2f" % (med_imp.mean(), med_imp.std(ddof=0)))
print("NaNs remaining:", int(mean_imp.isna().sum()))

Filling the single NaN with the column mean keeps the mean at 146.40 but shrinks the standard deviation from about 191.80 to 175.09 — the (1 - f) variance shrinkage from the math intuition, since one sixth of the values no longer deviate from the mean. Filling with the median (51.0) does not preserve the mean (it drops to 130.50) but is robust to the extreme 530. Both repaired columns leave zero NaNs.

z-score and IQR outlier detection

import numpy as np

x = np.array([60, 62, 58, 61, 59, 57, 63, 60, 900], dtype=float)

mu, sigma = x.mean(), x.std(ddof=0)
z = (x - mu) / sigma
print("mean %.2f  std %.2f" % (mu, sigma))
print("z-score of the 900: %.2f" % z[-1])

q1, q3 = np.percentile(x, [25, 75])
iqr = q3 - q1
lo, hi = q1 - 1.5 * iqr, q3 + 1.5 * iqr
print("Q1 %.1f  Q3 %.1f  IQR %.1f  fences [%.1f, %.1f]" % (q1, q3, iqr, lo, hi))
print("z-score outliers:", x[np.abs(z) > 3].tolist())
print("IQR outliers:", x[(x < lo) | (x > hi)].tolist())

clean = x[:-1]
q1c, q3c = np.percentile(clean, [25, 75])
print("fences without the 900: [%.1f, %.1f]" % (q1c - 1.5 * (q3c - q1c), q3c + 1.5 * (q3c - q1c)))

The single 900 pulls the mean up to 153.33 and inflates the standard deviation to 263.99, so the extreme value's own z-score is only 2.83 — below the 3 threshold, and the z-score flags nothing at all. The IQR method is untouched by the extreme: with Q1=59 and Q3=62 the IQR is 3, the fences land at 54.5 and 66.5, and the 900 is cleanly flagged. Dropping that point moves the fences only slightly, to [55.0, 65.0], because quartiles are median-based — IQR's answer does not depend on the very outlier it is hunting.

Papers

Inference and Missing Data (Rubin, Biometrika 1976)mice: Multivariate Imputation by Chained Equations in R (van Buuren & Groothuis-Oudshoorn, 2011)

Blogs

Working with Missing Data — pandas Khan Academy — Constructing a box and whisker plot (quartiles and the IQR)

Coding Exercise

Problem:

Given a DataFrame with missing values, replace NaN values in numerical columns with the median and in categorical columns with the mode.

Hints:

Use df.select_dtypes() to separate column types
Use fillna() with appropriate statistics

Definition

Feature preparation transforms raw features into the numerical form a model can learn from. Numeric columns are rescaled so their magnitudes are comparable, and categorical columns are expanded into binary indicator columns. Scaling uses standardization, $x \mapsto (x-\mu)/\sigma$ , or min-max normalization into $[0,1]$ ; encoding represents each of $k$ category levels by a distinct column that holds 1 when that row matches and 0 otherwise.

In this topic

1Feature Scaling
2One-Hot Encoding

1 of 2

Feature Scaling

$x_{\text{scaled}} = \frac{x - \mu}{\sigma}$ (standardization)

Features often sit on wildly different scales — ages in decades, incomes in tens of thousands — and a model on raw values lets the larger-magnitude feature dominate. Parameters are fitted by gradient descent, which repeatedly nudges each one in the direction that most reduces the loss, so a huge-valued feature takes larger steps and steers the walk while a small-valued one is ignored. Standardization applies the z-score $z=(x-\mu)/\sigma$ , with $\mu$ the mean and $\sigma$ the standard deviation from the axis reductions of earlier topics, leaving the column centered at 0 with spread 1. Min-max normalization maps into $[0,1]$ via $(x-\min)/(\max-\min)$ : bounded, but outlier-sensitive. Fit either on the training split.

Mathematical Intuition

Scaling is really about the geometry of the loss landscape. With unscaled features the landscape is a long, narrow valley: along the steep direction a fixed-size gradient step overshoots, while along the shallow direction it crawls, so descent zigzags. Rescaling reshapes that valley into a roughly circular basin where every step helps on all axes at once. Concretely, convergence takes $O(\kappa)$ iterations, where $\kappa$ is the condition number — the ratio of the largest to the smallest curvature (eigenvalues of the Hessian). Making feature magnitudes comparable typically shrinks $\kappa$ by orders of magnitude, turning an impractically slow walk into a fast, stable one.

Example:

Features: age (0-100) and income (0-1M). Training logistic regression without scaling. What happens?

2 of 2

One-Hot Encoding

A categorical feature takes one value from a fixed set of labels — 'red', 'blue', 'green' — unlike a numerical feature, an amount like 5.4 with arithmetic meaning. Mapping labels to integers 0, 1, 2 invents an ordering the data lacks. One-hot encoding gives each category its own binary column: $k$ categories become $k$ indicator columns, each 1 when the row's label matches, so 'red' is $[1,0,0]$ and 'green' $[0,0,1]$ — one active column per row, no implied order. pd.get_dummies does this directly; in NumPy you allocate an $n \times k$ zero array and set one 1 per row. The failure mode is dimensionality: $k$ explodes for high-cardinality columns.

Mathematical Intuition

Encoding $k$ levels as columns shifts the cost from storage to dimensionality. The dense one-hot matrix is $n\times k$ — $O(nk)$ numbers even though only $n$ of them are nonzero — so for $k$ in the thousands one stores almost entirely zeros. Because each row has exactly one nonzero, the sparse (CSR) representation stays $O(n)$ : it keeps only the column index of that single 1 per row. Geometrically, one-hot places the $k$ categories at the vertices of a regular $k$ -simplex, every pair at the same distance $\sqrt{2}$ , so no category is treated as closer to another — unlike the integer mapping 0,1,2. The columns are linearly dependent (each row sums to 1), a redundancy some regularized models flag; as a memory rule of thumb, go sparse when $k>50$ .

Example:

Feature 'city' has 1000 unique values. One-hot encode? What problems arise?

Code Examples

Scaling features with NumPy

import numpy as np

# Two features on very different scales: age (tens) vs income (tens of thousands)
X = np.array([[35.0, 60000.0], [50.0, 30000.0], [22.0, 90000.0]])

# Standardization (z-score): each column -> mean 0, std 1
mu = X.mean(axis=0)
sigma = X.std(axis=0)
Z = (X - mu) / sigma
print("z means:", np.round(Z.mean(axis=0), 6))
print("z stds :", np.round(Z.std(axis=0), 6))

# Min-max normalization: each column -> [0, 1]
lo = X.min(axis=0)
hi = X.max(axis=0)
M = (X - lo) / (hi - lo)
print("mm min :", np.round(M.min(axis=0), 6))
print("mm max :", np.round(M.max(axis=0), 6))

Standardize each column to zero mean and unit standard deviation, then min-max normalize each column into [0, 1]. The standardized columns print means near [0, 0] and standard deviations near [1, 1]; the min-max columns span exactly [0, 0] at their minimum and [1, 1] at their maximum, regardless of the original 60000-versus-35 scale gap.

One-hot encoding a categorical column

import numpy as np
import pandas as pd

colors = np.array(["red", "green", "blue", "red", "blue"])
df = pd.DataFrame({"color": colors})
print("dtype:", df["color"].dtype)

onehot = pd.get_dummies(df["color"])
print("shape:", onehot.shape)
print("column sums:", onehot.sum().to_numpy())
print("rows sum to 1:", bool((onehot.sum(axis=1) == 1).all()))

Convert a categorical column into one binary column per category. Expect the printed shape (5, 3) — 5 rows and 3 unique colors; column sums [2, 1, 2] matching the counts of blue, green, and red in that alphabetical column order; and each row summing to 1 (True), confirming exactly one column is active per sample.

Papers

Scikit-learn: Machine Learning in Python Entity Embeddings of Categorical Variables

Blogs

Preprocessing — scikit-learn StatQuest — Gradient Descent, Step-by-Step

Coding Exercise

Problem:

Take a small tabular dataset with two numeric features and one categorical column. Standardize one feature to mean 0 / std 1, min-max normalize the other to [0, 1], and one-hot encode the categorical column with pd.get_dummies; verify the resulting shapes and stats.

Hints:

Standardize with (x - x.mean()) / x.std(); normalize with (x - x.min()) / (x.max() - x.min()).
After standardizing, the column mean should be ~0 and std ~1; after min-max, values lie in [0, 1].
pd.get_dummies(df['color']) creates one binary column per category — check that the new column count equals the number of unique categories.

Array Creation

Indexing & Slicing

Broadcasting

Reshaping

Statistics

Linear Algebra

Pandas DataFrames

Data Cleaning

Feature Prep

9 topics in this chapter

Array Creation & Operations

Definition

In this topic

Creating Arrays

Array Shape and Dimensions

Data Types (dtype)

Special Array Constructors

Vectorization

Universal Functions (ufuncs)

Code Examples

Theory Exercise

Problem:

Hints:

Coding Exercise

Problem:

Hints:

Related Problems on PixelBank

Indexing & Slicing

Definition

In this topic

Basic Indexing

Slicing Syntax

Advanced Indexing

Views vs Copies

Boolean Indexing

Code Examples

Theory Exercise

Problem:

Hints:

Coding Exercise

Problem:

Hints:

Related Problems on PixelBank

Operations & Broadcasting

Definition

In this topic

Element-wise Operations

Broadcasting Rules

Common Broadcasting Patterns

Universal Functions (ufuncs)

Code Examples

Theory Exercise

Problem:

Hints:

Coding Exercise

Problem:

Hints:

Related Problems on PixelBank

Reshaping & Manipulation

Definition

In this topic

Reshape

Flatten and Ravel

Transpose and Axes

Stacking and Concatenating

Code Examples

Theory Exercise

Problem:

Hints:

Coding Exercise

Problem:

Hints:

Related Problems on PixelBank

Statistical Functions

Definition

In this topic

Basic Statistics

Axis Parameter

Percentiles and Histograms

Cumulative Operations