Loading...
Understanding the fundamentals of computer vision: what it is, how digital images work, historical context, and the mathematical foundations you'll need throughout this course.
Welcome to Computer Vision! This chapter establishes the foundation for everything that follows.
What is this chapter about? We'll answer the fundamental questions: What is computer vision? Why is it hard? How do computers see images? This context is essential before diving into algorithms.
Why does this matter? Understanding the challenges and capabilities of computer vision helps you choose the right approach for real problems. Knowing how images are represented lets you manipulate them effectively.
How the topics connect: We start by defining computer vision and its applications—understanding what problems we're solving. Then we dive into digital images—how they're represented as arrays of numbers. We trace the history of the field to understand why modern approaches work. Finally, we cover key mathematical concepts that appear throughout the course.
Click any topic to jump in
The vision pipeline, applications, and core challenges — why making computers see is fundamentally hard.
Pixels and coordinate systems
Pixel intensity, resolution, color models — images as mathematical functions on a grid.
Row-major ordering, coordinate transforms, and pixel center conventions.
Vectors, matrices, homogeneous coordinates — the mathematical language of computer vision.
Before learning how to do computer vision, we need to understand what it is and why it's challenging. This topic provides the big picture context that will guide your learning throughout the course.
Computer Vision is a field of artificial intelligence that enables computers to interpret and understand visual information from the world, such as images and videos. Just as humans use their eyes and brain to see and understand their surroundings, computer vision systems use cameras and algorithms to perform similar tasks.
The goal of computer vision is to automate tasks that the human visual system can do—recognizing objects, understanding scenes, tracking motion, and making decisions based on visual input.
Computer vision systems typically follow a processing pipeline:
Classical vs Deep Learning Pipelines:
Real-time Considerations: Each stage adds latency. For 30fps video, the entire pipeline must complete in ~33ms. This drives architectural choices like running detection on keyframes and tracking between them.
Think of it like how you read a sign: your eyes capture light (acquisition), your brain filters out irrelevant details (preprocessing), recognizes letter shapes (feature extraction), understands the word (analysis), and you act on it (decision).
Each pipeline stage is a function composition: . The output of one stage becomes the input to the next. Classical pipelines use hand-crafted functions; deep learning replaces the middle stages with a single learned function parameterized by neural network weights.
A face recognition system captures a photo, detects the face, extracts features, and matches to a database. Which pipeline stage handles each step?
Computer vision is used across virtually every industry:
Transportation & Mobility: • Autonomous Vehicles: Perception for self-driving (Tesla, Waymo) • Traffic Monitoring: Congestion analysis, incident detection
Healthcare: • Medical Imaging: Detecting tumors, diabetic retinopathy screening • Surgical Robotics: Visual guidance for minimally invasive procedures
Commerce & Retail: • Visual Search: Find products by photo (Google Lens, Pinterest) • Cashierless Stores: Amazon Go uses CV for checkout-free shopping
Security: • Facial Recognition: Access control, surveillance • Anomaly Detection: Identifying unusual behavior in video feeds
Creative & Entertainment: • AR/VR: Spatial mapping, object placement, hand tracking • Film VFX: Motion capture, rotoscoping, face replacement
Industry 4.0: • Quality Control: Automated defect detection on production lines • Robotics: Pick-and-place, bin picking with vision guidance
Most CV applications reduce to a few core mathematical problems: classification maps an image to a label vector , detection adds spatial coordinates , and segmentation produces a per-pixel label map . Recognizing which formulation fits your task is the first step.
Classify these CV tasks by application area: (1) Detecting scratches on smartphone screens, (2) Adding virtual furniture to room photos, (3) Counting cells in a microscope image.
Why is computer vision hard? The challenges fall into several categories:
Appearance Variation: • Viewpoint: Objects look different from different angles • Scale: Objects can be near or far from the camera • Illumination: Lighting changes appearance dramatically • Deformation: Non-rigid objects change shape (people, animals)
Scene Complexity: • Occlusion: Objects can be partially hidden behind other objects • Background Clutter: Complex backgrounds make object detection difficult • Multiple Instances: Crowded scenes with many overlapping objects
Semantic Gaps: • Intra-class Variation: Objects in the same category look very different (all types of chairs) • Fine-grained Recognition: Distinguishing similar categories (dog breeds, bird species) • Context Understanding: A stop sign means different things in different contexts
The Fundamental Problem: Images are 2D projections of a 3D world. We lose depth information, and many 3D scenes can produce the same 2D image. Humans handle these ambiguities using context and prior knowledge—teaching computers to do so remains an active research challenge.
The core difficulty is the many-to-one nature of projection: infinitely many 3D scenes produce the same 2D image. Mathematically, the mapping is not injective — recovering 3D from 2D is an ill-posed inverse problem requiring priors or additional constraints.
A pedestrian detection system fails when a person is partly behind a parked car. Which challenge category is this?
A self-driving car needs to stop at a red traffic light. List at least 5 challenges the computer vision system might face when trying to detect the traffic light, and briefly explain why each is a challenge.