Autonomous Vehicle Decision System
Design the ML decision-making pipeline for a self-driving car: perception, prediction, and planning.
Scenario: An autonomous vehicle company is building the software stack that takes raw sensor data (cameras, LiDAR, radar) and outputs steering, acceleration, and braking commands. The system must operate safely in complex urban environments at speeds up to 65 mph, handling pedestrians, cyclists, construction zones, and adverse weather.
Your Task: Design the three core ML subsystems: perception, prediction, and planning.
Your design should address:
- Fusing multi-sensor data into a unified world model
- Detecting and tracking all road actors (vehicles, pedestrians, cyclists)
- Predicting future trajectories of surrounding agents
- Planning a safe, comfortable, and efficient driving path
- Handling edge cases and safety-critical decisions
Think about: Sensor fusion strategies, uncertainty propagation through the pipeline, safety constraints, and real-time compute budgets (~100ms total cycle time).
Background Knowledge
The Autonomous Vehicle Decision System relies heavily on Machine Learning (ML) and Computer Vision to interpret sensor data and make informed decisions. The system's primary goal is to perceive its environment, predict the actions of surrounding agents, and plan a safe and efficient route. This process involves sensor fusion, which combines data from various sensors such as cameras, LiDAR, and radar to create a unified world model. The perception subsystem is responsible for detecting and tracking road actors, including vehicles, pedestrians, and cyclists, using techniques like object detection and tracking.
The prediction subsystem focuses on forecasting the future trajectories of surrounding agents, taking into account their current state, velocity, and acceleration. This is often achieved using motion forecasting models, which can be based on physics-based or data-driven approaches. The planning subsystem then uses the predicted trajectories to determine a safe and comfortable driving path, considering factors like collision avoidance, lane keeping, and speed regulation. To ensure the system's reliability and safety, it's essential to consider uncertainty propagation throughout the pipeline, as well as safety constraints and real-time compute budgets.
The ML decision-making pipeline must operate within a tight 100ms total cycle time, making real-time processing and efficient algorithm design crucial. The system should also be able to handle edge cases and safety-critical decisions, such as navigating through construction zones or responding to adverse weather conditions. To address these challenges, developers can leverage various ML architectures, including deep learning models like convolutional neural networks (CNNs) and recurrent neural networks (RNNs), as well as traditional computer vision techniques like feature extraction and tracking.
Algorithm/Approach
The general approach to solving this problem involves designing a modular pipeline consisting of the three core ML subsystems: perception, prediction, and planning. Each subsystem can be implemented using a combination of ML models and traditional computer vision techniques. The perception subsystem can utilize object detection algorithms like YOLO or SSD to detect and track road actors. The prediction subsystem can employ motion forecasting models like Kalman filters or LSTM networks to forecast future trajectories. The planning subsystem can use motion planning algorithms like A* or sampling-based methods to determine a safe and efficient driving path.
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
📝 Your Design Approach
Describe your system design approach. Consider components, data flow, and key decisions.