Unit 4: Robot Vision Sensors

ECE246 — Sensors For Robotics 11 min read

I. Orientation

Robot vision is the process by which a robot acquires, processes, and interprets electromagnetic or range measurements to perceive its environment. A complete vision system connects sensing hardware to computation, localization, mapping, planning, and control.

Governing principle: A sensor converts a physical quantity, such as light intensity or distance, into an electrical or digital signal.
Perception pipeline: Energy from a scene is captured, converted, sampled, transmitted, calibrated, processed, and interpreted.
Spatial dimensions:
- 2D sensing: Measures image intensity or color over pixel coordinates ((u,v)).
- 3D sensing: Estimates depth (Z) or Cartesian position ((X,Y,Z)).
Measurement quality: Resolution, accuracy, precision, range, field of view, frame rate, latency, and noise determine suitability.
Coordinate convention: Measurements may be expressed in sensor, robot-base, odometry, or world/map frames.
Robotic objective: Vision supports detection, recognition, tracking, manipulation, obstacle avoidance, localization, and mapping.
Integration principle: Reliable perception requires synchronized measurements, calibrated sensors, suitable communication, and uncertainty-aware fusion.

II. Vision-Sensing Foundations

A. Vision sensor generalities

A vision sensor measures scene information and converts it into data from which a robot can infer objects, motion, geometry, or position.

Components: A typical camera contains a lens, aperture, image sensor, timing circuit, ADC, processor, and communication interface.
Image sensors:
- CMOS: Offers low power, high integration, and fast readout; it dominates embedded robotics.
- CCD: Traditionally provides uniform, low-noise output but requires more power and external circuitry.
Optical model: The pinhole-camera equations relate a 3D point to its image:
TEXT
```
u = fx(X/Z) + cx
v = fy(Y/Z) + cy
```
Here, ((X,Y,Z)) is the point in camera coordinates, ((u,v)) is its pixel position, (f_x,f_y) are focal lengths in pixels, and ((c_x,c_y)) is the principal point.
Intrinsic calibration: Determines focal lengths, principal point, skew, and lens-distortion coefficients using known targets such as a checkerboard.
Extrinsic calibration: Determines rotation (R) and translation (t) between coordinate frames:
TEXT
```
Pc = R Pw + t
```
(P_w) is a world-frame point and (P_c) is the corresponding camera-frame point.
Performance factors: Illumination, motion blur, rolling shutter, lens distortion, exposure, noise, and occlusion affect measurement reliability.
Selection criteria: A mobile robot may prioritize wide field of view and low latency, whereas inspection may prioritize resolution and color accuracy.

B. Image acquisition fundamentals

Image acquisition transforms incident light into a discrete array of pixel values suitable for digital processing.

Exposure: Shutter time controls how long the sensor collects light; long exposure improves brightness but increases motion blur.
Aperture: The f-number (N=f/D), where (f) is focal length and (D) is aperture diameter, controls light and depth of field.
Sampling: Spatial sampling divides the image plane into pixels; insufficient resolution causes loss of fine detail and aliasing.
Quantization: An (n)-bit pixel has (2^n) possible levels; an 8-bit grayscale image represents values from 0 to 255.
Frame rate: Frames per second determines temporal sampling; at 30 fps, consecutive frames are approximately (33.3) ms apart.
Dynamic range: The ratio between the largest non-saturating signal and the noise floor determines whether bright and dark regions remain distinguishable.
Color acquisition: Bayer-filter cameras commonly sample red, green, and blue at different pixels, followed by demosaicing to estimate complete RGB values.
Preprocessing: Denoising, white balance, rectification, contrast adjustment, and timestamping prepare images for later algorithms.

III. Spatial Vision Technologies

A. 2D/3D sensors

Two-dimensional sensors measure projected appearance, while three-dimensional sensors additionally estimate scene depth or surface geometry.

2D sensors:
- Output: Monocular and color cameras produce intensity (I(u,v)) or RGB values at each pixel.
- Advantages: They are inexpensive, compact, information-rich, and suitable for recognition, line following, and optical flow.
- Limitation: A single uncalibrated image does not directly provide absolute depth because multiple 3D points can project to the same pixel.
3D sensors:
- Stereo vision: Depth is inferred from disparity between calibrated cameras:
  TEXT
```
Z = fB/d
```
  (Z) is depth, (f) is focal length in pixels, (B) is camera baseline in metres, and (d) is disparity in pixels.
- Structured light: A known infrared pattern is projected; deformation of the pattern reveals depth.
- Time of flight: Distance is estimated from light travel time, approximately (d=ct/2), where (c) is light speed and (t) is round-trip time.
- RGB-D output: Registered color and depth images support object pose estimation, grasping, and indoor mapping.
- Point clouds: 3D measurements are represented as sets of ((X,Y,Z)) points, sometimes with color, intensity, or confidence.
- Trade-off: Active depth sensors provide direct range but may suffer from sunlight, reflective surfaces, multipath interference, or limited range.

B. LiDAR

LiDAR measures distance by emitting laser energy and observing its reflected return, producing accurate geometric information about the environment.

Time-of-flight ranging:
TEXT
```
d = cΔt/2
```
(d) is target distance, (c) is the speed of light, and (\Delta t) is measured round-trip delay; division by two accounts for outward and return travel.
Scanning forms:
- 2D LiDAR: Sweeps one plane and returns polar samples ((r,\theta)), useful for indoor navigation.
- 3D LiDAR: Uses multiple beams or scanning mechanisms to produce a volumetric point cloud.
- Solid-state LiDAR: Uses limited or no mechanical rotation, improving compactness and durability.
Cartesian conversion: A planar measurement becomes (x=r\cos\theta), (y=r\sin\theta).
Returned information: Devices may report range, bearing, intensity, timestamp, ring/channel number, and multiple echoes.
Strengths: LiDAR works without ambient illumination and provides metrically accurate obstacle boundaries.
Limitations: Fog, rain, dust, dark absorptive materials, mirrors, and glass can weaken or corrupt returns; cost and data volume can also be significant.
Robotic uses: Occupancy mapping, scan matching, collision avoidance, terrain analysis, and loop-closure verification use LiDAR geometry.

IV. Sensor Integration and Communication

A. Interfacing of vision sensors

Interfacing connects a sensor’s electrical, timing, data, and software requirements to the robot’s computing platform.

Electrical compatibility: Supply voltage, logic level, grounding, current demand, connector pinout, and protection must match the controller.
Bandwidth: Required throughput is approximately:
TEXT
```
Bandwidth = width × height × channels × bits-per-channel × fps
```
A (640 \times 480), 8-bit grayscale stream at 30 fps produces about (73.7) Mbit/s before protocol overhead or compression.
Timing: Hardware triggers and timestamps align frames with wheel encoders, IMUs, or LiDAR scans.
Data interfaces: USB, Ethernet, MIPI CSI, Camera Link, and serial buses differ in range, throughput, latency, and complexity.
Software stages: A device driver configures exposure and frame rate, receives buffers, detects errors, and exposes measurements to processing software.
Calibration handling: Intrinsic and extrinsic parameters should be stored with sensor identity and applied consistently.

B. ADC and DAC

ADCs digitize analog sensor voltages, whereas DACs convert digital commands into analog outputs.

ADC operation:
- Resolution: An ideal (n)-bit ADC divides its reference range into (2^n) codes.
  TEXT
```
LSB size = Vref / 2^n
```
- Example: For a 12-bit ADC with (V_{ref}=3.3) V, one least-significant bit is approximately (0.806) mV.
- Errors: Quantization, offset, gain error, nonlinearity, and electrical noise reduce effective accuracy.
DAC operation:
- Purpose: DACs generate analog control values for illumination, actuators, test signals, or sensor bias circuits.
- Output: An ideal unipolar DAC produces a voltage proportional to the digital code, often followed by filtering or amplification.
- Sampling condition: The sampling frequency should exceed twice the highest signal frequency under the Nyquist criterion, with an anti-aliasing filter applied before conversion.

C. I2C

I2C is a synchronous, addressed two-wire bus commonly used to configure low-data-rate sensors and camera peripherals.

Signals: SDA carries bidirectional data, while SCL carries the controller-generated clock; both normally require pull-up resistors.
Transaction: A start condition is followed by an address, read/write bit, acknowledgment, data bytes, and a stop condition.
Addressing: Multiple devices can share the bus if their 7-bit or 10-bit addresses do not conflict.
Advantages: Only two signal lines are needed, and register-based sensor configuration is straightforward.
Limitations: Shared capacitance restricts bus length and speed; address conflicts and a stuck-low line can disrupt all devices.

D. SPI and UART communication

SPI emphasizes fast synchronous transfers, while UART provides simple asynchronous point-to-point serial communication.

SPI:
- Signals: SCLK, controller-out/peripheral-in, controller-in/peripheral-out, and chip-select lines support full-duplex transfer.
- Operation: Clock polarity and phase define SPI modes; both devices must use the same mode and word length.
- Trade-off: SPI offers high speed and low protocol overhead but requires a separate chip-select for each peripheral.
UART:
- Signals: TX and RX transmit framed data without a shared clock; a common ground is also required.
- Frame: A typical format is one start bit, eight data bits, optional parity, and one stop bit.
- Configuration: Baud rate, data bits, parity, and stop bits must match; 115200 8N1 denotes 115200 baud, eight data bits, no parity, and one stop bit.
- Trade-off: UART is easy to debug and useful for GPS or microcontrollers, but it lacks native addressing and clock synchronization.

E. ROS-based sensor communication

ROS-based communication standardizes how sensor drivers publish timestamped data for perception and control nodes.

Publish-subscribe model: A camera driver publishes messages to topics, while any number of processing nodes subscribe without direct coupling.
Common messages: sensor_msgs/Image, sensor_msgs/CameraInfo, sensor_msgs/LaserScan, sensor_msgs/PointCloud2, and sensor_msgs/Imu carry structured measurements.
Coordinate frames: The tf2 system maintains timed transforms among frames such as camera_link, base_link, odom, and map.
Quality of service: In ROS 2, reliability, durability, history, and queue depth are selected according to network conditions and latency requirements.
Synchronization: Exact or approximate time synchronizers associate images, depth, and other measurements using message timestamps.
Transport considerations: Raw images preserve data but consume bandwidth; compressed-image transports reduce traffic at the cost of processing and possible information loss.
Recording: ROS bag files capture topics and timestamps, enabling repeatable offline testing of perception algorithms.

V. Autonomous Localization and Perception

A. SLAM Navigation

Simultaneous Localization and Mapping estimates a robot’s pose while constructing or updating a map of an initially unknown environment.

State estimation: The system estimates pose (\mathbf{x}t) from controls (\mathbf{u}{1:t}) and observations (\mathbf{z}_{1:t}):
TEXT
```
p(x_t, m | z_1:t, u_1:t)
```
Here, (m) is the map and (p) denotes the joint probability distribution.
Front end: Extracts features or geometric structures, tracks motion, and generates constraints through visual odometry or scan matching.
Back end: Optimizes a pose graph or probabilistic state estimate to reduce accumulated trajectory error.
Loop closure: Recognizing a previously visited place adds a long-range constraint that corrects drift.
Map forms: Occupancy grids represent free and occupied cells; landmark maps store features; point-cloud and voxel maps preserve 3D geometry.
Navigation link: Localization supplies robot pose, the map supports path planning, and live sensing updates obstacles for local control.
Failure conditions: Repetitive scenes, rapid motion, poor lighting, dynamic objects, weak geometry, and inaccurate calibration can cause tracking loss.

B. Sensor fusion technique

Sensor fusion combines complementary measurements to obtain an estimate that is more robust or complete than any individual sensor output.

Complementarity: Cameras provide texture and semantics, LiDAR supplies geometry, IMUs measure rapid motion, and wheel encoders provide local displacement.
Fusion levels:
- Raw-data fusion: Combines minimally processed measurements but requires close synchronization and calibration.
- Feature-level fusion: Associates edges, landmarks, or detected objects across sensors.
- Decision-level fusion: Combines independent classifications or confidence scores.
Kalman filtering: For approximately linear Gaussian systems, prediction uses a motion model and correction weights measurements by their uncertainty.
TEXT
```
K = P Hᵀ(HPHᵀ + R)⁻¹
x̂ = x̂⁻ + K(z - Hx̂⁻)
```
(K) is Kalman gain, (P) is predicted covariance, (H) is the observation model, (R) is measurement-noise covariance, (z) is the measurement, and (\hat{x}) is the corrected state.
Nonlinear methods: Extended Kalman filters linearize models; particle filters represent multiple hypotheses; factor graphs optimize constraints over many states.
Essential requirements: Accurate timestamps, frame transformations, noise models, outlier rejection, and online health checks prevent inconsistent fusion.
Outcome: Camera-IMU fusion can preserve rapid motion estimates when images blur, while LiDAR-camera fusion can attach semantic labels to accurate 3D obstacles.