Unit 4: Camera Interfacing & Computer Vision

I. Foundations of Camera-Based IoT

Camera-based IoT systems convert light into digital images, extract useful information through computer vision, and communicate results or video over a network. On Raspberry Pi platforms, this process combines a camera sensor, the Camera Serial Interface 2 (CSI-2), the libcamera software stack, an Image Signal Processor (ISP), and application libraries such as Picamera2 and OpenCV.

• Governing principle: A camera measures spatially distributed light intensity; computer vision transforms those measurements into decisions such as “face detected” or “object classified.”
• Processing sequence:
  • Light passes through the lens and reaches the sensor.
  • The sensor produces raw Bayer-pattern pixel values.
  • The ISP converts raw data into a viewable RGB or YUV image.
  • Computer-vision algorithms extract features or run neural-network inference.
  • Results are stored, displayed, or transmitted.
• Core hardware: Raspberry Pi board, compatible camera module, CSI ribbon cable or USB connection, lens, illumination, power supply, and network interface.
• Core software: Raspberry Pi OS, libcamera, Picamera2, OpenCV, NumPy, and optional deep-learning runtimes such as TensorFlow Lite.
• Design assumptions: Image quality depends on resolution, frame rate, exposure, lighting, focus, motion, and available processing power.
• IoT constraint: Higher resolution improves visible detail but increases memory, inference time, network bandwidth, and power consumption.

II. Raspberry Pi Camera System — From Sensor to Software

A. Raspberry Pi camera architecture

The Raspberry Pi camera architecture provides a coordinated path from image capture hardware to user applications.

• Image sensor: A CMOS sensor converts photons into electrical charge and then into digital pixel values; examples include the Sony IMX219 in Camera Module 2 and IMX708 in Camera Module 3.
• Bayer colour filter: Most sensors record one colour component per photosite in a pattern such as RGGB; the ISP reconstructs complete colour pixels through demosaicing.
• CSI-2 interface: The Mobile Industry Processor Interface Camera Serial Interface 2 carries image data over differential serial lanes, offering lower overhead than typical USB camera transport.
• Camera connector: A flexible ribbon cable links the camera to the Raspberry Pi CSI connector; its conductive contacts must face the orientation specified for the board.
• Camera stack: Modern Raspberry Pi OS uses libcamera, while Picamera2 supplies a Python-friendly interface built on that stack.
• Processing hardware: Depending on the Raspberry Pi model and camera stack, image processing is handled through dedicated multimedia hardware and optimized software components.
• Data formats:
  • RAW/Bayer preserves sensor measurements for specialized processing.
  • RGB is convenient for OpenCV but requires three colour values per pixel.
  • YUV separates brightness from colour and is widely used in video pipelines.
  • JPEG/H.264 compression reduces storage or network requirements.

B. Setting up the Raspberry Pi camera

Camera setup requires correct physical installation, an updated operating system, and validation through the supported camera tools.

• Physical connection: Shut down and disconnect power before opening the CSI connector latch, inserting the ribbon cable fully, and securing the latch.
• Software update: Current firmware, kernel components, and camera packages reduce compatibility problems.

sudo apt update
sudo apt full-upgrade
sudo apt install rpicam-apps python3-picamera2

• Detection test: On current Raspberry Pi OS releases, camera applications commonly use the rpicam- command names.

rpicam-hello --list-cameras
rpicam-hello -t 5000
rpicam-still -o image.jpg

• Python capture: Picamera2 configures streams before starting the camera.

from picamera2 import Picamera2

camera = Picamera2()
camera.configure(camera.create_still_configuration())
camera.start()
camera.capture_file("capture.jpg")
camera.stop()

• Common faults:
  • A missing camera may indicate reversed cable orientation or an incompletely seated connector.
  • A dark image may result from insufficient illumination or exposure.
  • A blurred image may result from incorrect focus, subject movement, or camera vibration.
• Operational precaution: A stable power supply is essential because undervoltage can disrupt camera and network operation.

C. Camera pipeline and ISP fundamentals

The camera pipeline transforms raw sensor measurements into corrected, colour-balanced images suitable for display or analysis.

• Sensor acquisition: Resolution, exposure time, analogue gain, and frame rate determine how the scene is sampled.
• Black-level correction: The ISP subtracts the sensor’s non-zero response in darkness.
• Demosaicing: Missing red, green, and blue components are interpolated from neighbouring Bayer samples.
• Automatic exposure: Exposure time and gain are adjusted toward a target brightness; excessive gain increases visible noise.
• Automatic white balance: Red and blue channel gains compensate for illuminant colour so neutral objects appear neutral.
• Noise reduction: Spatial filtering uses neighbouring pixels, while temporal filtering can compare consecutive frames.
• Colour correction and gamma: A colour-correction matrix adjusts sensor colour response, and gamma mapping adapts linear sensor values for display.
• Scaling and encoding: Images may be resized and encoded as JPEG or H.264 before storage or transmission.
• Latency trade-off: Buffering and advanced processing improve quality but may delay real-time control; a 30-frame/s stream has only about (1/30 = 33.3) ms per frame.

III. Computer-Vision Processing — OpenCV and Image Quality

A. OpenCV installation and usage

OpenCV is an open-source computer-vision library that supplies image I/O, transformations, feature detectors, neural-network interfaces, and video operations.

• Installation choice: Raspberry Pi OS packages are straightforward and integrate with system-managed dependencies.

sudo apt install python3-opencv
python3 -c "import cv2; print(cv2.__version__)"

• Image representation: OpenCV images are NumPy arrays with shape (height, width, channels); a 640 × 480 BGR image normally has shape (480, 640, 3).
• Colour convention: cv2.imread() loads colour images in BGR order, not RGB order.
• Basic workflow:

import cv2

image = cv2.imread("capture.jpg")
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
edges = cv2.Canny(gray, 100, 200)
cv2.imwrite("edges.png", edges)

• Camera integration: Picamera2 can deliver NumPy arrays to OpenCV, avoiding an unnecessary save-and-reload cycle.
• Resource control: Reducing frame dimensions from 1920 × 1080 to 640 × 480 reduces the pixel count from 2,073,600 to 307,200, substantially lowering processing cost.
• Limitation: Desktop OpenCV wheels or source builds may consume significant storage and compilation time; package selection should match the required modules.

B. Image preprocessing and enhancement

Preprocessing improves consistency or emphasizes visual information before detection, recognition, or measurement.

• Resizing: Neural networks require fixed input sizes such as 300 × 300 or 640 × 640; aspect-ratio distortion should be avoided through padding or careful cropping.
• Grayscale conversion: Reducing three channels to one lowers computation when colour is unnecessary.
• Normalization: Pixel values may be transformed from ([0,255]) to ([0,1]):

x' = x / 255

Here, (x) is the original pixel value and (x') is the normalized value.

• Noise suppression: Gaussian blur reduces high-frequency noise, while median filtering is effective against salt-and-pepper noise.
• Contrast enhancement: Histogram equalization redistributes grayscale intensity; CLAHE performs local enhancement while limiting excessive noise amplification.
• Thresholding: Binary thresholding separates foreground and background; adaptive thresholding handles uneven illumination.
• Morphology: Erosion removes small bright regions, dilation expands them, opening removes noise, and closing fills small gaps.
• Geometric correction: Rotation, perspective transformation, and lens-distortion correction align images before measurement.
• Caution: Strong blur, sharpening, or contrast adjustment can destroy features required by a detector; preprocessing should match the model’s training procedure.

IV. Facial Analysis — Locating and Identifying People

A. Face detection and recognition

Face detection locates faces, whereas face recognition determines or verifies identity from facial features.

1. Face detection:

   • Output: A detector returns bounding boxes, confidence values, and sometimes facial landmarks.
   • Classical method: Haar cascades use rectangular intensity features evaluated through a trained cascade; OpenCV provides pretrained XML classifiers.
   • Modern method: Deep detectors are generally more robust to pose, expression, and lighting variation.
   • Concrete operation: detectMultiScale() searches multiple image scales because a face may occupy different pixel dimensions.

2. Face recognition:

   • Feature encoding: A neural model maps each aligned face to an embedding vector representing facial characteristics.
   • Comparison: Two embeddings (a) and (b) can be compared using Euclidean distance:

d(a,b) = sqrt(sum((a_i - b_i)^2))

Here, (a_i) and (b_i) are corresponding embedding components; a smaller (d) indicates greater similarity.

• Identification: One embedding is compared against multiple enrolled identities.
• Verification: Two embeddings are compared to test a claimed identity.

• Pipeline: Capture image, detect face, locate landmarks, align and crop, compute embedding, compare with enrolled embeddings, and apply a calibrated threshold.
• Performance factors: Accuracy declines with poor lighting, extreme head pose, blur, occlusion, low resolution, or demographic imbalance in training data.
• Privacy and security: Facial data is biometric information; systems require informed consent, restricted access, encryption, retention limits, and alternatives for users who cannot or do not consent.
• Anti-spoofing: Recognition alone may accept a photograph; liveness checks can use depth, controlled movement, texture analysis, or temporal cues.

V. Deep-Learning Vision — Detecting General Objects

A. Object detection using deep learning models

Deep-learning object detection predicts object categories and locations in an image, usually as bounding boxes with confidence scores.

• Detector types:
  • One-stage models such as YOLO and SSD predict boxes and classes directly, favouring speed.
  • Two-stage models such as Faster R-CNN first propose regions and then classify them, often favouring accuracy over edge-device speed.
• Model input: The frame is resized, reordered into the expected RGB or BGR format, normalized, and arranged in the tensor layout required by the model.
• Inference output: Each candidate typically contains box coordinates, a class identifier, and a confidence score between 0 and 1.
• Intersection over Union: Box overlap is measured by

IoU = area(A intersection B) / area(A union B)

Here, (A) is the predicted box and (B) is a reference or competing box.

• Non-maximum suppression: Lower-confidence boxes with high IoU overlap are removed so one object is not reported repeatedly.
• Edge deployment: TensorFlow Lite, ONNX Runtime, OpenCV DNN, or hardware accelerators can reduce latency on Raspberry Pi systems.
• Optimization: Quantization replaces some floating-point calculations with lower-precision arithmetic, reducing model size and often increasing inference speed.
• Evaluation: Precision measures the proportion of detections that are correct; recall measures the proportion of real objects detected.
• Limitations: Results depend on training classes, camera viewpoint, object scale, occlusion, dataset quality, and the confidence threshold.

VI. Networked Video — Remote Observation and Processing

A. Video streaming over a network

Video streaming continuously captures, compresses, transports, and reconstructs frames for remote viewing or analysis.

• Pipeline: Camera capture → colour conversion → video encoding → packet transport → decoding → display or processing.
• Encoding: H.264 compresses temporal and spatial redundancy and usually requires far less bandwidth than uncompressed RGB.
• Bandwidth example: Uncompressed 640 × 480, 24-bit RGB at 30 frame/s requires approximately

640 × 480 × 3 × 30 = 27,648,000 bytes/s

This is about 27.6 MB/s before transport overhead, demonstrating why compression is necessary.

• Transport choices:
  • RTSP/RTP supports managed real-time media sessions.
  • HTTP MJPEG sends separate JPEG frames and is easy to implement but bandwidth-heavy.
  • WebRTC supports low-latency browser communication and adaptive networking.
  • TCP prioritizes reliable ordered delivery; UDP avoids retransmission delays but may lose packets.
• Latency factors: Exposure, frame buffering, encoding, network congestion, decoding, and playback buffers all contribute to end-to-end delay.
• MJPEG principle: A server repeatedly encodes OpenCV frames with cv2.imencode(".jpg", frame) and sends them using the multipart MIME type multipart/x-mixed-replace.
• Security: Streams should use authentication, encrypted transport, network segmentation, and restricted listening interfaces; exposing an unauthenticated camera port creates a direct privacy risk.
• Reliability: Applications should handle dropped connections, stale frames, encoder failures, and reconnection without blocking the camera pipeline.
• Design trade-off: Resolution, frame rate, compression quality, and latency must be balanced against CPU capacity and available network bandwidth.