Unit 4: Camera Interfacing & Computer Vision - Subjective Questions

The Raspberry Pi camera system consists of several hardware and software components that work together to capture and process images.

• Camera sensor: Converts incoming light into electrical signals. Each photosite measures the intensity of light, usually through a Bayer color filter array.
• Lens assembly: Focuses light onto the sensor. Its focal length and aperture influence the field of view, depth of field, and brightness.
• CSI-2 interface: The Camera Serial Interface transfers image data from the camera module to the Raspberry Pi using MIPI CSI-2 communication.
• Camera connector: A ribbon cable physically connects the camera module to the CSI port of the Raspberry Pi.
• Image Signal Processor: The ISP converts raw sensor data into a usable image by performing demosaicing, noise reduction, white balance, color correction, sharpening, and exposure adjustment.
• VideoCore processor: Handles ISP operations and hardware-accelerated multimedia processing on Raspberry Pi models.
• Camera software stack: Modern Raspberry Pi systems use libcamera and applications such as rpicam-still and rpicam-vid to control image capture.
• Application layer: Programs written using Picamera2, OpenCV, or other libraries receive and process camera frames.

Thus, the general data flow is scene → lens → sensor → CSI-2 → ISP → memory → application or display.

The Raspberry Pi camera can be set up using the following procedure:

1. Power off the Raspberry Pi: Disconnect the power supply before connecting the camera.
2. Locate the camera connector: Identify the CSI camera port. The connector position differs between Raspberry Pi models.
3. Insert the ribbon cable: Lift the connector latch, insert the cable in the correct orientation, and secure the latch. The exposed contacts must face the appropriate direction for the board.
4. Boot the operating system: Connect power and start Raspberry Pi OS.
5. Update the system: Run sudo apt update followed by sudo apt full-upgrade when required.
6. Verify camera detection: Use rpicam-hello --list-cameras to list connected cameras.
7. Preview the camera: Run rpicam-hello to display a short preview.
8. Capture a still image: Use rpicam-still -o test.jpg.
9. Record video: Use rpicam-vid -t 10000 -o video.h264 to record approximately ten seconds of video.

If the camera is not detected, check the cable orientation, connector lock, operating-system compatibility, power supply, and whether the camera is connected to the correct port.

A camera pipeline is the sequence of hardware and software operations that converts light entering a camera into a displayable or storable image.

The main stages are:

1. Optical capture: The lens focuses light from the scene onto the image sensor.
2. Photoelectric conversion: Sensor pixels convert photons into electrical charge.
3. Analog processing: The sensor applies analog gain and reads the accumulated pixel values.
4. Analog-to-digital conversion: Electrical signals are converted into digital raw pixel values.
5. Black-level correction: The pipeline removes the sensor's baseline electrical offset.
6. Defective-pixel correction: Missing or abnormal pixel values are estimated from neighboring pixels.
7. Demosaicing: A full-color RGB value is reconstructed at each pixel from the Bayer-pattern data.
8. Noise reduction: Random sensor and electronic noise is suppressed.
9. Automatic controls: Auto-exposure, auto-white-balance, and auto-focus algorithms determine suitable camera settings.
10. Color and tone processing: Color correction, gamma correction, contrast adjustment, and tone mapping are applied.
11. Sharpening and scaling: Details may be enhanced, and the image may be resized.
12. Output conversion: The frame is converted to a format such as RGB, YUV, JPEG, or H.264.

The final frame can then be displayed, saved, streamed, or processed using computer-vision algorithms.

An Image Signal Processor, or ISP, is a specialized processing unit that converts raw sensor measurements into visually meaningful images or video frames.

Five important ISP operations are:

• Demosaicing: A Bayer sensor records only one color component at each pixel. Demosaicing estimates the missing color components to generate a full RGB image.
• White balance: Adjusts the red, green, and blue channels so that neutral objects appear neutral under different light sources.
• Noise reduction: Removes variations caused by low light, thermal effects, high sensor gain, and electronic interference.
• Color correction: Applies a color correction matrix to transform sensor-dependent colors into a standard color space.
• Gamma correction: Applies a nonlinear transformation to match human visual perception and display characteristics. A simplified expression is where controls brightness mapping.
• Sharpening: Enhances edges and fine details, often by emphasizing high-frequency image components.
• Exposure control: Adjusts shutter time and sensor gain to obtain suitable image brightness.

A well-designed ISP improves image quality while maintaining the frame rate required by real-time applications.

• Exposure time: It is the duration for which the sensor collects light for one frame. A longer exposure produces a brighter image but may cause motion blur.
• Sensor gain: It amplifies the electrical signal generated by the sensor. Higher gain brightens the image but also amplifies noise.
• Aperture: It is the opening in the lens through which light enters. A wider aperture admits more light and produces a shallower depth of field. Many basic Raspberry Pi camera modules have a fixed aperture.
• Frame rate: It is the number of frames captured per second, measured in frames per second or FPS. A higher frame rate provides smoother motion but restricts the maximum exposure time available for each frame.

The approximate relationship between exposure time and frame rate is:

For example, at FPS, the available frame period is approximately second. A practical camera system balances these parameters to achieve sufficient brightness, low noise, limited motion blur, and the required video smoothness.

OpenCV may be installed from Raspberry Pi OS packages or through Python package management.

Installation using operating-system packages:

1. Update package information using sudo apt update.
2. Install OpenCV using sudo apt install python3-opencv.
3. Install supporting tools, such as python3-pip and python3-venv, if required.

Installation in a virtual environment:

1. Create an environment using python3 -m venv --system-site-packages cv-env.
2. Activate it using source cv-env/bin/activate.
3. Depending on the operating system and project requirements, install an appropriate package such as opencv-python using pip.

Verification:

Run the following Python statements:

import cv2

print(cv2.__version__)

A displayed version number confirms that the module has loaded successfully. A further test can read an image using cv2.imread(), inspect its dimensions using image.shape, and save it using cv2.imwrite().

Package-based installation is usually simpler and compatible with Raspberry Pi OS, whereas source compilation offers customization but requires more time and memory.

In modern Raspberry Pi systems, Picamera2 can capture frames from the libcamera pipeline and provide them to OpenCV.

A typical procedure is:

1. Import Picamera2, OpenCV, and timing modules.
2. Create a Picamera2 object.
3. Configure a preview or video stream with a suitable resolution and pixel format.
4. Start the camera and allow automatic controls to stabilize.
5. Capture frames in a loop.
6. Convert the color format when necessary.
7. Apply OpenCV processing.
8. display, store, or stream the result.

A conceptual code sequence is:

camera = Picamera2()

camera.configure(camera.create_preview_configuration(main={"format": "RGB888"}))

camera.start()

frame = camera.capture_array()

gray = cv2.cvtColor(frame, cv2.COLOR_RGB2GRAY)

The frame may then undergo filtering, thresholding, feature extraction, face detection, or object detection. Resolution should be selected carefully because larger frames improve detail but require more processing time and memory. The application must also stop the camera and release display or network resources when it terminates.

Image preprocessing refers to operations applied before analysis to improve image quality, standardize the input, or reduce computational cost.

• Resizing: Changes image dimensions to match an algorithm or neural network's expected input size. Reducing dimensions also lowers processing time.
• Color conversion: Changes one color representation into another. For example, RGB or BGR may be converted to grayscale when color is unnecessary, or to HSV for color-based segmentation.
• Normalization: Scales pixel values to a standard range. If an 8-bit pixel value is , normalization to the interval can be performed using Neural networks often require this or another model-specific normalization rule.
• Thresholding: Separates pixels into classes based on intensity. For a threshold , binary thresholding can be represented as

Preprocessing should be selected according to the task. Excessive filtering or resizing may remove useful features and reduce detection accuracy.

Mean filter:

• Replaces each pixel with the average value of pixels in a local neighborhood.
• Reduces random variations but tends to blur edges.
• It is simple and computationally inexpensive.

Gaussian filter:

• Computes a weighted average in which nearby pixels receive greater weights than distant pixels.
• Produces smoother and more natural blurring than a basic mean filter.
• It is commonly used before edge detection because it suppresses high-frequency noise.
• The amount of smoothing depends on the kernel size and standard deviation .

Median filter:

• Replaces each pixel with the median value in its neighborhood.
• It is especially effective for salt-and-pepper or impulse noise.
• It preserves edges better than averaging filters but can be more computationally expensive.

Therefore, the mean filter is suitable for basic smoothing, the Gaussian filter is useful for normally distributed noise and scale-space processing, and the median filter is preferred for impulse noise. Kernel size must be chosen carefully because a large kernel can remove important details.

Histogram equalization is a contrast-enhancement technique that redistributes image intensity values so that the available range is used more effectively. It uses the cumulative distribution function of the histogram to map the original intensities to new values.

For an image with intensity levels, a simplified transformation is:

where is the probability of intensity .

Global histogram equalization:

• Uses a single transformation for the entire image.
• Works well when the image has uniformly poor contrast.
• May over-enhance noise or fail when different regions have different lighting conditions.

Contrast-Limited Adaptive Histogram Equalization:

• Divides the image into small tiles and equalizes each tile separately.
• Limits histogram amplification to reduce excessive noise enhancement.
• Combines neighboring tiles using interpolation to avoid visible boundaries.
• Is useful for faces, medical images, and outdoor scenes containing shadows and highlights.

For color images, enhancement is preferably applied to a luminance channel rather than independently to all RGB channels, because separate RGB equalization can distort colors.

The Canny edge detector identifies strong and thin boundaries while reducing false responses caused by noise.

Its main stages are:

1. Noise reduction: A Gaussian filter smooths the input image.
2. Gradient calculation: Horizontal and vertical intensity derivatives, and , are computed. Gradient magnitude is obtained using and orientation is obtained using
3. Non-maximum suppression: Pixels that are not local maxima along the gradient direction are removed, producing thin edges.
4. Double thresholding: Pixels above the high threshold become strong edges, pixels between the thresholds become weak edges, and lower-valued pixels are rejected.
5. Edge tracking by hysteresis: Weak edges connected to strong edges are retained, while isolated weak responses are removed.

Canny edges can support contour detection, shape analysis, lane detection, measurement, and object segmentation. Its performance depends on the smoothing level and the selected low and high thresholds.

A Haar cascade classifier detects objects such as faces using a cascade of boosted classifiers trained on positive and negative image samples.

Working principle:

• The input image is usually converted to grayscale.
• Haar-like features measure intensity differences between adjacent rectangular regions, representing structures such as eye regions, cheeks, and the nose bridge.
• An integral image allows rectangular sums to be calculated efficiently.
• AdaBoost selects useful features and combines weak classifiers into stronger classifiers.
• Classifiers are arranged in a cascade. Early stages reject obvious non-face windows quickly, while later stages examine promising regions more carefully.
• The detector scans the image at different positions and scales.

Advantages:

• Fast enough for many real-time applications.
• Requires relatively modest computational resources.
• Pretrained cascade files are readily available in OpenCV.

Limitations:

• Sensitive to pose, occlusion, lighting, and blur.
• May generate false positives.
• Usually performs best on near-frontal faces.
• Modern deep-learning face detectors are generally more robust.

Detection parameters such as scale factor, minimum neighbors, and minimum face size must be tuned for the application.

Face detection determines whether faces are present and returns their locations, usually as bounding boxes. Face recognition determines or verifies the identity of the person represented by a detected face.

A complete recognition workflow includes:

1. Image acquisition: Capture a frame from the camera.
2. Face detection: Locate one or more faces using a Haar cascade, HOG-based method, or deep neural network.
3. Landmark detection: Identify features such as eyes, nose, and mouth corners.
4. Alignment: Rotate and scale the face so that important landmarks are in consistent positions.
5. Preprocessing: Normalize size, illumination, and pixel values.
6. Feature extraction: Produce a discriminative feature vector or embedding using LBPH or a deep neural network.
7. Comparison: Compare the unknown embedding with stored embeddings using a distance measure such as Euclidean distance:

8. Decision: If the smallest distance is below a selected threshold, assign the corresponding identity; otherwise label the face as unknown.
9. Evaluation: Measure false acceptance, false rejection, accuracy, and performance under different lighting and poses.

Recognition systems should also include consent, data protection, access control, and measures against spoofing.

The Local Binary Pattern Histogram, or LBPH, method recognizes faces by describing local texture patterns.

For each pixel, the method compares neighboring pixel intensities with the center pixel. A neighbor is assigned 1 if its intensity is greater than or equal to the center value; otherwise, it is assigned 0. The resulting bits form a binary number called the local binary pattern.

A general LBP expression is:

where is the center intensity, represents neighboring intensities, and

The face image is divided into regions. A histogram of LBP codes is computed for each region, and all histograms are concatenated into a feature vector. During recognition, the feature vector of a test face is compared with stored training vectors.

Advantages: It is simple, relatively fast, and suitable for low-power devices.

Limitations: It can be affected by large pose changes, poor alignment, occlusion, and major illumination variations. Deep face embeddings generally provide better accuracy for complex datasets.

A deep-learning object detector locates and classifies multiple objects in an image using a trained neural network.

Working:

1. The frame is resized to the model's required dimensions.
2. Pixel values are normalized and arranged into an input tensor.
3. A convolutional neural network extracts hierarchical features such as edges, textures, shapes, and semantic patterns.
4. Detection layers predict object locations and class probabilities.
5. Predictions are filtered using confidence thresholds and non-maximum suppression.
6. The remaining detections are mapped back to the original image coordinates.

A bounding box is a rectangle representing an object's location, commonly expressed as or by center coordinates, width, and height.

A class score estimates how strongly a detected region belongs to a category such as person, car, or dog.

A confidence score represents the detector's certainty that a prediction contains an object of a particular class. Predictions below the confidence threshold are rejected.

A low threshold may improve recall but increase false positives, whereas a high threshold may improve precision but miss valid objects. The threshold must therefore be selected according to the application's risk and performance requirements.

One-stage detectors:

• Predict object classes and bounding boxes directly in a single network pass.
• Examples include YOLO, SSD, and RetinaNet.
• They generally provide high inference speed and are suitable for real-time video.
• Compact variants can operate on Raspberry Pi or other edge devices, especially with acceleration.
• Some one-stage models may have difficulty with very small or densely packed objects, depending on their design and resolution.

Two-stage detectors:

• First generate candidate object regions and then classify and refine them.
• Examples include Fast R-CNN and Faster R-CNN.
• They often provide strong localization and detection accuracy.
• They usually require more computation and have higher latency than lightweight one-stage detectors.

For live monitoring on a Raspberry Pi, a compact one-stage model such as a lightweight YOLO or MobileNet-SSD variant is usually preferred. A two-stage detector may be selected when accuracy is more important than real-time speed. The final decision should consider model size, available memory, input resolution, frame-rate requirements, accelerator availability, and the types of objects being detected.

Intersection over Union, or IoU, measures the overlap between two bounding boxes. If and are two boxes, IoU is defined as:

Its value ranges from to . A value of indicates no overlap, while indicates identical boxes. During evaluation, a prediction may be considered correct if its IoU with the ground-truth box exceeds a chosen threshold.

Non-maximum suppression, or NMS, removes duplicate predictions for the same object:

1. Reject boxes with low confidence.
2. Select the remaining box with the highest confidence.
3. Keep that box as a final detection.
4. Compute its IoU with the other boxes of the same class.
5. Remove boxes whose IoU exceeds the NMS threshold.
6. Repeat until no candidate boxes remain.

A very low NMS threshold may incorrectly suppress nearby objects, while a high threshold may retain multiple boxes around one object. Variants such as Soft-NMS reduce box scores instead of removing overlapping boxes immediately.

Real-time object detection on a Raspberry Pi requires careful optimization because processor speed, memory, thermal capacity, and power are limited.

Useful techniques include:

• Choose a lightweight model: Use architectures such as MobileNet-SSD, EfficientDet-Lite, tiny YOLO variants, or other edge-oriented detectors.
• Reduce input resolution: Smaller images require fewer calculations, although very small objects may become harder to detect.
• Quantize the model: Convert floating-point weights and operations to lower-precision formats such as 8-bit integers.
• Use efficient runtimes: TensorFlow Lite, ONNX Runtime, OpenCV DNN, or hardware-specific runtimes can reduce inference overhead.
• Employ an accelerator: A supported neural-processing unit or USB accelerator can perform inference more efficiently than the CPU.
• Skip frames: Run detection on every second or third frame and use tracking between detections.
• Limit classes: If supported, focus processing on only the classes required by the application.
• Separate pipeline stages: Use threads or processes for capture, inference, display, and streaming.
• Avoid unnecessary copies: Reuse buffers and select camera formats compatible with the model pipeline.
• Provide cooling: Heat sinks or fans can reduce thermal throttling.

Optimization must be evaluated using accuracy, inference latency, throughput, memory use, temperature, and power consumption rather than FPS alone.

A network video-streaming system contains a camera source, encoder, server or sender, transport protocol, network, and client.

A typical workflow is:

1. Capture frames using libcamera, Picamera2, GStreamer, or OpenCV.
2. Resize or preprocess frames to satisfy bandwidth and latency requirements.
3. Encode the frames using MJPEG, H.264, H.265, or another codec.
4. Packetize or embed the encoded video in a protocol such as HTTP, RTSP, RTP, or WebRTC.
5. Send the stream through Ethernet or Wi-Fi.
6. Decode and display the video in a browser, media player, or custom client.

Common approaches:

• MJPEG over HTTP: Easy to implement and view in browsers, but consumes more bandwidth because each frame is independently JPEG-compressed.
• H.264 over RTSP or RTP: Provides efficient compression and is suitable for surveillance or media-player clients.
• WebRTC: Supports low-latency interactive streaming and browser-based viewing but requires more complex signaling and connectivity handling.

A secure implementation should use authentication, encryption, firewall rules, and restricted network exposure. It should also handle client disconnections and release camera and socket resources correctly.

The quality and responsiveness of a video stream depend on camera, encoding, network, and client parameters.

Important factors:

• Resolution: More pixels improve detail but increase encoding work and data size.
• Frame rate: Higher FPS produces smoother motion but increases bandwidth and processor load.
• Compression codec: H.264 normally uses less bandwidth than MJPEG for comparable visual quality.
• Bit rate: A higher bit rate can improve quality but may exceed network capacity.
• Key-frame interval: Frequent key frames support faster recovery and seeking but increase bandwidth.
• Network conditions: Congestion, weak Wi-Fi, packet loss, and jitter reduce quality and increase delay.
• Buffering: Larger buffers improve smoothness but add latency.
• Processing load: Computer-vision inference and software encoding can delay frame delivery.

For an uncompressed stream, approximate bandwidth is:

where and are image dimensions, is the number of bits per pixel, and is the frame rate. Compression greatly lowers the transmitted bandwidth.

Improvements:

• Use hardware-accelerated encoding when available.
• Reduce resolution, frame rate, or bit rate.
• Prefer wired Ethernet or strong Wi-Fi.
• Use an efficient codec and tune its low-latency settings.
• Reduce unnecessary buffering.
• Separate camera capture, inference, encoding, and transmission tasks.
• Monitor dropped frames, end-to-end latency, CPU usage, temperature, and network throughput.