Unit 6: AI for Intelligent Systems and Robotics

I. Orientation: Artificial Intelligence in Physical Systems

Artificial intelligence enables machines to perceive their surroundings, interpret information, make decisions, communicate with people, and improve through experience. In intelligent systems and robotics, AI connects software reasoning with sensors, actuators, computers, and physical environments. A robot commonly follows the cycle sense → perceive → plan → act → learn, while an intelligent software system may operate entirely within digital data.

• Perception: Sensors such as cameras, microphones, lidar, ultrasonic sensors, and force sensors collect environmental data.
• Representation: Raw signals are converted into useful forms, such as objects, maps, speech transcripts, or numerical state vectors.
• Decision-making: Algorithms select actions according to goals, constraints, predictions, or learned policies.
• Action: Actuators, motors, grippers, displays, or dialogue systems produce an observable response.
• Learning: Machine learning identifies patterns from data or improves behavior through feedback.
• Uncertainty: Sensor noise, incomplete observations, changing environments, and ambiguous language require probabilistic or adaptive methods.
• Safety and ethics: Autonomous decisions must respect physical safety, privacy, fairness, accountability, and human control.

II. Computer Vision: Perception Through Images

Computer vision is the AI field concerned with extracting meaningful information from images and video. In robotics, it transforms camera pixels into representations such as object identity, position, depth, motion, and scene structure.

A. Computer Vision (Overview of Image Classification, Overview of Object Detection)

Computer vision systems apply trained models to visual data so that an intelligent system can interpret its environment.

• Input: An image is represented as a tensor of pixel values. A color image commonly has dimensions (H \times W \times 3), where (H) is height, (W) is width, and 3 represents red, green, and blue channels.
• Feature extraction: Convolutional neural networks (CNNs) detect local patterns such as edges, textures, and shapes before combining them into higher-level features.
• Prediction: The model produces labels, coordinates, masks, or probabilities.
• Training: Parameters are adjusted by minimizing a loss function. For classification, cross-entropy is commonly used:

L = -Σ y_c log(p_c)

Here, (y_c) is the true label indicator for class (c), and (p_c) is the predicted probability for that class.

• Robotic use: Vision supports navigation, object manipulation, inspection, face or gesture recognition, and detection of obstacles.

B. Overview of Image Classification

Image classification assigns one or more labels to an entire image or a selected region.

• Single-label classification: The image receives one dominant class, such as cat, road, or defective_part.
• Multi-label classification: Several labels may apply simultaneously, such as person, bicycle, and helmet.
• Output: A softmax layer converts model scores into class probabilities:

p(c | x) = exp(z_c) / Σ_j exp(z_j)

Here, (x) is the input image, (z_c) is the score for class (c), and (p(c|x)) is its predicted probability.

• Evaluation: Accuracy measures total correct predictions, while precision and recall are more informative for imbalanced or safety-critical data.
• Worked example: A warehouse robot classifies a camera image as 充? No—classification should use a clear label such as fragile_box; that label can trigger slower movement and careful gripping.
• Limitation: Classification does not normally identify where an object is located. An image containing three objects may receive only a general scene label.

C. Overview of Object Detection

Object detection identifies object categories and their locations within an image.

• Bounding box: A detected object is commonly represented as ((x{\min}, y{\min}, x{\max}, y{\max})), defining the upper-left and lower-right corners.
• Confidence score: The model estimates how likely a box is to contain a particular class.
• Intersection over Union (IoU): Detection overlap is measured by:

IoU = area of intersection / area of union

An IoU of 1 indicates identical boxes; an IoU of 0 indicates no overlap.

• Detection approaches:
  1. Two-stage detectors: Region proposals are generated first, then classified and refined; they often provide high accuracy.
  2. One-stage detectors: Classification and box prediction occur in one pass; they are usually faster for real-time robotics.
• Non-maximum suppression: When several overlapping boxes describe one object, the system retains the highest-confidence box and removes redundant boxes.
• Robotic use: A mobile robot can detect people, doors, tools, traffic signs, and obstacles, then combine locations with motion planning.

D. Applications and Limitations

Vision is valuable when a robot must understand physical objects, but performance depends strongly on data and operating conditions.

• Lighting: Shadows, glare, darkness, or changing sunlight can alter pixel patterns and reduce accuracy.
• Occlusion: A partially hidden object may produce incomplete or incorrect detections.
• Domain shift: A model trained on indoor images may perform poorly outdoors or with unfamiliar equipment.
• Latency: A high-accuracy model may be unsuitable if inference is too slow for a moving robot.
• Safety: Vision predictions should be combined with redundant sensors and conservative stopping rules in dangerous environments.

III. Natural Language Processing: Communication With People

Natural Language Processing (NLP) allows computers and robots to process human language in written or spoken form. It supports commands, questions, explanations, collaboration, and access to information.

A. Natural Language Processing (NLP Fundamentals, Chatbots and Conversational AI) for Intelligent Systems and Robotics

NLP converts language into structured meaning and generates appropriate responses or actions.

• Language pipeline: Speech recognition may produce text; tokenization divides text into units; a language model interprets context; an action manager selects a response.
• Representation: Words and sentences can be encoded as vectors, allowing models to measure semantic relationships.
• Context: The phrase “pick it up” requires the system to identify what “it” refers to in the surrounding dialogue or scene.
• Embodied meaning: A robot must connect language with perception and action, such as linking “the red cup” to a detected object.

B. NLP Fundamentals

NLP fundamentals include the methods used to analyze, interpret, and generate language.

• Tokenization: The sentence “Move to the charging station” is divided into words or subword units.
• Syntax: Part-of-speech tagging and dependency analysis identify grammatical roles, such as the object of the verb “move.”
• Semantics: The system maps words to meaning; “charge” may refer to powering a robot or imposing a fee depending on context.
• Intent and entities: In a command such as “set the temperature to 22 degrees,” the intent is temperature control and 22 degrees is an entity or parameter.
• Language models: Transformer models use attention to relate tokens across a sequence and predict or generate contextually appropriate text.
• Speech integration: Automatic speech recognition converts audio to text, while text-to-speech converts a response back into spoken language.
• Limitation: Ambiguity, accents, background noise, sarcasm, and uncommon phrasing can cause incorrect interpretation.

C. Chatbots and Conversational AI

Chatbots and conversational AI systems conduct interactions through text or speech, often combining language models with dialogue management and external tools.

• Dialogue state: The system records relevant context, such as the user’s request, previous turns, unresolved tasks, and identified parameters.
• Response generation: A response may be retrieved from a fixed knowledge base or generated by a language model.
• Tool use: A robot assistant may call a navigation, calendar, database, or sensor service instead of relying only on generated text.
• Grounding: Statements should be tied to verified data or physical observations. For example, “The door is open” should be based on a sensor or vision result.
• Turn-taking: A conversational system must handle interruptions, confirmations, clarification requests, and task completion.
• Robotic example: “Bring me the blue bottle” requires speech recognition, object detection, reference resolution, path planning, grasping, and a spoken completion message.
• Risks: Hallucinated answers, unauthorized commands, privacy leakage, and failure to distinguish a user’s voice from background speech require safeguards.

D. Applications and Limitations

NLP improves accessibility and collaboration, but language alone does not guarantee correct physical action.

• Applications: Voice-controlled robots, customer-service agents, medical assistants, educational tutors, and human-robot collaboration.
• Constraint handling: Commands must be checked against battery level, object availability, workspace limits, and safety rules.
• Human oversight: High-risk actions should require confirmation or remain under human supervision.
• Evaluation: Success should measure task completion, grounding accuracy, response time, and user understanding, not only grammatical fluency.

IV. Reinforcement Learning: Learning Through Interaction

Reinforcement Learning (RL) is a framework in which an agent learns to choose actions by receiving rewards from an environment. It is particularly useful when correct behavior is difficult to specify with labelled examples but can be evaluated through outcomes.

A. Reinforcement Learning (Concepts - Agent, State, Action, Reward) in Context Intelligent Systems and Robotics

An RL problem is commonly modeled as a Markov Decision Process (MDP), represented by states, actions, transition probabilities, and rewards.

• Agent: The learner or decision-maker, such as a robotic arm controller.
• State: A description of the current situation, including position, velocity, battery level, detected objects, or distance to a target.
• Action: A command available to the agent, such as turning left, increasing motor speed, or closing a gripper.
• Reward: A numerical signal indicating the immediate value of an action, such as (+10) for successful delivery and (-100) for collision.
• Policy: A mapping (\pi(a|s)) from state (s) to the probability of selecting action (a).
• Return: The discounted total reward is:

G_t = R_(t+1) + γR_(t+2) + γ²R_(t+3) + ...

Here, (R_{t+k}) is a future reward and (\gamma), with (0 \leq \gamma < 1), controls the importance of future rewards.

• Exploration and exploitation:
  1. Exploration: Try unfamiliar actions to discover better behavior.
  2. Exploitation: Choose actions already known to produce high rewards.
• Robotic context: The state may be estimated from noisy sensors, so the robot may need a belief state rather than a perfectly known state.

B. Applications and Limitations

RL can learn complex control strategies, but training must be designed carefully.

• Applications: Robot locomotion, grasping, warehouse routing, adaptive energy management, and game-playing robots.
• Reward design: A robot rewarded only for speed may ignore safety; a useful reward can combine progress, energy use, smoothness, and collision penalties.
• Simulation: Training in simulation reduces physical risk, but the “sim-to-real” gap can occur when simulated friction, lighting, or object dynamics differ from reality.
• Sample efficiency: Robots may require thousands or millions of interactions, making direct physical training expensive.
• Safety: Action constraints, emergency stops, human supervision, and offline training data limit dangerous exploration.

V. AI for Autonomous Robots: Integrated Decision-Making

AI for autonomous robots combines perception, localization, planning, control, and learning so that a robot can perform tasks with limited direct human control.

A. AI for Autonomous Robots

An autonomous robot must transform uncertain sensor data into safe, goal-directed physical behavior.

• Localization: The robot estimates its position using wheel odometry, inertial sensors, lidar, cameras, or GPS.
• Mapping: Simultaneous Localization and Mapping (SLAM) builds a map while estimating the robot’s location.
• Planning: A global planner selects a route, while a local planner avoids moving obstacles and respects robot dynamics.
• Control: A controller converts a planned trajectory into motor commands; feedback corrects errors between desired and actual motion.
• Sensor fusion: Combining camera, lidar, and inertial measurements can improve reliability compared with one sensor alone.
• Autonomy levels: A robot may operate under teleoperation, shared control, supervised autonomy, or full autonomy depending on human involvement.
• Example: A delivery robot detects a pedestrian, updates its local map, slows down, replans around the person, and resumes its route after the path is clear.
• Constraints: Limited battery, computation, communication, payload, and actuator precision shape the feasible behavior.

B. Applications and Limitations

Autonomous robots are deployed where repeated, hazardous, distant, or precise work justifies their complexity.

• Applications: Agricultural monitoring, logistics, inspection, search and rescue, domestic assistance, surgery, and planetary exploration.
• Robustness: The robot should degrade gracefully when a camera fails, a map changes, or communication is lost.
• Verification: Testing should include unusual environments, sensor failures, unexpected obstacles, and adversarial conditions.
• Human factors: People need understandable robot behavior, predictable stopping, and clear responsibility for decisions.
• Ethical concerns: Surveillance, employment effects, weaponization, unequal access, and accountability must be addressed during design.

VI. Future Trends in AI for Intelligent Systems and Robotics

Future development will emphasize more capable, efficient, adaptive, and trustworthy systems that connect multimodal intelligence with physical action.

A. Future Trends in AI for Intelligent Systems and Robotics

Emerging systems are moving toward robots that can understand broader context, learn efficiently, and cooperate with humans.

• Multimodal models: Vision, language, audio, touch, and action data are integrated so a robot can connect “pick up the object beside the book” with sensory evidence.
• Embodied AI: Intelligence is trained through interaction with physical or simulated environments rather than language or images alone.
• Edge AI: Models run on onboard processors, reducing communication delay and improving operation when cloud connectivity is unavailable.
• Continual learning: Robots update knowledge as objects, layouts, and user preferences change, while preventing harmful forgetting.
• Human-robot collaboration: Robots increasingly infer intent, share workspace safely, and explain planned actions.
• Foundation models for robotics: Large pretrained models may provide general skills that are adapted to specific robots and tasks.
• Trustworthy autonomy: Explainability, uncertainty estimation, privacy protection, formal verification, and fail-safe behavior will become central engineering requirements.
• Energy efficiency: Smaller models, specialized hardware, and event-driven sensing can reduce power consumption for mobile robots.
• Regulation and standards: Safety certification, data governance, and accountability frameworks will influence which autonomous systems can be deployed.