Unit 6: Advanced Generative AI Applications - Subjective Questions

Vertex AI is Google Cloud's unified machine learning platform that brings together tools for building, deploying, and scaling ML and generative AI models under a single environment.

Key Capabilities:

• Unified Platform: Combines AutoML and custom training workflows in one interface, eliminating the need to stitch together separate services.
• Data Management: Offers managed datasets for images, text, tabular, and video data.
• Model Training: Supports both no-code (AutoML) and code-based (custom containers, pre-built frameworks like TensorFlow, PyTorch) training.
• Model Deployment: Provides endpoints for online and batch prediction with autoscaling.
• MLOps Tools: Includes Vertex AI Pipelines, Model Registry, Feature Store, and Model Monitoring for continuous evaluation.
• Generative AI Support: Integrates Generative AI Studio, Model Garden, and Gemini models.

Support Across the ML Lifecycle:

1. Data preparation — labeling and managed datasets.
2. Model development — training and tuning.
3. Deployment — scalable endpoints.
4. Monitoring — drift and skew detection.

In essence, Vertex AI reduces operational complexity and accelerates the path from experimentation to production.

Generative AI Studio is a managed environment within Vertex AI designed to help developers and non-experts quickly experiment with, tune, and deploy generative AI models.

Purpose:

• To provide a low-friction interface for testing foundation models before committing to full-scale development.
• To bridge the gap between raw model APIs and production-ready applications.

Key Features:

• Prompt Design & Testing: Interactive playground for crafting and refining prompts for text, chat, and code.
• Parameter Tuning: Adjust parameters such as temperature, top-k, top-p, and max output tokens to control creativity and determinism.
• Model Tuning: Supports fine-tuning and adapter-based tuning (e.g., prompt tuning) on custom data.
• Multimodal Support: Enables working with text, image, and code prompts.
• Export to Code: Generates SDK/API code snippets for integration into applications.

How it Helps Prototyping:

1. Rapid iteration on prompts without writing boilerplate code.
2. Immediate visual feedback on model responses.
3. Seamless transition from experiment to deployment via one-click endpoints.

This accelerates the prototype-to-production cycle for generative applications.

Gemini is Google's family of natively multimodal large language models capable of understanding and reasoning across text, images, audio, video, and code simultaneously.

Natively Multimodal Design:

• Unlike models that bolt on separate encoders after training, Gemini is trained from the ground up on multiple modalities. This allows it to seamlessly interleave and reason across different data types within a single context.
• It can, for example, analyze a chart image, read accompanying text, and produce a written summary in one pass.

Main Variants:

• Gemini Ultra: The largest and most capable model, suited for highly complex reasoning tasks.
• Gemini Pro: A balanced model optimized for a wide range of tasks with strong performance and scalability.
• Gemini Flash: A lightweight, fast, and cost-efficient model designed for high-volume, low-latency applications.
• Gemini Nano: A compact model designed for on-device deployment (e.g., mobile).

Key Strengths:

• Long context windows enabling large-document and long-video reasoning.
• Strong cross-modal reasoning.
• Advanced code understanding and generation.

This versatility makes Gemini suitable for everything from mobile apps to enterprise-grade multimodal systems.

Model Garden is a centralized repository within Vertex AI that provides a curated catalog of foundation models, task-specific models, and open-source models, all accessible from a single interface.

Role in Model Discovery:

• Catalog of Models: Includes Google's proprietary models (Gemini, Imagen, Codey), third-party models, and open-source models (e.g., Llama, Gemma).
• Search & Filter: Developers can filter models by task type (text generation, image generation, embeddings), modality, and licensing.
• Model Cards: Each model has documentation describing capabilities, limitations, use cases, and performance benchmarks.

Role in Model Selection & Use:

• One-Click Deployment: Many models can be deployed directly to Vertex AI endpoints.
• Fine-Tuning Options: Provides pathways to customize models on proprietary data.
• Try Before Deploy: Offers sample notebooks and playgrounds.

Benefits:

• Reduces the effort of finding the right model for a task.
• Provides transparency through model cards.
• Standardizes the workflow across proprietary and open models.

In short, Model Garden is the starting point for choosing the appropriate model in a generative AI project.

The End-to-End AI Development Workflow on Vertex AI covers the complete lifecycle of building and operating AI systems.

Major Stages:

1. Problem Definition & Data Collection

   • Define objectives and success metrics.
   • Gather and store data (Cloud Storage, BigQuery).

2. Data Preparation

   • Cleaning, labeling (Vertex AI Data Labeling), and feature engineering.
   • Manage features with Feature Store.

3. Model Selection / Development

   • Choose a model from Model Garden or train a custom model.
   • Use Generative AI Studio for prompt design and tuning.

4. Training & Tuning

   • Run AutoML or custom training jobs.
   • Perform hyperparameter tuning.

5. Evaluation

   • Assess accuracy, fairness, and quality metrics.
   • Compare model versions.

6. Deployment

   • Deploy to online endpoints (real-time) or batch prediction.
   • Register in Model Registry.

7. Monitoring & Maintenance

   • Track data drift, prediction skew, and latency.
   • Retrain as needed (continuous MLOps loop).

Orchestration: Vertex AI Pipelines automate and reproduce these steps as a directed graph, ensuring reproducibility and governance.

This workflow emphasizes iteration, automation, and monitoring for reliable production AI.

Text-to-Image Generation is the task of producing a realistic or stylized image that visually represents a natural-language text prompt.

Underlying Approach — Diffusion Models:

Models like Imagen use a diffusion process consisting of two phases:

• Forward Process: Gaussian noise is progressively added to a training image over steps until it becomes pure noise.
• Reverse Process: A neural network learns to denoise step by step, reconstructing a coherent image from noise while conditioned on the text prompt.

Role of Text Conditioning:

• A text encoder (often a large language model) converts the prompt into embeddings.
• These embeddings guide the denoising network so that the generated image matches the prompt semantics.

Simplified Objective:

The model is trained to predict the noise added at each step:

where is the text conditioning and is the noisy image at step .

Pipeline Steps:

1. Encode the text prompt.
2. Start from random noise.
3. Iteratively denoise conditioned on text.
4. (Optionally) upsample with super-resolution models.

Applications: Marketing assets, design prototyping, art generation, and synthetic data creation.

Image Understanding refers to a model's ability to interpret, analyze, and extract meaningful information from visual inputs, going beyond mere pixel processing to semantic comprehension.

Common Image Understanding Tasks:

• Image Captioning: Generating a natural-language description of an image.
• Visual Question Answering (VQA): Answering questions about image content.
• Object Detection & Recognition: Identifying and locating objects.
• Optical Character Recognition (OCR): Reading text within images.
• Scene & Context Analysis: Understanding relationships between objects.
• Chart & Diagram Interpretation: Extracting insights from graphs and infographics.

How Gemini Performs These Tasks:

• Gemini processes images through its native multimodal architecture, embedding visual tokens alongside text tokens.
• This allows it to reason jointly over the image and any accompanying text prompt.
• For example, given a photo of a receipt, Gemini can extract totals, categorize expenses, and answer follow-up questions.

Advantages:

• No need for separate specialized models for each task.
• Contextual reasoning across image and text.
• Handles complex composite queries (e.g., "Compare the two charts and summarize the trend").

Video Analysis involves understanding the content, actions, and temporal dynamics within video data using AI models.

Key Tasks in Video Analysis:

• Video Summarization: Producing concise descriptions of long videos.
• Action Recognition: Identifying activities and events.
• Temporal Localization: Finding when specific events occur (timestamping).
• Video Q&A: Answering questions about video content.
• Object & Scene Tracking: Following entities across frames.

Challenges:

• Temporal Dimension: Video adds time, requiring understanding of sequences and motion, not just static frames.
• Data Volume: Videos contain thousands of frames, creating heavy computational and context-length demands.
• Multimodality: Videos combine visuals, audio, and sometimes text (subtitles).
• Synchronization: Aligning audio and visual events.

Role of Long-Context Multimodal Reasoning:

• Models like Gemini offer very large context windows, enabling them to ingest long video segments (sampled frames + audio) in a single pass.
• This allows reasoning over the entire narrative rather than isolated clips.
• Cross-modal fusion lets the model connect spoken words with on-screen actions.

Applications: Content moderation, sports analysis, surveillance summarization, and educational video indexing.

Audio Processing in generative AI refers to the analysis, understanding, and generation of audio signals, including speech, music, and environmental sounds.

Common Audio Tasks:

• Automatic Speech Recognition (ASR): Converting spoken audio into text (transcription).
• Text-to-Speech (TTS): Generating natural-sounding speech from text.
• Speaker Diarization: Identifying who spoke when in a conversation.
• Audio Classification: Detecting sounds/events (e.g., alarms, music genres).
• Sentiment / Emotion Detection: Inferring tone and emotion from voice.
• Audio Summarization: Summarizing spoken content (meetings, podcasts).
• Translation: Speech-to-speech or speech-to-text translation.

How Multimodal Models Handle Audio:

• Audio is converted into representations (e.g., spectrograms or audio tokens) that the model can process alongside text.
• Native multimodal models like Gemini can accept audio input directly and reason about it in combination with text prompts.

Example Use Cases:

• Meeting transcription and action-item extraction.
• Voice assistants and accessibility tools.
• Podcast summarization and indexing.
• Call-center analytics with sentiment scoring.

Audio processing is essential for building conversational and accessibility-focused applications.

Cross-Modal Reasoning is the ability of an AI model to combine and reason over information from two or more different modalities (e.g., text, image, audio, video) to draw conclusions or generate responses that no single modality alone could support.

Key Idea:

• Instead of treating modalities separately, the model builds a shared representation space where concepts from different modalities can be compared and integrated.

Example Scenario:

Suppose a user uploads:

• A video of a cooking demonstration,
• Its audio narration, and
• A text question: "What ingredient was added right after the oil started smoking, and how much?"

Reasoning Steps:

1. Visual analysis: Detect the moment oil begins smoking in the video frames.
2. Audio analysis: Parse the narration around that timestamp.
3. Temporal alignment: Synchronize the visual event with the spoken instruction.
4. Integration: Combine cues to identify the ingredient and quantity.
5. Response: "Two teaspoons of cumin seeds were added right after the oil began smoking."

Why It Matters:

• Enables richer, context-aware applications.
• Mirrors human perception, which naturally integrates senses.
• Powers advanced assistants, robotics, and accessibility tools.

Cross-modal reasoning is the foundation of truly intelligent multimodal systems.

Multimodal Application Design is the practice of building applications that accept, process, and generate information across multiple modalities (text, image, audio, video).

Key Design Principles:

• Input Handling: Support diverse input formats and validate/normalize them (resize images, sample video frames, transcode audio).
• Prompt Engineering: Craft prompts that clearly instruct the model on how to combine modalities.
• Context Management: Efficiently manage large context windows (especially for video/audio) to stay within token limits and control cost.
• Latency & Cost: Multimodal inputs are heavier; choose appropriate model tiers (e.g., Gemini Flash for speed).
• User Experience (UX): Provide intuitive ways to upload and combine modalities and to display multimodal outputs.

Additional Considerations:

• Grounding & Accuracy: Reduce hallucinations by grounding outputs in provided data.
• Safety & Responsible AI: Filter harmful content across all modalities; respect privacy (faces, voices).
• Fallback Handling: Gracefully handle unsupported or corrupt inputs.
• Scalability: Design for autoscaling endpoints and batch processing.
• Evaluation: Define metrics for each modality and for combined outputs.

Typical Architecture:

1. Input layer (upload/preprocessing).
2. Model layer (multimodal LLM via Vertex AI).
3. Post-processing & grounding layer.
4. Presentation layer.

Good design balances capability, cost, latency, and responsibility.

Vertex AI supports two primary model development approaches: AutoML and Custom Training.

Comparison Table:

AutoML:

• Automates model selection, feature engineering, and hyperparameter tuning.
• Ideal for teams without deep ML expertise or for rapid prototyping.
• Best for common tasks: image classification, tabular prediction, text classification.

Custom Training:

• Full control over model architecture, training loop, and dependencies.
• Supports custom containers and distributed training.
• Best for research-grade or highly specialized requirements.

When to Choose:

• Choose AutoML when speed, simplicity, and standard tasks matter.
• Choose Custom Training when you need maximum flexibility, custom architectures, or advanced optimization.

Many teams start with AutoML for a baseline, then move to custom training for refinement.

In Generative AI Studio, prompt parameters control the randomness and creativity of a model's text generation. They shape how the model samples the next token from its probability distribution.

1. Temperature:

• Controls the randomness of predictions.
• Ranges typically from to (or higher).
• Low temperature (e.g., ) → deterministic, focused, repetitive outputs.
• High temperature (e.g., ) → creative, diverse, but potentially less coherent outputs.
• Mathematically, it scales the logits before softmax:

where is the temperature.

2. Top-k Sampling:

• Restricts sampling to the k most probable tokens.
• E.g., means only the top 40 candidate tokens are considered.
• Smaller → more focused output.

3. Top-p (Nucleus) Sampling:

• Selects the smallest set of tokens whose cumulative probability exceeds a threshold .
• E.g., keeps tokens making up 90% of probability mass.
• Adapts the candidate pool size dynamically.

Combined Effect:

• These parameters together balance coherence vs. creativity.
• For factual tasks: low temperature, low top-p.
• For creative writing: higher temperature, higher top-p.

Tuning them is essential for aligning output style with the application's needs.

Unimodal and multimodal models differ fundamentally in the types of data they can process.

Definitions:

• Unimodal Model: Processes a single modality (e.g., text-only or image-only).
• Multimodal Model: Processes and reasons across multiple modalities simultaneously (text, image, audio, video).

Comparison:

Advantages of Multimodal Models (Gemini):

• Richer Context: Combines information from different sources for more accurate understanding.
• Human-Like Perception: Mirrors how humans integrate sight, sound, and language.
• Versatility: A single model handles diverse tasks (captioning, VQA, transcription, summarization).
• Cross-Modal Reasoning: Can answer queries that span modalities (e.g., "Explain this chart image in words").
• Reduced Tooling: Eliminates need to chain many specialized models.

Limitations of Multimodal Models:

• Higher computational cost.
• More complex to evaluate.

Overall, multimodal models unlock applications impossible with unimodal systems, at the cost of increased resource demands.

Vertex AI Pipelines and MLOps provide the automation and governance needed to operate AI systems reliably in production.

Vertex AI Pipelines:

• Orchestrate ML workflows as a directed acyclic graph (DAG) of reusable, containerized steps.
• Automate stages: data ingestion → preprocessing → training → evaluation → deployment.
• Ensure reproducibility and versioning of every run.
• Enable CI/CD for ML (continuous integration and delivery).

MLOps (Machine Learning Operations):

• Applies DevOps principles to ML lifecycle management.
• Key components in Vertex AI:
  • Model Registry: Central versioned store of models.
  • Feature Store: Consistent feature serving for training and inference.
  • Model Monitoring: Detects drift and skew.
  • Metadata Tracking: Lineage of data and artifacts.

Why Monitoring Matters Post-Deployment:

• Data Drift: Real-world input distributions change over time, degrading accuracy.
• Prediction Skew: Differences between training and serving data.
• Performance Degradation: Latency and quality can decline.
• Responsible AI: Detect harmful or biased outputs.

Benefits:

• Early detection of problems triggers retraining.
• Maintains model reliability, fairness, and compliance.
• Closes the loop for continuous improvement.

Without monitoring, models silently decay, leading to poor decisions and user distrust.

Model Tuning is the process of adapting a pre-trained foundation model to perform better on a specific task or domain using additional training data.

Types of Tuning in Vertex AI:

• Full Fine-Tuning: Updates all model weights on task-specific data. Powerful but computationally expensive.
• Parameter-Efficient Tuning (PEFT): Updates only a small subset of parameters, keeping the base model frozen. Examples:
  • Prompt Tuning / Soft Prompts: Learns continuous prompt embeddings.
  • Adapter / LoRA: Inserts small trainable layers.

Advantages of PEFT:

• Lower compute and storage cost.
• Faster training.
• Less risk of catastrophic forgetting.

Prompting vs. Tuning:

When Tuning is Preferred:

• The task requires consistent, specialized behavior not achievable through prompts.
• You have a sufficient labeled dataset.
• Domain-specific vocabulary or style is needed (e.g., legal, medical).
• Prompt engineering alone yields inconsistent results.

When Prompting Suffices:

• Quick prototyping.
• General tasks well within the model's capabilities.
• Limited training data.

Tuning trades upfront effort for reliable, task-optimized performance.

A multimodal customer-support assistant integrates several modalities to resolve user issues efficiently. Below is a high-level architecture built on Vertex AI.

Architecture Components:

1. Input Layer (Multimodal Ingestion)

   • Accepts text (chat), images (screenshots of errors), and voice (audio queries).
   • Preprocessing: resize images, transcode audio, validate formats.

2. Preprocessing & Normalization

   • Audio → Text: Speech recognition (ASR) or direct audio input to Gemini.
   • Image handling: Extract text via OCR if needed.

3. Orchestration Layer

   • Routes inputs and manages conversation context/session state.
   • Constructs a unified multimodal prompt.

4. Model Layer (Gemini via Vertex AI)

   • Performs cross-modal reasoning — e.g., reads the screenshot, understands the spoken complaint, and consults the query.
   • Grounded with a knowledge base (RAG) of product docs.

5. Grounding / Retrieval (RAG)

   • Retrieves relevant support articles using embeddings and a vector database to reduce hallucinations.

6. Post-Processing & Safety

   • Filters unsafe content, formats the response, and may generate a voice reply (TTS).

7. Presentation Layer

   • Displays text/voice responses and suggested actions.

8. Monitoring & Feedback

   • Logs interactions, collects user feedback, and monitors for drift.

Design Considerations:

• Use Gemini Flash for low latency.
• Manage context window carefully for long sessions.
• Ensure privacy for user-uploaded screenshots and voice.

This architecture delivers a responsive, grounded, and safe support experience.

Grounding refers to connecting a generative model's outputs to verifiable, authoritative sources of information so that responses are factually accurate and traceable.

Retrieval-Augmented Generation (RAG):

RAG is a technique that augments a model's prompt with relevant external knowledge retrieved at query time.

How RAG Works:

1. Indexing: Documents are split into chunks and converted into vector embeddings, stored in a vector database.
2. Retrieval: For a user query, the query is embedded and the most semantically similar chunks are retrieved.
3. Augmentation: Retrieved chunks are inserted into the prompt as context.
4. Generation: The model generates an answer grounded in the retrieved context.

Similarity Measure:

Retrieval commonly uses cosine similarity:

Why Grounding/RAG Reduce Hallucinations:

• Hallucinations occur when models generate plausible but false information from parametric memory.
• RAG supplies factual, up-to-date context, so the model relies less on unreliable internal memory.
• Enables citations, improving trust and verifiability.
• Allows use of private/enterprise data without retraining.

Benefits:

• Improved accuracy and freshness.
• Domain adaptation without fine-tuning.
• Transparency through source attribution.

Grounding and RAG are essential for building trustworthy enterprise generative AI applications.

Responsible AI ensures that generative AI systems are safe, fair, transparent, and aligned with ethical and legal standards.

Key Responsible AI Considerations:

1. Safety Filtering

   • Detect and block harmful, hateful, sexual, or violent content across text, image, audio, and video.
   • Vertex AI provides configurable safety filters and thresholds.

2. Bias and Fairness

   • Foundation models may reflect biases in training data.
   • Evaluate outputs across demographic groups and mitigate discriminatory behavior.

3. Privacy

   • Protect PII in inputs (faces in images, voices in audio).
   • Comply with regulations (GDPR, etc.); avoid storing sensitive data unnecessarily.

4. Transparency & Explainability

   • Provide model cards and, where possible, citations (via grounding/RAG).
   • Inform users they are interacting with AI.

5. Hallucination Mitigation

   • Use grounding, RAG, and human review for high-stakes decisions.

6. Accountability & Governance

   • Maintain audit logs, model lineage, and access controls.

7. Misuse Prevention

   • Guard against generation of deepfakes, misinformation, or malicious content.

Practical Measures in Vertex AI:

• Built-in safety attributes and content moderation.
• Human-in-the-loop review for sensitive outputs.
• Monitoring for drift and harmful outputs.

Conclusion:

Responsible AI is not optional — it is central to building trustworthy, compliant, and socially beneficial generative applications.

The Gemini family offers multiple variants optimized for different trade-offs between capability, speed, and cost.

Comparison Table:

Detailed Notes:

• Gemini Ultra: Best for the most demanding tasks requiring deep reasoning and nuanced multimodal understanding. Resource-intensive.

• Gemini Pro: A versatile workhorse balancing quality and efficiency — suitable for most production applications.

• Gemini Flash: Designed for low latency and cost efficiency at scale, ideal for interactive and high-throughput workloads. Slightly lower reasoning depth than Pro/Ultra.

• Gemini Nano: Lightweight enough to run on-device (e.g., smartphones), enabling privacy-preserving, offline inference.

Selection Guidance:

• Prioritize quality → Ultra/Pro.
• Prioritize speed & cost → Flash.
• Need on-device/offline → Nano.

Choosing the right variant is a key part of balancing performance, cost, and user experience in application design.