Unit 6: Advanced Generative AI Applications

I. Orientation: Enterprise Generative AI on a Managed Platform

Vertex AI is Google Cloud's unified machine-learning platform (generally available 2021, generative features expanded 2023–2024) that consolidates data preparation, model training, tuning, serving and monitoring behind one API surface. This unit treats generative AI as a productionable discipline: not just prompting a model, but building, evaluating and deploying multimodal applications on managed infrastructure.

The unit depends on the following recurring ideas:

• Managed foundation models: Pre-trained large models exposed as APIs so teams consume capability without owning training infrastructure.
• Multimodality: A single model accepts and reasons over mixed inputs — text, image, video, audio — rather than one modality per model.
• Prompt-to-endpoint lifecycle: Every project moves from experimentation to a governed, monitored deployment.
• Grounding and evaluation: Outputs are anchored to trusted data and scored against metrics, because generative outputs are probabilistic.
• Tokens and context window: Inputs and outputs are billed and bounded in tokens; long context enables whole-document or whole-video reasoning.

II. The Vertex AI Development Environment

The tooling layer that turns foundation models into applications.

A. Overview of Vertex AI

Vertex AI is the umbrella platform providing the compute, storage and MLOps services around generative models.

• Unified API: One SDK (google-cloud-aiplatform) and REST surface for training, tuning, prediction and generation, replacing separate legacy AI Platform and AutoML products.
• Core services: Feature Store (feature reuse), Pipelines (Kubeflow-based orchestration), Model Registry (versioned model catalogue), Endpoints (scalable serving) and Experiments (run tracking).
• Managed hardware: Access to GPUs and Google TPUs without provisioning clusters manually.
• Governance: IAM roles, VPC Service Controls and audit logging enforce who can call which model and where data flows.

B. Generative AI Studio

Generative AI Studio is the low-code console surface for prototyping prompts and tuning before writing production code.

• Prompt design: Freeform and structured prompt editors let you set the system instruction, few-shot examples and the user turn interactively.
• Parameter controls: Sliders expose temperature (randomness), top_k/top_p (candidate token filtering), max_output_tokens and safety settings.
• Modality tabs: Separate workspaces for language, vision and speech tasks.
• Export path: Any tuned prompt exports to Python, Node.js or curl so the studio experiment becomes application code directly.

C. Gemini Models

Gemini is Google's family of natively multimodal foundation models, trained from the start on text, images, audio and video together.

• Native multimodality: A single forward pass interleaves modalities, e.g. a prompt can mix a paragraph and a photograph without a separate vision encoder pipeline.
• Model tiers: Ultra (highest capability), Pro (balanced general-purpose), Flash (low-latency, cost-optimised) and Nano (on-device) trade capability against latency and cost.
• Long context window: Pro tiers accept very large context (hundreds of thousands to over a million tokens), enabling whole-book or hour-long-video inputs.
• Function calling: The model can emit a structured JSON call describing a tool to invoke, letting it act as a reasoning core over external APIs.

from vertexai.generative_models import GenerativeModel, Part
model = GenerativeModel("gemini-1.5-pro")
resp = model.generate_content(
    [Part.from_uri("gs://bucket/chart.png", "image/png"),
     "Summarise the trend shown."])
print(resp.text)

D. Model Garden

Model Garden is the discoverable catalogue of models available to deploy from within Vertex AI.

• Three sources: Google first-party models (Gemini, Imagen), curated open models (e.g. Llama, Gemma) and partner models.
• Task filtering: Models are tagged by modality and task — generation, classification, embedding — to shortlist candidates.
• One-click deploy: A model card exposes sample code, licence terms and a deploy button that provisions an endpoint.
• Build-vs-buy decision point: Garden is where a team chooses between a managed API model and a self-hosted open-weights model based on cost, control and licence.

E. End-to-End AI Development Workflow

The workflow is the ordered lifecycle from problem framing to monitored production.

• 1. Data preparation: Ingest and clean data; for tuning, format into prompt/response pairs stored in Cloud Storage as JSONL.
• 2. Prototype: Design prompts in Generative AI Studio and pick a base model from Model Garden.
• 3. Tune or ground: Apply supervised fine-tuning or parameter-efficient tuning, or attach retrieval grounding instead of full retraining.
• 4. Evaluate: Score outputs with automatic metrics (ROUGE, BLEU) or model-based rubric evaluation on a held-out set.
• 5. Deploy: Register the version in Model Registry and expose a scalable Endpoint.
• 6. Monitor: Track latency, cost per token, and input/output drift; feed failures back into step 1.

III. Multimodal Generation and Understanding

The capability categories the platform's models expose.

Each capability below is a distinct input/output modality pairing; the same Gemini or specialised model is invoked through the shared API with different Part types.

A. Text-to-Image Generation

Generating novel raster images from a natural-language description, served by the Imagen model family.

• Prompt engineering for images: Descriptors for subject, style, lighting and composition steer output, e.g. "isometric, soft studio lighting, pastel palette".
• Diffusion basis: Imagen uses a denoising diffusion process, iteratively removing noise conditioned on the text embedding.
• Controls: number_of_images, aspect_ratio, seed (reproducibility) and negative prompts to exclude features.
• Editing modes: Inpainting (fill a masked region) and outpainting (extend beyond original borders) extend generation to editing.
• Safety: Generated images carry an invisible SynthID watermark to mark them as AI-produced.

B. Image Understanding

Interpreting an existing image to answer questions or extract structure, a vision-to-text task.

• Visual question answering: The model answers questions grounded in image content, e.g. "How many people wear helmets?"
• Captioning and OCR: Produces descriptive captions and reads embedded text from documents or signage.
• Object and relationship detection: Identifies entities and their spatial relations, enabling scene description.
• Worked example: Passing a receipt image with the prompt "Return line items and totals as JSON" yields structured data — combining OCR and reasoning in one call.

C. Video Analysis

Reasoning over video, where the model samples frames plus audio track across time.

• Temporal understanding: Answers questions requiring event ordering, e.g. "What happened before the door opened?"
• Timestamped output: Can return moments as MM:SS references, supporting chapterisation and highlight extraction.
• Frame sampling: Video is decomposed into frames at a set rate; long context lets the model hold an entire clip.
• Use cases: Summarising lectures, moderating content, indexing footage for search.

D. Audio Processing

Handling speech and non-speech audio as model input or output.

• Transcription: Speech-to-text converts spoken audio to written text with speaker and timestamp metadata.
• Audio understanding: Beyond words, the model can describe tone, identify non-speech sounds and answer questions about a recording.
• Speech synthesis: Text-to-speech produces natural audio output, closing the loop for voice interfaces.
• Direct ingestion: Gemini accepts an audio Part directly, so "summarise this meeting" needs no separate transcription step.

E. Cross-Modal Reasoning

Drawing conclusions that require combining evidence from two or more modalities simultaneously.

• Definition: The answer exists in none of the inputs alone but only in their relationship, e.g. matching a spoken instruction to an object visible in a photo.
• Grounded alignment: The model aligns entities across modalities — the word "the red car" is bound to pixels and to a spoken mention.
• Example task: Given a chart image and a narrated commentary, "State where the speaker's claim disagrees with the chart" requires jointly parsing image and audio.
• Enabling factor: Native multimodal pretraining, not late fusion of separate models, is what makes this reliable.

IV. Multimodal Application Design

Assembling capabilities into robust products.

A. Multimodal Application Design

Designing systems that route mixed user inputs to model capabilities and return grounded, safe responses.

• Input routing: A front layer classifies incoming modality and constructs the appropriate Part list for the model call.
• Grounding with RAG: Retrieval-Augmented Generation fetches relevant documents or embeddings and injects them into context so answers cite trusted sources rather than hallucinating.
• Orchestration and tools: Function calling connects the model to databases, search and business APIs, turning it into an agent that acts on requests.
• Latency and cost design: Route simple turns to Flash and complex reasoning to Pro; cache repeated context to cut token cost.
• Responsible-AI controls: Configurable safety filters, human-in-the-loop review for high-stakes outputs, and watermarking of generated media.
• Significance and limitations: Multimodal design collapses several formerly separate pipelines into one model call, but constraints remain — bounded context windows, per-token cost at scale, residual hallucination risk, and the need for domain grounding and continuous evaluation to keep outputs trustworthy.