DEV Community: Google Developer Experts

Stop Calling getGenerativeModel() Everywhere! The Hidden Traps of Scaling Firebase AI to Multi-Model Angular Apps

Connie Leung — Wed, 11 Nov 2026 10:53:39 +0000

Calling getGenerativeModel() directly inside VisionService and TextToSpeechService duplicates firebase/ai imports, safety settings, and thinking configurations while recreating client-side GenerativeModel objects on every request and triggering unnecessary garbage collection. In this post, we extract AiModelFactory to centralize and cache model instances for multi-model routing, and we resolve two hidden caching traps: request-level generationConfig overwrites and stale AI backends after Remote Config updates.

Architecture & Mental Model

VisionService and TextToSpeechService delegate model creation and lifecycle management to AiModelFactory:

On-Demand Model Requests: VisionService calls getModel(options) with AiModelOptions, while TextToSpeechService calls getTtsModel().
Backend Validation and Cache Eviction (getValidBackend()): Before returning a model, getValidBackend() compares ConfigService.getAiBackend() against #cachedBackend. If Remote Config refreshes the AI backend, it clears #models and resets #ttsModel to null.
Keyed Vision Model Cache (#models): getModel(options) builds a deterministic key via generateKey() and caches instances in #models (Map<string, GenerativeModel>).
Singleton TTS Model Cache (#ttsModel): getTtsModel() caches a single #ttsModel instance configured with responseModalities: [ResponseModality.AUDIO].

flowchart TD
    subgraph Callers["Domain Services"]
        VS["VisionService (1 model now -> 3 models with MediaPipe routing)"]
        TTS["TextToSpeechService (per-request voice speechConfig)"]
    end

    subgraph Factory["AiModelFactory (@/core/firebase)"]
        GVB["getValidBackend(): compare with #cachedBackend"]
        EVICT["Backend changed? #models.clear() & #ttsModel = null"]
        GM["getModel(options?: AiModelOptions)"]
        KEY["generateKey(): model_thinking_instruction_schema_tools"]
        MAP[("#models: Map<string, GenerativeModel>")]
        GTTS["getTtsModel()"]
        TTS_CACHE[("#ttsModel: GenerativeModel | null")]
    end

    VS -- "getModel(options)" --> GVB
    TTS -- "getTtsModel()" --> GVB
    GVB -- "New AI instance" --> EVICT --> GM & GTTS
    GVB -- "Same AI instance" --> GM & GTTS
    GM --> KEY --> MAP
    GTTS --> TTS_CACHE

1. Defining `AiModelOptions` and Deterministic Cache Keys

1.1. The `AiModelOptions` Interface

The AiModelOptions interface defines the configuration needed to create a vision model and compute its cache key:

Optional Configuration Properties: systemInstruction, schema (SchemaRequest | TypedSchema), and the tools array (Tool[]) let callers configure structured JSON output, plain-text responses, or grounding tools.

// src/app/core/firebase/ai-model-options.interface.ts
import type { SchemaRequest, Tool, TypedSchema } from 'firebase/ai';

export interface AiModelOptions {
  systemInstruction?: string;
  schema?: SchemaRequest | TypedSchema;
  tools?: Tool[];
}

1.2. Private State and `generateKey()` in `AiModelFactory`

Configuration Injection (#configService): AiModelFactory injects ConfigService into #configService to read Remote Config values and obtain the Firebase AI Logic backend.
Cached AI Backend (#cachedBackend): #cachedBackend stores the active AI backend reference used to create generative models.
Singleton TTS Cache (#ttsModel): #ttsModel holds the single-instance TTS model used to synthesize speech from English text.
Vision Model Map (#models and generateKey): #models is a Map<string, GenerativeModel> that stores vision models by the key returned from generateKey(). The key joins the model name (geminiModelName), thinkingLevel, systemInstruction, the structured-output schema, and tools with underscore (_) characters.

// src/app/core/firebase/ai-model-factory.service.ts (fields & key generation)
import type { AI, GenerationConfig, GenerativeModel, ModelParams, SafetySetting } from 'firebase/ai';

@Service()
export class AiModelFactory {
  readonly #configService = inject(ConfigService);
  readonly #models = new Map<string, GenerativeModel>();
  #ttsModel: GenerativeModel | null = null;
  #cachedBackend: AI | null = null;

  private generateKey(options?: AiModelOptions): string {
    const { geminiModelName, thinkingLevel } = this.#configService.appConfig;
    const instruction = options?.systemInstruction ?? '';
    const schemaKey = options?.schema ? JSON.stringify(options.schema) : '';
    const toolsKey = options?.tools?.length ? JSON.stringify(options.tools) : '';

    return `${geminiModelName}_${thinkingLevel}_${instruction}_${schemaKey}_${toolsKey}`;
  }

2. Creating Models On Demand and Invalidating the Cache in `AiModelFactory`

2.1. Lazily Loading the SDK and Assembling Parameters (`createModel`, `createModelParams`, `createGenerationConfig`)

It is tempting to import the HarmBlockThreshold and HarmCategory enums at the top of the file, but doing so leaks firebase/ai runtime code into main.js and increases the initial bundle size:

Dynamic SDK Import (createModel): createModel() dynamically imports getGenerativeModel, HarmBlockThreshold, and HarmCategory from firebase/ai.
Shared Safety Settings (createModel): createModel() builds the SafetySetting[] array and calls getGenerativeModel(backend, modelParams).
Thinking Config (createGenerationConfig): createGenerationConfig() configures thinkingLevel with includeThoughts: true.
Structured Output Schema (createGenerationConfig): createGenerationConfig() attaches responseSchema and responseMimeType: 'application/json' when the caller provides options?.schema.
Model Parameter Assembly (createModelParams): createModelParams() combines the model name, generationConfig, and safetySettings, and conditionally adds tools and systemInstruction.

  // src/app/core/firebase/ai-model-factory.service.ts (model creation helpers)
  private async createModel(backend: AI, options?: AiModelOptions): Promise<GenerativeModel> {
    const { getGenerativeModel, HarmBlockThreshold, HarmCategory } = await import('firebase/ai');
    const safetySettings: SafetySetting[] = [
      { category: HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT, threshold: HarmBlockThreshold.BLOCK_ONLY_HIGH },
      { category: HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT, threshold: HarmBlockThreshold.BLOCK_ONLY_HIGH },
      { category: HarmCategory.HARM_CATEGORY_HARASSMENT, threshold: HarmBlockThreshold.BLOCK_ONLY_HIGH },
      { category: HarmCategory.HARM_CATEGORY_HATE_SPEECH, threshold: HarmBlockThreshold.BLOCK_ONLY_HIGH },
    ];

    const modelParams = this.createModelParams(safetySettings, options);
    return getGenerativeModel(backend, modelParams);
  }

  private createModelParams(safetySettings: SafetySetting[], options: AiModelOptions | undefined): ModelParams {
    const generationConfig = this.createGenerationConfig(options);

    const modelParams: ModelParams = {
      model: this.#configService.appConfig.geminiModelName,
      generationConfig,
      safetySettings,
    };

    if (options?.tools?.length) {
      modelParams.tools = options.tools;
    }

    if (options?.systemInstruction) {
      modelParams.systemInstruction = options.systemInstruction;
    }
    return modelParams;
  }

  private createGenerationConfig(options: AiModelOptions | undefined): GenerationConfig {
    const generationConfig: GenerationConfig = {
      thinkingConfig: {
        thinkingLevel: this.#configService.appConfig.thinkingLevel,
        includeThoughts: true,
      },
    };

    if (options?.schema) {
      generationConfig.responseSchema = options.schema;
      generationConfig.responseMimeType = 'application/json';
    }
    return generationConfig;
  }

2.2. Reusing Models On Demand (`getModel` and `getTtsModel`)

Obtaining a vision model and a TTS model follows two different caching flows in AiModelFactory:

Vision Model Lookup (getModel): getModel() generates a deterministic cache key from AiModelOptions via generateKey(options) and checks this.#models.get(key).
Vision Model Cache Miss: When the key is not in #models, getModel() calls createModel() to instantiate the model and stores it in the map.
TTS Model Flow (getTtsModel): getTtsModel() calls createTtsModel() once to instantiate a singleton TTS model configured with responseModalities: [ResponseModality.AUDIO] and assigns it to #ttsModel.
Eliminating Per-Call Allocation and Garbage Collection: Reusing cached instances across requests avoids repeated getGenerativeModel() calls and unnecessary garbage collection.

  // src/app/core/firebase/ai-model-factory.service.ts (on-demand getters)
  async getModel(options?: AiModelOptions): Promise<GenerativeModel> {
    const backend = await this.getValidBackend();
    const key = this.generateKey(options);
    let model = this.#models.get(key);
    if (!model) {
      model = await this.createModel(backend, options);
      this.#models.set(key, model);
    }

    return model;
  }

  private async createTtsModel(ai: AI): Promise<GenerativeModel> {
    const { getGenerativeModel, ResponseModality } = await import('firebase/ai');

    return getGenerativeModel(ai, {
      model: this.#configService.appConfig.geminiTTSModelName,
      generationConfig: {
        responseModalities: [ResponseModality.AUDIO],
      },
    });
  }

  async getTtsModel(): Promise<GenerativeModel> {
    const backend = await this.getValidBackend();
    if (!this.#ttsModel) {
      this.#ttsModel = await this.createTtsModel(backend);
    }

    return this.#ttsModel;
  }

2.3. Evicting Cached Models When Remote Config Rotates the Backend (`getValidBackend`)

Stale Backend on Remote Config Refresh: Constructing a GenerativeModel requires a valid Firebase AI Logic AI backend, which becomes outdated when dependent Firebase Remote Config parameters (such as useLimitedUseAppCheckTokens and vertexAILocation) receive updated values and ConfigService resets its AI instance.
Purging Cached Models (getValidBackend): When getValidBackend() detects that this.#configService.getAiBackend() no longer matches #cachedBackend, it updates #cachedBackend, clears the #models map, and resets #ttsModel to null.

  // src/app/core/firebase/ai-model-factory.service.ts (cache invalidation)
  private async getValidBackend(): Promise<AI> {
    const backend = await this.#configService.getAiBackend();
    if (backend !== this.#cachedBackend) {
      this.#cachedBackend = backend;
      this.#models.clear();
      this.#ttsModel = null;
    }

    return this.#cachedBackend;
  }
}

3. Refactoring `VisionService` and `TextToSpeechService` (and the Shallow-Spread Trap)

3.1. Simplifying `VisionService` for Future Multi-Model Routing

We refactored VisionService for extensibility and scalability by delegating model creation to AiModelFactory:

Removing SDK Setup Boilerplate: Introducing AiModelFactory removes SAFETY_SETTINGS, ConfigService, and the private getGenerativeAIModel() helper from VisionService.
Extracting altTextPrompt: Moving altTextPrompt to vision.prompt.ts keeps VisionService focused on orchestrating image analysis—generating tags, writing accessible alt text, and finding an obscure fact.
On-Demand Model Retrieval with Custom Tools: VisionService injects AiModelFactory into #aiModelFactory and calls this.#aiModelFactory.getModel({ schema: ImageAnalysisSchema, tools: [{ googleSearch: {} }] }). While the current model uses the Google Search grounding tool, VisionService can pass a Google Maps tool or custom function-calling tools through AiModelOptions to configure additional models.
Ready for MediaPipe Multi-Model Routing: Because getModel() caches instances by their configuration key, we can extend VisionService with an on-device MediaPipe classifier to request 3 specialized models (places, people and faces, and general objects) on demand.

// src/app/core/vision/vision.service.ts (refactored excerpt)
@Service()
export class VisionService {
  readonly #window = inject(WINDOW);
  readonly #aiModelFactory = inject(AiModelFactory);

  async analyzeImage(image: File): Promise<ImageAnalysisResponse> {
    const { data, mimeType, optimizationMetrics } = await preprocessImageForVision(image, this.#window);
    const imagePart: InlineDataPart = {
      type: 'inlineData',
      inlineData: { data, mimeType },
    };

    const aiModel = await this.#aiModelFactory.getModel({
      schema: ImageAnalysisSchema,
      tools: [{ googleSearch: {} }],
    });
    const result = await aiModel.generateContent([altTextPrompt, imagePart]);
    // ... parse response, tokenUsage, and grounding citations
  }
}

3.2. Reusing `#ttsModel` in `TextToSpeechService` Without Wiping Out `responseModalities`

Similarly, TextToSpeechService delegates TTS model creation to AiModelFactory while passing the selected voice per request:

Shared TTS Model Lookup: TextToSpeechService injects AiModelFactory into #aiModelFactory, and createAIModelGenerationRequest() retrieves the cached TTS model via this.#aiModelFactory.getTtsModel() instead of instantiating a new model for every synthesis call.
Per-Request speechConfig Assembly: The createAIModelGenerationRequest(text, voiceName) helper builds the GenerateContentRequest (shared by synthesizeStream() and synthesize()) and configures speechConfig with the requested voiceName and languageCode: 'en-US'.
Preserving ResponseModality.AUDIO Against Shallow Overwrites: In @firebase/ai, passing a request-level generationConfig overwrites the model's default aiModel.generationConfig. Spreading ...aiModel.generationConfig into the request's generationConfig preserves responseModalities: [ResponseModality.AUDIO] alongside speechConfig.

// src/app/core/speech/text-to-speech.service.ts (refactored excerpt)
@Service()
export class TextToSpeechService {
  readonly #aiModelFactory = inject(AiModelFactory);

  private async createAIModelGenerationRequest(text: string, voiceName: string): Promise<AIModelGenerativeRequest> {
    const aiModel = await this.#aiModelFactory.getTtsModel();
    const contents: Content[] = [
      {
        role: 'user',
        parts: [{ type: 'text', text }],
      },
    ];
    const generationConfig: GenerationConfig = {
      ...aiModel.generationConfig,
      speechConfig: {
        voiceConfig: {
          prebuiltVoiceConfig: { voiceName },
        },
        languageCode: 'en-US',
      },
    };

    return {
      aiModel,
      generationRequest: { contents, generationConfig },
    };
  }

  async *synthesizeStream({ text, voice }: SpeechPrompt): AsyncGenerator<AudioStreamChunk> {
    const { aiModel, generationRequest } = await this.createAIModelGenerationRequest(text, voice);
    const responseStream = await aiModel.generateContentStream(generationRequest);

    for await (const chunk of responseStream.stream) {
      const chunkData = this.extractValidChunkData(chunk);
      if (chunkData) {
        yield decodeAudioChunk(chunkData.data, chunkData.mimeType);
      }
    }
  }

  async synthesize({ text, voice }: SpeechPrompt): Promise<Blob> {
    const { aiModel, generationRequest } = await this.createAIModelGenerationRequest(text, voice);
    const result = await aiModel.generateContent(generationRequest);

    const chunk = this.extractValidChunkData(result.response);
    if (!chunk) {
      throw new Error('No audio data received in response.');
    }
    return toWavBlob(chunk.data, chunk.mimeType);
  }
}

Takeaways

Centralized Lazy Model Creation: AiModelFactory dynamically imports firebase/ai to construct both text-to-speech and image-analysis models. It centralizes safety settings and thinkingLevel for the image-analysis model while locking responseModalities to [ResponseModality.AUDIO] for the TTS model.
Instance Reuse Over Per-Request Allocation: AiModelFactory caches models in the #models map or the #ttsModel field and reuses them across requests, eliminating unnecessary per-request GenerativeModel instantiation and garbage collection.
Scalable Multi-Model Routing for VisionService: Keyed caching in the #models map makes VisionService scalable so an upcoming MediaPipe classifier can route images to specialized models (places, people and faces, and general objects) for analysis.
Preserving ResponseModality.AUDIO Across Per-Request Voices: Because text-to-speech synthesis accepts a voice per request, spreading ...aiModel.generationConfig into the request's generationConfig preserves ResponseModality.AUDIO alongside the selected voiceName.
Automatic Cache Invalidation on Remote Config Updates: When ConfigService fetches Remote Config values in the background and invalidates the cached AI backend, AiModelFactory purges all cached models and recreates fresh instances on the next call.

Resources

Google AI Classroom Part 10: Green AI FinOps, Self-Cleaning Storage Lifecycles & 1-Command Terraform IaC for Multi-School Deployment

Cyrus Wong — Wed, 11 Nov 2026 08:21:46 +0000

Author: Cyrus Wong (黃俊彥) — Google Developer Expert - Cloud (AI), Senior Lecturer at HKIIT

1. The Three Pillars of Institutional Day-2 Production Readiness

Building a flashy AI classroom demo for a hackathon takes a weekend; operating a production AI platform for teaching, learning, student performance evaluation, and proctored examinations across an academic department with dozens of lecturers, thousands of students, and strict public-institution budget governance requires three Day-2 Engineering Guarantees:

Granular FinOps & Atomic Quota Circuit Breakers: Department administrators must see exact, itemized USD spending down to the sixth decimal place across 🧠 Gemini 4 Argon (gemini-4-argon), ⚡ Gemini 3.8 Flash (gemini-3.8-flash), 🪶 Gemini 3.5 Flash-Lite (gemini-3.5-flash-lite), and 🎙️ Gemini 3.5 Transcribe (gemini-3.5-transcribe-preview), combined with real-time Cloud Storage byte tracking across 9 distinct media categories and automated circuit breakers that halt runaway spending before a single class can exceed its budget.
Self-Cleaning Storage Lifecycles & Zero-Orphan Cascading Purge: Over a semester of recorded lectures, practical task grading videos, capstone demos, and proctored exam evidence, classes generate thousands of media files. Student screenshots, lab videos, voice consultations, and exam clips must expire automatically via configurable TTL policies after grading and exam appeals conclude—and deleting a class at the end of an academic year must purge 100% of associated Cloud Storage blobs and 20+ Firestore subcollections/collections with zero orphaned records.
1-Command Reproducible Terraform Infrastructure-as-Code (./setup-new-project.sh) & Continuous Verification: Any university, polytechnic, or Google Developer community worldwide must be able to provision a fresh Google Cloud project—enabling all 27 APIs, Firestore Native databases, Cloud Storage buckets, Identity Platform (GCIP), and a 24/7 Demo Sandbox—with a single terminal command, guarded by automated unit, security rules, and Playwright E2E test suites.

In Part 10 (the finale of our 10-part Google Developer Expert series), we walk through the exact production code and infrastructure patterns that deliver all three guarantees.

2. Diagram 1: Real-Time AI FinOps Pipeline, Live Cloud Billing SKU Sync & Atomic Quota Circuit Breakers

In scheduledTasks.js, cost.js, jobLogger.js, useCloudPricing.js, and aiCostAggregator.js, every AI invocation and storage event is metered in real time against live Google Cloud Billing Catalog SKU rates:

Production Code 1: Live Google Cloud Billing Catalog SKU Sync (`syncGeminiPricing`) & Multi-Stage Cost Engine (`cost.js`)

Hardcoding cloud pricing constants in frontend code inevitably drifts as new model tiers launch or regional storage SKU rates change. Every 24 hours, syncGeminiPricing in functions/scheduled_tasks/scheduledTasks.js queries the official Google Cloud Billing Catalog API v1 for both the Gemini / Cloud AI service (C7E2-9256-1C43) and Regional Standard Cloud Storage in Hong Kong (95FF-2EF5-5EA1, asia-east2), writing the authoritative rate table into system_config/pricing:

// Source: functions/scheduled_tasks/scheduledTasks.js
export const syncGeminiPricing = onSchedule({
  schedule: 'every 24 hours',
  memory: '256MB',
  region: FUNCTION_REGION,
}, async () => {
  const VERTEX_SERVICE_ID = 'C7E2-9256-1C43';
  const STORAGE_SERVICE_ID = '95FF-2EF5-5EA1';
  const pricingData = {
    'gemini-4-argon': { input: 2.00, output: 10.00 },
    'gemini-3.5-flash-lite': { input: 0.30, output: 2.50 },
    'gemini-3.8-flash': { input: 0.75, output: 3.75 },
    'gemini-3.7-pro': { input: 3.00, output: 15.00 },
    'gemini-3.5-transcribe-preview': { input: 0.50, output: 2.50 },
    'cloud-storage': {
      unit: 'GiB/month',
      ratePerGibMonth: 0.023,
      region: 'asia-east2',
      currency: 'USD',
      description: 'Standard Storage Hong Kong (Baseline)',
    },
    lastSyncedAt: new Date().toISOString(),
    source: 'catalog_sync_or_baseline',
  };

  // Queries https://cloudbilling.googleapis.com/v1/services/{SERVICE_ID}/skus
  // Extracts live nanos/units tieredRates for asia-east2 & updates system_config/pricing
  await db.collection('system_config').doc('pricing').set(pricingData, { merge: true });
});

On the AI execution plane, functions/ai_flows/cost.js maintains a 12-hour warm memory cache of system_config/pricing, normalizes token usage metadata across Genkit (inputTokens/outputTokens), the Google GenAI SDK (promptTokenCount/candidatesTokenCount), and composite two-stage pipelines (gemini-3.5-transcribe-preview + gemini-4-argon), and computes exact sub-cent USD cost (Number((inputCost + outputCost).toFixed(6))):

// Source: functions/ai_flows/cost.js
export const MODEL_PRICING = {
  'gemini-4-argon': { input: 2.00, output: 10.00 },
  'gemini-3.5-flash-lite': { input: 0.30, output: 2.50 },
  'gemini-3.8-flash': { input: 0.75, output: 3.75 },
  'gemini-3.7-flash': { input: 0.75, output: 3.75 },
  'gemini-3.7-pro': { input: 3.00, output: 15.00 },
  'gemini-3.5-transcribe': { input: 0.50, output: 2.50 },
  'gemini-3.5-transcribe-preview': { input: 0.50, output: 2.50 },
};

export function calculateCost(usageMetadata, model = DEFAULT_MODEL) {
  if (!usageMetadata || typeof usageMetadata !== 'object') return 0;
  const pricing = getModelPricing(model);
  const promptTokens = Math.max(
    0,
    Number(usageMetadata.promptTokenCount ?? usageMetadata.inputTokens ?? usageMetadata.promptTokens ?? 0)
  );
  const candidatesTokens = Math.max(
    0,
    Number(usageMetadata.candidatesTokenCount ?? usageMetadata.outputTokens ?? usageMetadata.completionTokens ?? 0)
  );
  const inputCost = (promptTokens / 1_000_000) * (pricing?.input ?? MODEL_PRICING[DEFAULT_MODEL].input);
  const outputCost = (candidatesTokens / 1_000_000) * (pricing?.output ?? MODEL_PRICING[DEFAULT_MODEL].output);
  return Number((inputCost + outputCost).toFixed(6)) || 0;
}

In AiCostReportView.jsx, lecturers and system administrators can inspect itemized AI token costs alongside an interactive 1–12 Month Cloud Storage Holding Period Slider, viewing exact storage & AI costs across 18 categories—including 👥 Project Meeting Audio, Demos & Photos (voiceClipsCount, demoClipsCount, photosCount) and 📋 Task Rubric & Submissions.

3. Diagram 2: 9-Category Event-Driven Storage Quota & 4-Stage Cascading Class Purge (`onClassDocDeleted`)

How do we prevent a class from uploading unlimited gigabytes of video, and how do we guarantee zero orphaned files when a class is deleted?

Production Code 2: 9-Category Event-Driven Cloud Storage Quota Metering & Self-Healing Non-Negative Drift Guard (`storageQuota.js`)

Every time a file is finalized or deleted in our Google Cloud Storage bucket, updateStorageUsageOnUpload and updateStorageUsageOnDelete in functions/storage_triggers/storageQuota.js classify the blob path into one of 9 granular storage categories via getStorageCategoryAndClass(filePath):

Storage Path Prefix	Firestore Metadata Counter (`classes/{id}/metadata/storage`)	Classroom Asset Category
`screenshots/{classId}/...`	`storageUsageScreenShots`	Raw periodic screen & webcam JPEGs (auto-purged after MP4 compilation)
`videos/{classId}/...`	`storageUsageVideos`	Compiled 1-FPS H.264 student session timelapse MP4s
`recordings/{classId}/...` & `subtitles/{classId}/...`	`storageUsageRecordings`	Teacher lecture recordings, normalized M4A audio & `.vtt`/`.srt` subtitles
`audio/{classId}/...`	`storageUsageAudio`	Student acoustic anomaly snippets & session audio chunks
`irregularities/{classId}/...`	`storageUsageIrregularities`	High-severity proctoring incident snapshots
`zips/{classId}/...`	`storageUsageZips`	Bulk evidence archive `.zip` exports (24-hour auto-TTL)
`classes/{classId}/tasks/...`, `tasks/...`, `submissions/...`	`storageUsageTasks`	Practical task reference demo videos & sliced student attempt MP4s
`reports/{classId}/...`	`storageUsageReports`	Court-ready `.docx` & `.csv` forensic incident dossiers
`projectMeetings/{classId}/...`	`storageUsageProjectMeetings`	Capstone consultation voice clips, `720p` mobile camera demo videos & `0.1s` photos

If an uploaded file pushes storageUsage above the class's configured storageQuota, updateStorageUsageOnUpload immediately deletes the offending blob from Cloud Storage and atomically reverts the counter via FieldValue.increment(-fileSize)! Furthermore, updateStorageUsageOnDelete includes a Self-Healing Non-Negative Drift Guard so out-of-order deletion events can never drive any category counter below 0:

// Source: functions/storage_triggers/storageQuota.js
export function getStorageCategoryAndClass(filePath) {
  if (!filePath || typeof filePath !== 'string') return null;
  const parts = filePath.split('/');
  if (parts.length < 3) return null;
  if (parts[0] === 'classes') return { classId: parts[1], usageField: 'storageUsageTasks' };

  const classId = parts[1];
  const map = {
    screenshots: 'storageUsageScreenShots',
    videos: 'storageUsageVideos',
    recordings: 'storageUsageRecordings',
    subtitles: 'storageUsageRecordings',
    audio: 'storageUsageAudio',
    irregularities: 'storageUsageIrregularities',
    zips: 'storageUsageZips',
    tasks: 'storageUsageTasks',
    submissions: 'storageUsageTasks',
    reports: 'storageUsageReports',
    projectMeetings: 'storageUsageProjectMeetings',
  };
  return map[parts[0]] ? { classId, usageField: map[parts[0]] } : null;
}

And if a lecturer or system administrator ever wants to verify that Firestore's event-driven byte counters match the physical bytes stored in the Google Cloud Storage bucket after a bulk manual archive purge, clicking 🔄 Audit & Recalculate Storage invokes recalculateStorageUsage in functions/storage_triggers/storageQuota.js, scanning every physical prefix in parallel and stamping lastAuditedAt:

// Source: functions/storage_triggers/storageQuota.js
export async function recalculateStorageUsageInternal(classId) {
  const bucket = adminStorage.bucket();
  const categories = [
    { prefix: `screenshots/${classId}/`, field: 'storageUsageScreenShots' },
    { prefix: `videos/${classId}/`, field: 'storageUsageVideos' },
    { prefix: `zips/${classId}/`, field: 'storageUsageZips' },
    { prefix: `audio/${classId}/`, field: 'storageUsageAudio' },
    { prefix: `recordings/${classId}/`, field: 'storageUsageRecordings' },
    { prefix: `subtitles/${classId}/`, field: 'storageUsageRecordings' },
    { prefix: `irregularities/${classId}/`, field: 'storageUsageIrregularities' },
    { prefix: `classes/${classId}/tasks/`, field: 'storageUsageTasks' },
    { prefix: `tasks/${classId}/`, field: 'storageUsageTasks' },
    { prefix: `submissions/${classId}/`, field: 'storageUsageTasks' },
    { prefix: `reports/${classId}/`, field: 'storageUsageReports' },
  ];

  const results = {
    storageUsageScreenShots: 0, storageUsageVideos: 0, storageUsageZips: 0,
    storageUsageAudio: 0, storageUsageRecordings: 0, storageUsageIrregularities: 0,
    storageUsageTasks: 0, storageUsageReports: 0, storageUsage: 0,
  };

  for (const cat of categories) {
    const [files] = await bucket.getFiles({ prefix: cat.prefix });
    let catTotal = 0;
    for (const file of files) {
      const sz = parseInt(file.metadata?.size, 10);
      if (!isNaN(sz) && sz > 0) catTotal += sz;
    }
    results[cat.field] = (results[cat.field] || 0) + catTotal;
    results.storageUsage += catTotal;
  }

  await db.doc(`classes/${classId}/metadata/storage`).set(
    { ...results, lastAuditedAt: FieldValue.serverTimestamp() },
    { merge: true }
  );
  await db.doc(`classes/${classId}`).set({ storageUsage: results.storageUsage }, { merge: true });
  return results;
}

Production Code 3: The 4-Stage Zero-Orphan Cascading Class Purge (`onClassDocDeleted` in `cleanupTriggers.js`)

In Firestore, deleting a parent document (classes/{classId}) does not automatically delete its subcollections, top-level job documents referencing classId, or physical files in Cloud Storage.

To guarantee 100% Zero-Orphan Cleanup when a teacher deletes a class at the end of a semester, onClassDocDeleted in functions/storage_triggers/cleanupTriggers.js executes a comprehensive 4-Stage Cascading Purge:

Here is the production implementation from functions/storage_triggers/cleanupTriggers.js, using 500-document chunked batch commits (deleteCollectionByClassId and deleteSubcollectionByPath) to respect Firestore's 500-operation batch ceiling:

// Source: functions/storage_triggers/cleanupTriggers.js
export const onClassDocDeleted = onDocumentDeleted({
  document: 'classes/{classId}',
  region: FUNCTION_REGION,
  timeoutSeconds: 540,
  memory: '512MiB',
}, async (event) => {
  const classId = event.params.classId;
  const bucket = storage.bucket();

  // Stage 1: Purge all 13 Cloud Storage prefixes (including projectMeetings & taskDemos)
  const prefixes = [
    `screenshots/${classId}/`, `videos/${classId}/`, `zips/${classId}/`,
    `audio/${classId}/`, `recordings/${classId}/`, `subtitles/${classId}/`,
    `irregularities/${classId}/`, `classes/${classId}/`, `tasks/${classId}/`,
    `submissions/${classId}/`, `reports/${classId}/`, `taskDemos/${classId}/`,
    `projectMeetings/${classId}/`
  ];
  for (const prefix of prefixes) {
    await bucket.deleteFiles({ prefix, force: true }).catch(() => {});
  }

  // Stage 2: Purge top-level collections matching classId in 500-doc batches
  const collectionsToClean = [
    'screenshots', 'audio', 'videoJobs', 'zipJobs', 'videoAnalysisJobs',
    'irregularities', 'progress', 'propertyUploadJobs',
    'lectureMergeJobs', 'lectureSubtitleJobs', 'aiJobs'
  ];
  for (const col of collectionsToClean) {
    await deleteCollectionByClassId(col, classId);
  }

  // Stage 3: Purge all 20+ nested subcollections in 500-doc batches
  const subcollectionsToPurge = [
    'projectMeetings', 'projectGroups', 'projectStudentProfiles',
    'activeMeetingCaptures', 'lectureRecordings', 'bingoSessions',
    'status', 'lessons', 'bingoRecords', 'attendanceAdjustments',
    'studentProperties', 'classProperties', 'passkeyBypassRequests',
    'screenBroadcastViewers', 'messages', 'audio_audits', 'irregularities', 'questionBank'
  ];
  for (const subcol of subcollectionsToPurge) {
    await deleteSubcollectionByPath(`classes/${classId}/${subcol}`);
  }

  // Stage 4: Unlink classId from teacherProfiles & studentProfiles
  await removeClassFromProfiles('teacherProfiles', classId);
  await removeClassFromProfiles('studentProfiles', classId);
});

4. Diagram 3: 1-Command Zero-Click Terraform Onboarding & Multi-Layer Automated Verification Pyramid

To empower any university, school, or Google Developer Group (GDG) to deploy their own sovereign instance of the Google AI Classroom without clicking through dozens of Cloud Console wizards, setup-new-project.sh, terraform/, and deploy.sh automate the entire provisioning and verification lifecycle:

How `./setup-new-project.sh` Eliminates Manual Cloud Console Clicks

Running ./setup-new-project.sh my-school-ai-2026 01ABCD-23EFGH-45IJKL executes an end-to-end automated pipeline:

Project Creation & Billing Link: Creates the GCP project (if not already present) and attaches the billing account.
Terraform State & 27 APIs: Runs terraform init && terraform apply inside terraform/, enabling Compute, Cloud Run, Cloud Functions Gen 2, Cloud Build, Artifact Registry, Eventarc, Cloud Tasks, Cloud Scheduler, Secret Manager, Google Cloud AI / Firebase AI Logic (aiplatform.googleapis.com & firebasevertexai.googleapis.com), Cloud Billing Catalog, Firestore Native, Firebase Storage, and Identity Platform (GCIP).
Cross-Codebase Config Synchronization (sync-config.sh): Automatically extracts the newly generated Firebase Web App SDK credentials and writes them into web-app/.env and functions/config.js, then runs ./sync-config.sh to propagate identical region, CORS, and model constants across all 7 isolated Cloud Functions codebases (ai_flows, media_processing, lecture_processing, scheduled_tasks, storage_triggers, ClassFunctions, and user_management).
Zero-Cold-Start 24/7 Demo Sandbox Seeding (seed-demo-data.mjs): Pre-populates the IT114115-Demo class with realistic student telemetry, pre-compiled H.264 lab videos, bilingual .vtt lecture recordings, and all 13 Default AI System Prompts so teachers can test every Gemini 4 Argon studio within 60 seconds of deployment!

5. Summary of the 10-Part Google AI Classroom Series

Across this 10-part architectural deep dive, we have shared every layer of the Google AI Classroom platform for teaching, learning, student performance evaluation, and proctored assessments:

Part	Architectural Focus	Key Google AI & Cloud Technologies
Google AI Classroom Part 1 (GitHub)	Solving the $25-Per-Exam Problem: Serverless Architecture	Gemini 4 Argon, Gemma 4 E2B, Firebase Gen 2
Google AI Classroom Part 2 (GitHub)	Zero-Egress Browser Edge AI Investigator (Gemma 4 E2B, MediaPipe, Whisper)	WebGPU Gemma 4 E2B, MediaPipe 3D Mesh, LiteRT Whisper
Google AI Classroom Part 3 (GitHub)	Frontier Multimodal Orchestration with Gemini 4 & Genkit	`gemini-4-argon`, `3.8-flash`, `3.5-lite`, Google Genkit
Google AI Classroom Part 4 (GitHub)	WebAuthn Passkeys, Rotating HMAC QRs & GCIP Blocking	FIDO2 WebAuthn, GCIP Blocking Hooks, 15s HMAC QR
Google AI Classroom Part 5 (GitHub)	Real-Time Multilingual Live Subtitles & Code-Switching	Gemini 4/3.8, LiteRT Gemma 4 E2B, Audio Worklet
Google AI Classroom Part 6 (GitHub)	Low-Bandwidth Classroom Screen Broadcast & Lecture Recording Studio	Firestore WebP Delta Sync, Gemini 3.5 Transcribe + Gemini 4
Google AI Classroom Part 7 (GitHub)	Map-Reduce-Map Video Intelligence & Practical Task Grading	Cloud Run FFmpeg, Cloud Tasks, Gemini 4 Rubric Engine
Google AI Classroom Part 8 (GitHub)	Streamlined Mobile 720p Capstone Supervision & Log Auditing	Single-Device Mobile Studio, 2-Stage Diarization + Gemini 4
Google AI Classroom Part 9 (GitHub)	Gamified Bingo Active Presence & Zero-Trust Proctored Exam Mode	Two-Strike Cloud Tasks, `.docx` Dossiers, Storage Rules
Google AI Classroom Part 10 (GitHub)	Green AI FinOps, Storage Lifecycles & 1-Command Terraform	Cloud Billing Catalog API, Cascading Purge, Terraform IaC

By combining Browser WebGPU Edge AI Investigation (Gemma 4 E2B, MediaPipe 3D Face Mesh, LiteRT Whisper) with Frontier Cloud Multimodal Reasoning (Gemini 4 Argon, Gemini 3.8 Flash, Gemini 3.5 Flash-Lite), we achieved:

Real-Time Proctored Exam Investigation Across 40+ Seats (catching fleeting 3-second visual and spoken exam irregularities at 60 FPS with synchronized screen, webcam, and audio evidence),
Universal Teacher Agency for Teaching & Student Performance to choose Gemini 4 (gemini-4-argon) across every live subtitle, screen broadcast, bingo, practical video grading, and capstone supervision studio, and
Scale-to-Zero FinOps & Automated Storage Lifecycles (eliminating idle server billing and orphaned storage via on-device edge inference, atomic AI quota circuit breakers, and cascading class purges).

Explore the Open-Source Repository

Google AI Classroom Part 9: Gamified "Bingo" Active Presence Verification & Zero-Trust Proctored Exam Mode on Google Cloud

Cyrus Wong — Wed, 11 Nov 2026 07:49:09 +0000

Author: Cyrus Wong (黃俊彥) — Google Developer Expert - Cloud (AI), Senior Lecturer at HKIIT

1. Problem Statement: Defeating "Fake Video Playback", Proxy Classmates & Static Screen Decoys

In a 3-hour coding lab or a 100-student hybrid lecture, passive screen-share telemetry and traditional door-scan attendance have three classic blind spots:

The Static Screen Decoy (AFK): A student opens VS Code or a lecture slide deck, starts screen sharing, and walks away from their workstation for 45 minutes while their screen capture continues uploading identical frames.
The "Give Password to a Classmate + Play a Pre-Recorded Video" Proxy Lab Exploit: Even worse, an absent student gives their account password to a friend sitting in the computer lab. That friend logs into the absent student's account on an adjacent lab PC, starts screen sharing, and plays a pre-recorded coding video in full-screen so the screen-capture feed looks like someone is typing code! When a standard web-based Bingo quiz pops up on that second monitor, the classmate tries to click an answer on their friend's behalf.
The "Tap Student ID & Run" Lecture Evasion Exploit (Why Attendance Must Be Verifiable Anytime): In modern higher education institutions , lecture halls often provide RFID card readers or kiosks at the entrance where students tap their physical student badge or enter an attendance code right at 9:00 AM. Once tapped, the campus system logs them as "Present"—whereupon disengaged students immediately leave the lecture hall to go to the cafeteria, hang out, or leave campus! A single attendance tap at the beginning of a 3-hour session is fundamentally vulnerable. Lecturers need the ability to take attendance anytime during class (mid-lecture, unannounced, or near the end).

Conversely, during a formal proctored test or examination, full-screen monitor capture, webcam/microphone investigation, and tamper-evident evidence retention must be strictly enforced.

Solving both modes effectively requires addressing three operational challenges:

Formative Teaching vs. High-Stakes Proctored Exams: During ordinary lectures and labs, instructors want lightweight, interactive comprehension checks that keep students engaged in learning; during formal tests and exams, invigilators require strict full-monitor capture (displaySurface === 'monitor'), continuous webcam/audio investigation, and write-once Cloud Storage locks.
Distinguishing Honest Learning Mistakes from Absence: If an instructor sends a pop quiz during a lecture and docks attendance marks when a student selects the wrong answer, students are penalized for struggling with new technical material even though they were sitting attentively at their desks!
Eliminating Idle Polling Bills: Running a Cloud Scheduler cron job every 60 seconds to scan pending attendance timers across 50 classes burns tens of thousands of idle Firestore reads per month.

In Part 9 of our Google Developer Expert series, we detail how we engineered two complementary classroom integrity systems:

Gamified "Classroom Bingo" Active Presence Verification (BingoModal.jsx, BingoQuestionBankModal.jsx, BingoResultsView.jsx, LectureQrBingoModal.jsx, and bingoFlows.js) featuring 4 Sourced Verification Types (including 2 QR Code Attendance Modes to take attendance anytime), a Gemini 4 Argon (gemini-4-argon) Question Bank AI Studio, Fisher-Yates positional bias elimination, and an event-driven Non-Punitive Two-Strike State Machine powered by Google Cloud Tasks (dispatchBingoRetryTask).
Zero-Trust Exam Mode Lockdown & Court-Ready Forensic Dossiers (ExamReadinessWizard.jsx, storage.rules, getStudentVideoPlaybackUrl.js, and processReportJob.js), enforcing full-monitor capture (displaySurface === 'monitor'), write-once Cloud Storage metadata locks (isExam == 'true'), Student Portal confidentiality shielding during exam windows, and 1-click export of formal Microsoft Word (.docx) & RFC 4180 CSV Incident Dossiers.

2. Diagram 1: Four Classroom Bingo Verification Modes & The Gemini 4 Question Bank Studio

When a lecturer triggers a Classroom Bingo round (either manually with one click, via a Staggered Jitter Queue that challenges random subsets of students every few minutes so neighbours cannot shout out "Option B!", or via Auto-Bingo during live screen broadcasts), resolveBingoQuestion in functions/ai_flows/bingoFlows.js supports four distinct verification types:

Type 1 — Predefined Question Bank (question_bank): ($0.00 / Zero AI Tokens) Instant, zero-cost comprehension checks pulled from an instructor-curated question bank (bulk-imported from Moodle Aiken format or batch-generated via Gemini 4 Argon).
Type 2 — Multimodal Teacher Broadcast Screen + Subtitle AI Fusion (teacher_screen): A single multimodal Gemini call (~$0.00015 USD total cohort cost!) that fuses the teacher's live broadcast frame (liveFrame) with the last 5 minutes of spoken captions (liveSubtitles) and an anti-duplication rule.
Type 3 — Personalized Student Lab Screen Inspection (student_screen): Targeted Gemini vision evaluation of an individual student's workstation monitor to verify active lab coding work and defeat static window decoys or pre-recorded video playback.
Type 4 — WebAuthn Biometric QR Attendance (2 QR Modes to Take Attendance Anytime): Designed specifically to defeat the "Tap Student ID Card & Run" lecture evasion exploit. Instead of relying on a one-time badge tap at the entrance door, instructors can trigger attendance verification at any moment during the lesson via two complementary QR modes:
- Mode 4A: Workstation Mobile Passkey QR (mobile_passkey): Displayed on the student's lab PC monitor. The student must scan it with their paired smartphone and authenticate via on-device Face ID / Fingerprint. A proxy classmate sitting at an adjacent PC cannot click or verify on their behalf because WebAuthn enforces hardware device binding (1-Phone = 1-Student).
- Mode 4B: Lecture Hall Dynamic Rotating Projector QR (lecture_passkey_qr): Displayed on the central classroom/lecture hall projector screen. The QR code dynamically rotates every 10 seconds with HMAC time-signatures. Students physically sitting in the hall point their smartphone camera at the projector screen to sign in. Students who tapped their card at the door and left cannot scan the projector, and remote friends cannot use a forwarded photo because the QR code expires and rotates in 10 seconds!

Engineering Highlights of the Bingo Question Engine

Universal Model Agency in the Question Bank Studio (BingoQuestionBankModal.jsx): Before class, teachers can open BingoQuestionBankModal.jsx, type a lesson topic (or customize the system prompt), and select 🧠 Gemini 4 Argon (gemini-4-argon), ⚡ Gemini 3.8 Flash (gemini-3.8-flash), or 🪶 Gemini 3.5 Flash-Lite (gemini-3.5-flash-lite) to generate 1 to 15 multiple-choice questions at once—or bulk-import existing quizzes in standard Moodle Aiken text format.
Eliminating LLM Positional Bias (shuffleOptionsAndCorrectIndex): When generating 4-option multiple-choice questions, LLMs exhibit a strong statistical bias toward placing the correct answer at index 0 (Option A) or index 1 (Option B). Every question generated by generateBingoQuestionBank or resolveBingoQuestion passes through a deterministic Fisher-Yates permutation (shuffleOptionsAndCorrectIndex) that randomizes option order and updates correctIndex accordingly.
Multimodal Teacher Screen + 5-Minute Live Subtitle Fusion (Mode 2): In teacher_screen mode, resolveBingoQuestion makes a single multimodal Gemini call for the entire 50-student class (~$0.00015 USD total cohort cost!). Even better, it fuses the teacher's live broadcast frame (classes/{classId}/screenBroadcast/liveFrame) with the last 5 minutes of the teacher's spoken commentary (classes/{classId}/liveSubtitles/current) and injects an Anti-Duplication Rule containing the last 5 asked questions (isDuplicateQuestion with $80\%$ lexical Jaccard similarity rejection) so Gemini never repeats the same question while the teacher stays on a single slide!
Zero-Leak Client Payload Sanitization: A savvy computer science student will immediately open Chrome DevTools (Network / Firestore tab) when a quiz pops up. In bingoFlows.js, correctIndex is written exclusively to classes/{classId}/bingoRecords/{bingoId} (which students are blocked from reading in firestore.rules). The payload pushed to studentProperties/{studentUid}.activeBingo contains only question, options, and expiresAtMillis—making client-side answer inspection impossible.

Production Code 1: Genkit Structured Schema Generation & Fisher-Yates Positional De-Biasing (`bingoFlows.js`)

Let's inspect generateBingoQuestionBank and shuffleOptionsAndCorrectIndex in functions/ai_flows/bingoFlows.js:

// Source: functions/ai_flows/bingoFlows.js
export async function generateBingoQuestionBank({ topic, count = 5, customPrompt = null, classId = null, model = null }) {
  const targetModel = model || AI_MODEL;
  const safeCount = Math.min(Math.max(Number(count) || 5, 1), 15);
  const prompt = customPrompt
    ? customPrompt.replace(/\{\{topic\}\}/g, topic || '').replace(/\{\{count\}\}/g, String(safeCount))
    : `You are a computer science instructor preparing quick-check multiple choice questions for a lab/lecture.
Topic / Lesson Material: "${topic}".
Generate exactly ${safeCount} multiple-choice questions testing immediate comprehension.
Each question MUST have exactly 4 options and a correctIndex (0, 1, 2, or 3).`;

  const { response, modelUsed } = await generateWithResilience({
    prompt,
    output: {
      schema: z.object({
        questions: z.array(
          z.object({
            question: z.string().describe('The question prompt'),
            options: z.array(z.string()).length(4).describe('4 multiple choice options'),
            correctIndex: z.number().min(0).max(3).describe('Index of the correct option (0-3)'),
          })
        ),
      }),
    },
  }, targetModel);

  const shuffledQuestions = response.output.questions.map((q) => {
    const { options, correctIndex } = shuffleOptionsAndCorrectIndex(q.options, q.correctIndex);
    return { ...q, options, correctIndex };
  });

  return { questions: shuffledQuestions };
}

export function shuffleOptionsAndCorrectIndex(options, correctIndex) {
  const safeCorrectIndex = (typeof correctIndex === 'number' && correctIndex >= 0 && correctIndex < options.length)
    ? correctIndex
    : 0;
  const items = options.map((opt, idx) => ({ text: opt, isCorrect: idx === safeCorrectIndex }));

  // Unbiased Fisher-Yates shuffle eliminates LLM Option-A bias
  for (let i = items.length - 1; i > 0; i--) {
    const j = Math.floor(Math.random() * (i + 1));
    [items[i], items[j]] = [items[j], items[i]];
  }

  return {
    options: items.map((item) => item.text),
    correctIndex: items.findIndex((item) => item.isCorrect),
  };
}

The 3 Prompts Powering Gamified Classroom Bingo (`bingoFlows.js`)

Here are the exact prompts executed across the three AI-powered Bingo modes in functions/ai_flows/bingoFlows.js and admin/prompts/images/:

1. Question Bank Studio Prompt (`Bingo Question Bank Generator.md` / `generateBingoQuestionBank`)

Used when a teacher opens BingoQuestionBankModal.jsx and generates a reusable bank of 1–15 multiple-choice questions with gemini-4-argon, gemini-3.8-flash, or gemini-3.5-flash-lite:

# Bingo Question Bank Generator

You are an instructor preparing quick-check multiple choice questions for a classroom lab or lecture.

Topic / Lesson Material: "{{topic}}".

## Rules & Requirements
1. Generate exactly {{count}} multiple-choice questions testing immediate comprehension.
2. Each question MUST have exactly 4 options and a `correctIndex` (0, 1, 2, or 3).
3. Keep questions concise, practical, and unambiguous.
4. Directly test core principles, code patterns, or command usage from the stated topic.

2. Mode 2 Prompt — 1-to-Many Teacher Broadcast Screen + 5-Minute Live Subtitle Fusion + Anti-Duplication Rule (`resolveBingoQuestion`)

When the lecturer triggers a teacher_screen Bingo check during a live broadcast, resolveBingoQuestion combines the current classes/{classId}/screenBroadcast/liveFrame image, the last 5 minutes of classes/{classId}/liveSubtitles/current spoken commentary, and the last 5 asked questions from classes/{classId}/bingoRecords into a single multimodal call for the entire cohort:

You are an invigilator verifying student attention during a live lecture/explanation.
Analyze the instructor's shared screen image alongside their recent spoken commentary/captions:
Recent Instructor Spoken Commentary: "{{captionContext}}"

Formulate a 4-option multiple-choice question testing whether a student was actively following the instructor's lecture explanation.
The question can reference what the instructor just explained verbally, key details visible on the shared screen, or the connection between them.
ONE option MUST be the true detail from the screen or verbal explanation.
THREE options MUST be plausible but incorrect distractors.
Respond with JSON matching the schema:
{
  "question": "Which file was the instructor editing?",
  "options": ["main.py", "test.js", "app.py", "utils.go"],
  "correctIndex": 0,
  "observedEvidence": "main.py is open in VS Code with Python code displayed"
}

CRITICAL ANTI-DUPLICATION RULE:
The instructor may be holding or explaining the same slide/window. The following questions were ALREADY recently asked to students:
1. "{{recentQuestion1}}"
2. "{{recentQuestion2}}"
3. "{{recentQuestion3}}"

You MUST NOT repeat or rephrase any of the above questions.
You MUST choose a completely DIFFERENT detail, spoken concept, code line, toolbar item, question angle, or UI element that was NOT covered above.

3. Mode 3 Prompt — Personalized Student Lab Screen Presence Verification (`Bingo Student Screen Presence Question Generator.md`)

Used during hands-on computer labs when each student is working at their own pace on their own workstation:

# Bingo Student Screen Presence Question Generator

You are a classroom invigilator checking student presence in a computer lab.
Analyze this student's computer screen screenshot.

## Rules & Requirements
1. Formulate a 4-option multiple-choice question testing immediate awareness of their screen state (e.g., active editor file, command in terminal, running app, or lab guide step).
2. ONE option MUST be the true visible detail on the student's screen.
3. THREE options MUST be plausible distractors.
4. Respond with valid JSON matching the schema:
{
  "question": "What task or window is currently active on your screen?",
  "options": ["Running test suite in terminal", "Reading documentation in browser", "Editing config file", "Debugging breakpoint"],
  "correctIndex": 0,
  "observedEvidence": "Terminal running pytest is in the foreground"
}

3. Diagram 2: The Non-Punitive Two-Strike State Machine with Google Cloud Tasks & 3-Color Attendance Heatmap

What happens after the high-priority BingoModal.jsx (z-index: 2147483647, accompanied by an optional Web Audio API chime) appears on the student's screen with a 45-second countdown timer?

Our Two-Strike State Machine (submitBingoResponse, enqueueBingoRetryTask, and handleDispatchBingoRetry in functions/ai_flows/bingoFlows.js) strictly decouples academic comprehension from physical desk presence:

Case 1 — Correct Answer (passed): The student earns base points (100 pts), a speed bonus (up to +50 pts scaled by remaining time ratio), and a top-3 podium rank bonus (+50 / +30 / +20 pts). Any prior strike is cleared (strikeNumber = 0).
Case 2 — Wrong Answer (failed_incorrect): Because the student actively clicked an option within the 45-second window, physical presence at the workstation is 100% verified! The student receives instant visual feedback on the correct answer, zero strikes are triggered, and zero attendance minutes are deducted.
Case 3 — Strike 1 Timeout (missed_timeout on strikeNumber === 1): What if the student stepped away to the washroom for 2 minutes? Instead of docking attendance on a single missed check, submitBingoResponse sets pendingRetryBingo: true and enqueues a delayed task in Google Cloud Tasks (locations/asia-east2/functions/dispatchBingoRetryTask) scheduled for retryDelayMinutes later (configurable 1–15 minutes, default 3 minutes, $0.00 idle polling cost).
Case 4 — Strike 2 Consecutive Timeout (missed_timeout on strikeNumber >= 2): When the Cloud Task wakes up after 3 minutes, it performs pre-flight idempotency checks (verifying the class is still active and the teacher hasn't cancelled the session) and dispatches a personal Strike 2 Follow-Up Challenge. If the student responds to Strike 2 (even with a wrong answer!), presence is recovered. Only if the student misses both Strike 1 and Strike 2 consecutively does the backend calculate the elapsed unverified minutes ($\Delta t = t_{\text{Strike 2}} - t_{\text{Strike 1}}$) and write an attendanceAdjustments deduction record!

Production Code 2: Event-Driven Two-Strike Cloud Tasks Scheduling & Elapsed AFK Deduction (`bingoFlows.js`)

Here is the exact Two-Strike state transition and deterministic Cloud Task deduplication logic inside functions/ai_flows/bingoFlows.js:

// Source: functions/ai_flows/bingoFlows.js
  // 3. Handle Two-Strike Attendance Protocol & Irregularities
  if (result === 'passed') {
    updatePayload.strikeNumber = 0;
    updatePayload.pendingRetryBingo = false;
  } else if (result === 'failed_incorrect') {
    // Student was present at desk, answered wrong. Do NOT dock attendance.
    updatePayload.lastInattentiveAt = FieldValue.serverTimestamp();
  } else if (result === 'missed_timeout') {
    const currentStrike = Number(record.strikeNumber) || 1;

    if (currentStrike === 1) {
      // Strike 1: Schedule follow-up retry in configured delayMinutes (default: 3 mins)
      updatePayload.pendingRetryBingo = true;
      updatePayload.retryBingoScheduledAtMillis = Date.now() + (retryDelaySeconds * 1000);
      updatePayload.priorMissedBingoId = bingoId;
      updatePayload.retryDelayMinutes = retryDelayMinutes;

      // Enqueue to Google Cloud Tasks for serverless event-driven execution ($0 idle polling)
      await enqueueBingoRetryTask({
        classId,
        studentUid,
        priorBingoId: bingoId,
        delaySeconds: retryDelaySeconds,
      });
    } else if (currentStrike >= 2) {
      // Strike 2 Confirmed Absence! Calculate elapsed unverified window (t_Strike2 - t_Strike1)
      updatePayload.pendingRetryBingo = false;
      const priorBingoId = record.priorBingoId;
      let startMillis = record.issuedAtMillis - (retryDelaySeconds * 1000);

      if (priorBingoId) {
        const priorDoc = await db.doc(`classes/${classId}/bingoRecords/${priorBingoId}`).get();
        if (priorDoc.exists && priorDoc.data()?.issuedAtMillis) {
          startMillis = priorDoc.data().issuedAtMillis;
        }
      }

      const endMillis = Date.now();
      const deductedMinutes = Math.max(1, Math.round((endMillis - startMillis) / 60000));

      // Record attendance adjustment (renders as Red Bitmask Code 2 in AttendanceView.jsx)
      await db.collection(`classes/${classId}/attendanceAdjustments`).add({
        classId,
        studentUid,
        startTime: new Date(startMillis),
        endTime: new Date(endMillis),
        startMillis,
        endMillis,
        deductedMinutes,
        voidedMinutesCount: deductedMinutes,
        reason: `Missed 2 consecutive Bingo checks (AFK/Decoy)`,
        strike1BingoId: priorBingoId || 'strike_1',
        strike2BingoId: bingoId,
        createdAt: FieldValue.serverTimestamp(),
      });
    }
  }

In AttendanceView.jsx, every minute of a lesson is encoded as a 5-State Minute Bitmask (0–4):

1 / 3 / 4 (🟢 Green): Verified Active Presence (Screen Shared / Passkey Verified / Teacher Override).
2 (🔴 Red): Voided AFK Window (attendanceAdjustments from 2 consecutive missed Bingo checks, showing exact tooltip timestamps and a 1-Click Teacher Excuse / Restore (excused: true) button).
0 (⚪ Grey): Offline / Disconnected.

4. Diagram 3: Zero-Trust Exam Mode Lockdown & Court-Ready `.docx` Forensic Dossiers

While Classroom Bingo keeps daily lectures and labs lively and non-punitive, formal practical examinations require Zero-Trust Assessment Security. When a class enters a scheduled examPeriods window (or lessonType === 'exam'), four defense-in-depth layers activate automatically across the client browser, Cloud Run FFmpeg, Cloud Storage Security Rules, and our Signed URL Broker:

5. Deep Dive into the Zero-Trust Exam Code: Storage Metadata Locks, Signed URL Shielding & `.docx` Dossiers

Production Code 3: Write-Once Exam Metadata Locks in Cloud Storage (`storage.rules`) & Signed URL Broker (`getStudentVideoPlaybackUrl.js`)

During a practical exam, students must not be able to delete uploaded screenshots/videos or replay their own exam recording afterward to leak exam paper questions to another class section.

We enforce this at two independent boundaries:

Direct Cloud Storage Rules (storage.rules): Students have write-only access to screenshots/ and irregularities/ (under 2 MB / 5 MB size limits), zero delete permissions, and videos/{classId}/{videoId} explicitly denies student reads whenever resource.metadata.isExam == 'true':

// Source: storage.rules
    match /videos/{classId}/{videoId} {
      // Allow authenticated teachers to read any video in a class.
      // Students can only read non-exam videos that belong to them.
      allow read: if request.auth != null && (
        request.auth.token.role == 'teacher' ||
        (
          resource.metadata != null &&
          'studentUid' in resource.metadata &&
          resource.metadata.studentUid == request.auth.uid &&
          (!('isExam' in resource.metadata) || resource.metadata.isExam != 'true')
        )
      );
      allow delete: if request.auth != null && request.auth.token.role == 'teacher';
    }

Server-Side Signed URL Broker (getStudentVideoPlaybackUrl.js): Even if a video was compiled before an instructor retroactively added an examPeriods range to the class settings, evaluateStudentRecordingsAccess checks both the jobData.isExam flag and isTimestampInExamPeriods(jobData.startTime, examPeriods, timeZone):

// Source: functions/media_processing/getStudentVideoPlaybackUrl.js
export function evaluateStudentRecordingsAccess({
  examPeriods = [],
  policy = 'always_enabled',
  releaseDate = null,
  jobData = {},
  timeZone = 'Asia/Hong_Kong',
}) {
  // Recordings for exam/test periods or stamped as exam are strictly confidential and not shared
  const isExam = Boolean(
    jobData.isExam ||
    jobData.lessonType === 'exam' ||
    isTimestampInExamPeriods(jobData.startTime, examPeriods, timeZone) ||
    isTimestampInExamPeriods(jobData.createdAt, examPeriods, timeZone)
  );

  if (isExam) {
    return {
      allowed: false,
      reason: 'Screen recordings for exam or test periods are confidential and not shared with students to protect assessment questions.',
    };
  }
  // Also enforces 'disabled' and 'delayed_release' policies
  return { allowed: true };
}

Production Code 4: Court-Ready Microsoft Word (`.docx`) & RFC 4180 CSV Forensic Dossier Compiler (`processReportJob.js`)

When an academic integrity board reviews a suspected exam irregularity, raw JSON logs in a database are difficult for non-technical committee members to inspect.

When an instructor clicks 📄 Export Formal Dossier (.docx) in IncidentDossierExportModal.jsx (accessed via IrregularitiesView.jsx), processReportJob.js (1 vCPU, 2 GiB RAM, 540s timeout) queries all time-scoped irregularities, audio transcripts (from Edge Whisper + Gemini verification), and screenshots for the class, classifies each student's risk level (HIGH RISK, MODERATE RISK, NORMAL / CLEAN), and uses the docx and csv-stringify libraries to compile a downloadable Microsoft Word (.docx) Incident Dossier complete with embedded ImageRun evidence snapshots, 3D head-pose/gaze telemetry tables, and verbatim speech transcripts:

// Source: functions/media_processing/processReportJob.js
export const processReportJob = onDocumentCreated(
  {
    document: 'reportJobs/{jobId}',
    region: FUNCTION_REGION,
    cpu: 1,
    memory: '2GiB',
    timeoutSeconds: 540,
  },
  async (event) => {
    // ... Fetches classData, studentProperties, time-scoped irregularities, audioTranscripts & screenshots
    const studentReports = targetStudentIds.map((uid) => {
      const student = studentMap[uid] || { id: uid };
      const studentIrregs = irregularities.filter((ir) => ir.studentUid === uid || ir.userId === uid);
      const studentAudios = audioTranscripts.filter((a) => a.studentUid === uid);
      const studentScreens = screenshots.filter((s) => s.studentUid === uid);

      const highSeverityCount = studentIrregs.filter((i) => i.severity === 'high' || i.type === 'multiple_faces').length;
      const totalIncidents = studentIrregs.length;
      const riskLevel = highSeverityCount > 2 ? 'HIGH RISK' : totalIncidents > 3 ? 'MODERATE RISK' : 'NORMAL / CLEAN';

      return { student, irregularities: studentIrregs, audios: studentAudios, screenshots: studentScreens, totalIncidents, highSeverityCount, riskLevel };
    });

    // Compiles structured Microsoft Word (.docx) with embedded ImageRun evidence & RFC 4180 CSV
    // Uploads to reports/{classId}/{jobId}/ and updates reportJobs/{jobId} with signed download URLs
  }
);

The 3 Zero-Trust Exam Mode Invigilation Prompts Feeding the `.docx` Dossier

During a proctored exam, whenever the Tier-1 Browser Edge Investigator or teacher-triggered cloud sweeps escalate a visual or acoustic anomaly, these three prompts in admin/prompts/ evaluate the evidence and invoke Genkit tools (recordIrregularity, recordAudioIrregularity, recordAudioAudit) to populate the .docx forensic dossier:

1. Screen Irregularity & Tool-Calling Invigilator (`admin/prompts/images/AI invigilator.md`)

**You are an AI invigilator. Your task is to analyze a student's screen, take appropriate action using tools, and then report the actions you took in a final text summary.**

## Guidelines
* **Minor Distraction:** Briefly looking away, non-exam apps visible but not in use.
* **Significant Distraction:** Actively using another app, phone use, talking to someone, unauthorized communication or AI tools during an exam.

## Action & Response Protocol
1. **Analyze the screen.**
2. **If there are no issues, your final answer MUST be the exact text:** "All students are focused and on track."
3. **If there is an issue:**
   * First, call all necessary tools in parallel. You **MUST** call `sendMessageToStudent` and `sendMessageToTeacher`. If the issue is a **Significant Distraction**, you **MUST ALSO** call `recordIrregularity`.
   * Then, as your final answer, you **MUST** provide a detailed summary of the issue and the actions you took.

## Output Guidelines
* When referring to a student in any output, you **MUST** use the format 'email (uid)'.

2. Cloud Fallback Face & Gaze Invigilator (`admin/prompts/images/Cloud Fallback Face & Gaze Invigilator.md`)

# Cloud Fallback Face & Gaze Invigilator

Analyze this classroom invigilation webcam photo of student {{studentEmail}} (UID: {{studentUid}}, class: {{classId}}).
Determine the student's face presence and gaze orientation.

## Detection Rules
1. Is there a human face present? If no face is visible, status is 'no_face'.
2. Are multiple faces present? If more than 1 person is in frame, status is 'multiple_faces'.
3. Is the student looking forward/centered at their computer screen? If their head or gaze is turned significantly away (left, right, looking at another device, looking away from exam), status is 'looking_away'.
4. If the student is sitting normally facing their screen/work, status is 'normal'.

## Output Format
Respond ONLY with valid JSON in this exact structure:
{
  "faceStatus": "normal" | "looking_away" | "no_face" | "multiple_faces",
  "confidence": 0.95,
  "reason": "Brief explanation"
}

3. Live Rolling Audio Collusion & Whisper Invigilator (`admin/prompts/audios/AI Voice Invigilator (Live Rolling Audio).md`)

# AI Voice Invigilator (Live Rolling Audio)

You are an expert AI Classroom Invigilator analyzing a live student audio stream during an examination.

## Context
- Class ID: {{classId}}
- Student UID: {{studentUid}}
- Student Email: {{studentEmail}}
- Audio Duration: Rolling 20–45s window

## Audio Transcript
"""
{{transcript}}
"""

## Instructions
1. Analyze the transcript for unauthorized speech, exam collusion, question recitation, answer trading, or whispering to external parties/devices.
2. If multi-speaker discussion, unauthorized talking, or answer recitation is detected:
   - Call the `recordAudioIrregularity` tool with `title`, `message`, `riskLevel` ('low' | 'medium' | 'high'), `speakerCount`, and `transcriptSnippet`.
3. Call the `recordAudioAudit` tool with `verdict` ('clean_exam' | 'suspicious_collaboration' | 'whisper_detected' | 'background_noise' | 'inconclusive'), `speakerCount`, `summary`, and `transcript`.

What's Next in Part 10?

We have now explored the entire pedagogical and multimodal AI stack—from Browser Edge AI (Gemma 4 E2B, MediaPipe, LiteRT Whisper) to Cloud Broadcasts, Gemini 4 Argon Video Grading, Capstone Mobile Supervision, and Zero-Trust Exams.

In Part 10 (our Series Finale), we look under the hood at Day-2 Institutional Operations: how we govern AI and storage budgets using Live Google Cloud Billing SKU Sync (syncGeminiPricing), 9-Category Event-Driven Storage Quota Enforcement (storageQuota.js), 4-Stage Zero-Orphan Cascading Class Purge (onClassDocDeleted), and 1-Command Terraform Infrastructure-as-Code (./setup-new-project.sh).

Google AI Classroom Part 8: AI Project Supervision — Streamlined Mobile 720p Demo, 2-Stage Diarization & Fake-Log Auditing with Gemini 4

Cyrus Wong — Wed, 11 Nov 2026 07:41:52 +0000

Author: Cyrus Wong (黃俊彥) — Google Developer Expert - Cloud (AI), Senior Lecturer at HKIIT

1. The Capstone Supervision Bottleneck: ChatGPT Logbooks, Free-Riders & Multi-Device Consultation Friction

Supervising 15 final-year engineering capstone project groups (60+ students) across a 16-week semester is one of the most rewarding yet administratively exhausting responsibilities in higher education. In our Software Engineering and Cloud Computing Higher Diploma programmes, three recurring pain points threatened both pedagogical quality and assessment fairness:

The "ChatGPT Logbook" Epidemic (suspected_ai_fake) & 1-Word Filler (vague_useless): Students are required to submit weekly progress logbooks via Google Forms before their consultation meeting. Some students type one-word filler ("done", "finish", "thinking"), while others paste LLM-generated corporate buzzword salads ("Implemented real-time microservice telemetry, RBAC authentication, and mock AI chatbot") right after three consecutive weeks of reporting zero work.
Why Capstone Supervision Is Never "Just Voice" — And Why We Replaced Student PC Screen Sharing with a Single-Device Mobile Studio: A real engineering capstone consultation is inherently multimodal, not just a voice conversation. Around a physical lab table, students show physical IoT microcontroller breadboards, hand-drawn ER diagrams on paper/whiteboards, mobile app prototypes, and live code/terminal outputs on their personal laptops. We no longer ask students to join a web session and share their PC screen—waiting for 4 students to authenticate and switch desktop screen shares wasted 3 to 4 minutes of a 15-minute consultation and could never capture hardware wiring or whiteboard architecture diagrams. Instead, the supervisor captures all three modalities from a single smartphone: 🎙️ Continuous Voice Consultation, 📸 Zero-Pause 0.1s Instant Photo Snaps (1280px JPEG), and 📹 Auto-Sliced 720p (1280×720) Mobile Camera Demo Videos.
Multilingual Speaker Attribution in Noisy Labs: When 4 project teammates and a supervisor debate system architecture around a single table—code-switching rapidly between conversational Cantonese, Mandarin, and English technical terms (React, Firestore, Docker, Cloud Run, PR)—a single-pass LLM prompt cannot reliably separate who actually answered the technical challenge questions and demonstrated working prototypes from who sat silently and free-rode on their teammates' work.

In Part 8 of our Google Developer Expert series, we dive deep into the architecture of the Capstone Project Supervision & Weekly Log AI Pipeline (ProjectMeetingsView.jsx and projectMeetingFlows.js), showing how we engineered:

A Self-Healing On-Demand Google Sheets Logbook Sync with Hamming Distance = 1 Student ID Typo Healing and holiday-aware academic week calculation.
A Pre-Meeting Fake-Log AI Auditor (prepareProjectMeetingLogQuestions) that cross-examines each student's current weekly log against their prior weeks' history and generates pointed live code-walkthrough challenge questions.
A Streamlined Single-Device Teacher Mobile Supervision Studio on the supervisor's smartphone: combining 🎙️ Continuous Voice Recording (+ 1-tap 🗣️ Speaking Now student voice anchors), 📸 Zero-Pause 0.1s Photo Snaps (1280px JPEG), and 📹 Auto-Sliced 720p (1280×720) Mobile Camera Demo Videos that completely eliminate student PC screen-capture friction.
A Two-Stage Multimodal Cross-Examination Pipeline (analyzeProjectMeeting): pairing Stage 1 Acoustic Speaker Diarization (gemini-3.5-transcribe-preview with automatic gemini-3.8-flash low-volume & long-clip truncation recovery) with Stage 2 Multimodal Verification (gemini-4-argon / gemini-3.8-flash) and 1-Click Secret Manager SMTP HTML Report Dispatch (batchEmailProjectMeetingReports).

2. Diagram 1: Self-Healing Google Sheets Sync & Pre-Meeting Fake-Log AI Auditor

Before the weekly consultation begins, the supervisor clicks 🔄 Sync Weekly Logs inside ProjectMeetingsView.jsx. Rather than running wasteful background cron jobs every 5 minutes, the platform performs an on-demand incremental delta sync against the class's linked Google Form spreadsheet (Form Responses 2), fetching only newly appended rows (startRow = lastSyncedRowCount + 1).

Why Hamming Distance = 1 Student ID Healing Saved Hours of Manual Fixes

On mobile Google Forms, students frequently mistype a single digit of their 9-digit Student ID (e.g., typing 250105349 instead of 250105348) or submit from a personal @gmail.com address. A strict string equality check drops those logs as orphaned records.

Our ingestion resolver compares any unmatched 9-digit numeric ID against the enrolled class roster (projectGroups). If exactly one enrolled student in the class has a 9-digit ID of identical length with Hamming distance $= 1$ (differing by exactly 1 character position), or if the normalized student name uniquely matches an enrolled group member, the sync engine automatically heals the typo, links the log to the student's group and academic week, and flags the healed record in the UI!

Production Code 1: Pre-Meeting Fake-Log Detection & Live Code-Walkthrough Challenge Generator (`handlePrepareProjectLogQuestions`)

Once the weekly logs are synced into classes/{classId}/projectMeetings/{meetingId}, the supervisor clicks 🔍 Audit Logs & Prepare Questions (or runs ⚡ Smart Batch AI Mode 1 across all groups for Week $W$). Let's look at handlePrepareProjectLogQuestions in functions/ai_flows/projectMeetingFlows.js, which feeds each student's Week $W$ log plus their multi-week historical log trajectory into Gemini 4 Argon (gemini-4-argon) or Gemini 3.8 Flash (gemini-3.8-flash):

// Source: functions/ai_flows/projectMeetingFlows.js
export async function handlePrepareProjectLogQuestions({
  classId,
  meetingId,
  groupId,
  groupName,
  projectTopic,
  weekNumber = 1,
  weeklyLogs = [],
  preferredModel,
  auth,
}) {
  const reasoningModel = preferredModel || DEFAULT_LECTURE_AI_MODEL || 'gemini-3.8-flash';

  const logsFormatted = weeklyLogs
    .map((l, idx) => {
      const priorSummary = Array.isArray(l.priorHistory) && l.priorHistory.length > 0
        ? l.priorHistory
            .map(
              (p) =>
                `  * Week ${p.weekNumber}: Doing=${p.isDoingSomething ? 'Yes' : 'No'} | WorkDone="${p.workDone || p.inactiveReason || ''}" | Plan="${p.nextPlan || ''}"`
            )
            .join('\n')
        : '  * No prior weekly logs submitted.';
      return `### Student ${idx + 1}: ${l.studentName || l.name || 'Student'} (ID: ${l.studentId || 'N/A'}, Email: ${l.email || 'N/A'})
- Week ${weekNumber} Submitted: ${l.submitted === false ? 'NO LOG SUBMITTED THIS WEEK' : 'Yes'}
- "I am doing something this week": ${l.isDoingSomething ? 'Yes' : 'No'}
- "Work done & findings since last recording": "${l.workDone || ''}"
- "Part of last plan not yet finished": "${l.unfinished || ''}"
- "Plan of work before next recording": "${l.nextPlan || ''}"
- Prior Weeks Log History (for Continuity & Sudden-Jump Fake Detection):
${priorSummary}`;
    })
    .join('\n\n');

  // ... (Constructs prompt instructing Gemini to classify each log into
  // "genuine" | "vague_useless" | "suspected_ai_fake" | "inactive"
  // and generate 2-3 live code-walkthrough challengeQuestions)

  const { response, modelUsed } = await generateWithResilience(
    {
      prompt: [{ text: prompt }],
      config: {
        temperature: 0.2,
        maxOutputTokens: 16384,
        responseMimeType: 'application/json',
      },
    },
    reasoningModel
  );
  // Persists preMeetingLogAudit[] onto classes/{classId}/projectMeetings/{meetingId}
}

Here is the exact Pre-Meeting Fake-Log Audit & Challenge Question Prompt executed inside handlePrepareProjectLogQuestions:

## Pre-Meeting Weekly Log Audit & Fake-Log Challenge Question Task
You are helping the project supervisor prepare for **{{groupName}} — Week {{weekNumber}}** supervision meeting (Project Topic: {{projectTopic}}).

In this course, **writing a high-quality weekly log that accurately reflects real engineering work is an explicit grading criterion**.
However, two common student problems occur in the weekly log sheet:
1. **Useless / Vague Logs (`vague_useless`)**: 1-to-4 word filler like "done", "finish", "thinking", "do website", "proposal", or copy-pasting previous weeks' logs without concrete technical findings or deliverables.
2. **Fake / AI-Generated Logs (`suspected_ai_fake`)**: Logs that appear written by ChatGPT/AI to sound impressive (buzzword-heavy corporate prose like "Implemented real-time dashboards, mock AI chatbot, and role-based access control" or sudden massive multi-module claims right after weeks of writing "done" or "No"), without specific file names, bugs encountered, or continuity with last week's plan.

## Students' Week {{weekNumber}} Logs & Prior History
{{logsFormatted}}

## Instructions
For EVERY student listed above:
1. Classify `logQualityStatus` as one of:
   - `"genuine"`: Specific, honest, technically grounded log that connects logically to prior plans.
   - `"vague_useless"`: Too brief, generic ("done", "thinking"), or copy-pasted to evaluate real work.
   - `"suspected_ai_fake"`: Overly polished AI buzzword dump, suspicious sudden leap from prior "done"/"No" logs, or claims too many unverified subsystems at once.
   - `"inactive"`: Answered "No" to doing work this week or did not submit a log.
2. Assign `logQualityScore` from `0` to `10` (e.g., 8-10 for genuine detailed logs; 3-5 for suspected AI/fake logs pending verification; 1-4 for vague/useless logs; 0-2 for inactive/missing).
3. Write a concise `logAuditSummary` (1-2 sentences explaining why the log received this score and what looks genuine, vague, or AI-faked).
4. Generate 2 to 3 sharp, specific `challengeQuestions` for the supervisor to ask the student live during the meeting:
   - If `suspected_ai_fake`: Ask live code-walkthrough & debugging challenge questions that a student who faked the log with AI cannot answer without demonstrating real code or hardware live (e.g., "Open the exact Flask route / database table on your laptop right now for our 720p demo recording and show how X handles Y", "What specific error did you hit when integrating X and how did you fix it?").
   - If `vague_useless`: Demand an immediate live demonstration (captured via 720p mobile demo video or photo snap) of concrete artifacts and measurable next-week deliverables.
   - If `genuine`: Ask constructive technical design & edge-case verification questions about their reported findings.

Respond strictly with valid JSON matching this structure:
{
  "preMeetingLogAudit": [
    {
      "studentId": "250105348",
      "email": "250105348@stu.vtc.edu.hk",
      "name": "Student Name",
      "logQualityStatus": "suspected_ai_fake",
      "logQualityScore": 4,
      "logAuditSummary": "Sudden leap from 'done' in Weeks 6-7 to claiming full Flask backend, real-time dashboards, mock AI chatbot, and RBAC in one week.",
      "challengeQuestions": [
        "Open your Flask route file for a live demonstration: walk us through how the Doctor/Patient role-based access control check is implemented in code.",
        "Which database tables back your real-time dashboard and history page, and can you trigger a live state update right now?",
        "What was the hardest bug you fixed while connecting the frontend to Flask this week?"
      ]
    }
  ]
}

When a student submits a suspected ChatGPT log claiming they built an entire Flask RBAC backend in one week after writing "done" in Weeks 6 and 7, Gemini flags the badge as 🟠 Suspected AI / Fake (4/10) right on the supervisor's smartphone card and prepares 3 targeted verification questions for live in-person demonstration:

"Open your Flask route file for a live demonstration: walk us through how the Doctor/Patient role-based access control decorator is implemented in code."

"Which database tables back your real-time dashboard, and can you trigger a live state update right now while we capture a 720p demo clip?"

"What was the hardest bug you hit while connecting the React frontend to Flask this week?"

3. Diagram 2: Streamlined Single-Device Teacher Mobile Supervision Studio (Voice + `0.1s` Photo Snap + Auto-Sliced `720p` Demo Video)

When the group sits down for their weekly consultation, the supervisor opens the meeting card on their smartphone or tablet.

To make live supervision effortless with one thumb, we eliminated cross-device student PC screen-capture signaling completely and consolidated all multimodal capture into a Streamlined Single-Device Teacher Mobile Supervision Studio in ProjectMeetingsView.jsx:

🎙️ Continuous Voice Consultation (audio/webm / audio/mp4): Protected by the browser Screen Wake Lock API (navigator.wakeLock.request('screen')) so the mobile browser never sleeps mid-consultation. As each student speaks or answers a challenge question, the supervisor taps their 🗣️ Speaking Now pill, stamping { studentUid, name, email, timeSeconds, timestamp, clipIndex } into voiceAnchors[].
📸 Zero-Pause Instant Photo Snap (0.1s, 1280px JPEG @ 0.85 quality): When a student sketches an ER diagram on paper, shows an IoT microcontroller circuit, or points to a stack trace on their laptop, tapping 📸 Snap Photo captures a crisp 1280px image and uploads it to projectMeetings/{classId}/{groupId}/{meetingId}/ in 0.1 seconds without pausing or interrupting the ongoing voice recording!
📹 Auto-Sliced 720p Mobile Camera Demo Video (1280×720 @ ~1.5 Mbps): When a student demonstrates their working software or hardware prototype on their laptop or phone, the supervisor simply taps 📹 Record Demo Video (720p) and points their smartphone camera at the student's demo:
- The studio automatically finalizes and uploads Voice Part N in the background.
- It activates the mobile rear/front camera at 720p (1280×720, ~1.5 Mbps bitrate)—sharp enough for Gemini 4 Argon to read laptop UI states and terminal output, yet compact enough (~11 MB/min) to upload in seconds over campus Wi-Fi.
- The instant the supervisor taps ⏹️ Stop Demo, the studio uploads the 720p demo clip (type: 'mobile_camera') and automatically resumes Voice Part N+1 so not a single second of verbal feedback is lost!

Atomic Stale Analysis Invalidation When Moving Clips Across Groups or Weeks

During back-to-back consultations with 10 project groups, a supervisor occasionally records Group 3's voice clip or demo video while Group 2's card is still open. In ProjectMeetingsView.jsx, we built a 1-Click Clip Reassignment Modal (➡️ Move Clip to Another Meeting / Week).

Whenever a voice clip, 720p demo video, or photo is moved from Meeting $A$ to Meeting $B$ (or deleted after an AI report was already generated), a single atomic Firestore writeBatch() moves the media metadata, shifts any associated voiceAnchors, and automatically purges stale AI report fields (executiveSummary, keyInsights, attendanceRecord, studentContributions, actionItems, rawDiarizedTranscript) on both meetings using deleteField() while setting needsReanalysis: true. This guarantees a supervisor can never accidentally email a stale AI report after modifying a meeting's evidence!

4. Diagram 3: Two-Stage Multimodal Diarization & Gemini 4 Argon Cross-Examination (`analyzeProjectMeeting`)

Once the consultation concludes, the supervisor selects 🧠 Gemini 4 Argon (gemini-4-argon) or ⚡ Gemini 3.8 Flash (gemini-3.8-flash) directly in the meeting toolbar (or triggers Smart Batch AI Mode 1 Queuing, which sequentially processes all ready meetings in Week $W$ and skips already-analyzed meetings unless needsReanalysis === true).

Why split analyzeProjectMeeting into two distinct AI stages instead of feeding raw audio, video, photos, and logs into a single prompt?

Stage 1 (gemini-3.5-transcribe-preview with gemini-3.8-flash Resilience Fallbacks) is dedicated 100% to verbatim multilingual acoustic & demo-video dialogue diarization across every voice clip (audioClips[]) and 720p mobile camera demo video (screenClips[]), shifting text timestamps cumulatively (offsetTranscriptTimestamps) across Voice Part 1, Demo 1, and Voice Part 2 to assemble a single chronological text transcript without ever merging or re-encoding the underlying media files.
Stage 2 (gemini-4-argon or gemini-3.8-flash) receives five multimodal evidence streams simultaneously in a single multi-part Gemini API call (prompt: [{ text: reasoningPrompt }, ...reasoningMediaParts]):
1. The Stage 1 Combined Diarized Transcript (Text),
2. The supervisor's 1-Tap voiceAnchors[] Ground Truth Speaker Stamps (Text),
3. The individual 720p Mobile Camera Demo Video segments (screenClips[]) passed as separate gs:// video/mp4 / video/webm media parts in reasoningMediaParts,
4. The individual Snapped Meeting Photos (photos[]) of hardware breadboards, whiteboard ERDs, and laptop code passed as separate gs:// image/jpeg media parts in reasoningMediaParts, and
5. The Students' Submitted Weekly Log Sheet Entries (weeklyLogs[]) + Prior Week's Assigned Action Items (priorWeekContext).

Architectural Clarification — Zero FFmpeg Video Stitching: The platform never merges or re-encodes the voice clips (audioClips[]), still photos (photos[]), and 720p video segments (screenClips[]) into a single combined video/audio file before analysis. Every voice part, 0.1s JPEG photo, and 720p demo clip remains an independent object in Cloud Storage (gs://...). Stage 1 transcribes each voice/video segment individually and stitches only their text transcripts using cumulative [MM:SS] timestamp offsets (offsetTranscriptTimestamps), while Stage 2 passes the individual photo and 720p video gs:// URIs as a native multi-part media array directly to Gemini!

5. Deep Dive into the Production Code: Long-Clip Truncation Recovery & Deterministic Roster Reconciliation

Production Code 2: Stage 1 Acoustic Diarization with Automatic Low-Volume & Long-Clip Truncation Recovery (`projectMeetingFlows.js`)

During real classroom testing with 25-minute capstone consultations, we discovered two edge cases with dedicated speech-to-text preview models (gemini-3.5-transcribe-preview):

Low-Volume / Quiet Clips: If students sit far from the supervisor's phone in a quiet room, dedicated STT filters occasionally classify soft speech as background noise and return an empty string.
Mid-Meeting Silence Pause Truncation (isLongClipTranscriptTruncated): If a group pauses speaking for 20 seconds at minute 08:00 while compiling code on a laptop during a 20-minute recording (>= 360s), gemini-3.5-transcribe-preview can prematurely emit an end-of-sequence token, truncating the second half of the consultation!

In functions/ai_flows/projectMeetingFlows.js, we built two automatic self-healing guards (isLongClipTranscriptTruncated and offsetTranscriptTimestamps) that detect both anomalies and seamlessly recover the full verbatim Cantonese/English diarization using multimodal gemini-3.8-flash (thinkingBudget: 0):

// Source: functions/ai_flows/projectMeetingFlows.js

/**
 * Shifts [MM:SS] timestamps in a diarized transcript by offsetSeconds for multi-clip meetings.
 */
export function offsetTranscriptTimestamps(transcriptText, offsetSeconds = 0) {
  if (!transcriptText || !offsetSeconds || offsetSeconds <= 0) {
    return transcriptText || '';
  }
  return String(transcriptText).replace(/\[(\d{1,2}:\d{2}(?::\d{2})?)\]/g, (_match, ts) => {
    const secs = parseTimestampToSeconds(ts) + Math.floor(offsetSeconds);
    return `[${formatSecondsToMmSs(secs)}]`;
  });
}

/**
 * Detects whether `gemini-3.5-transcribe-preview` prematurely stopped (e.g. due to a 15s silence pause
 * or unary audio token ceiling) on a long meeting clip (>= 360s / 6+ minutes).
 */
export function isLongClipTranscriptTruncated(transcriptText, durationSeconds = 0) {
  const dur = Number(durationSeconds) || 0;
  if (dur < 360) return false;
  const clean = String(transcriptText || '').trim();
  if (clean.length < 120) return true;
  const matches = [...clean.matchAll(/\[(\d{1,2}:\d{2}(?::\d{2})?)\]/g)];
  if (matches.length === 0) return false;
  let maxTsSecs = 0;
  for (const m of matches) {
    const secs = parseTimestampToSeconds(m[1]);
    if (secs > maxTsSecs) maxTsSecs = secs;
  }
  return maxTsSecs > 0 && maxTsSecs < dur * 0.45;
}

When isLongClipTranscriptTruncated(clipText, clipDurationSecs) returns true, Stage 1 automatically re-invokes generateWithResilience with an explicit full-duration boundary instruction ("IMPORTANT: This meeting recording is 22:15 long. Do NOT stop early on pauses; transcribe the entire recording from [00:00] through [22:15]."), ensuring 100% of the consultation dialogue reaches Stage 2!

The 2-Stage Multimodal Prompts Powering `analyzeProjectMeeting`

Here are the exact prompts executed across Stage 1 and Stage 2 of functions/ai_flows/projectMeetingFlows.js (and admin/prompts/audios/Project Group Meeting Analyzer.md):

1. Stage 1 Prompt — Multilingual Acoustic Speaker Diarization & Mobile Demo Video Transcription (`gemini-3.5-transcribe-preview`)

# Stage 1A: Voice Consultation Clip Diarization Prompt
Transcribe this project supervision meeting audio clip ({{clipLabel}}) verbatim in its original spoken language (Cantonese, Mandarin, or English, preserving English technical terms like React, Firestore, Docker, API).
Identify distinct voices using speaker diarization labels with [MM:SS] timestamps (e.g., [00:05] Speaker 1: ..., [00:42] Speaker 2: ...).
If a speaker states their name or is addressed by name, include their name next to the speaker label (e.g., [00:42] Speaker 2 (Alice): ...).

# Stage 1B: Long-Clip Truncation Recovery Suffix (Injected if maxTsSecs < dur * 0.45)
IMPORTANT: This meeting recording is {{clipDurationMmSs}} long. Do NOT stop early on pauses; transcribe the entire recording from [00:00] through [{{clipDurationMmSs}}].

# Stage 1C: 720p Mobile Camera Demo Video Dialogue Prompt
Transcribe any speech or spoken dialogue during this live project demonstration video clip ({{demoLabel}}) verbatim with speaker diarization and [MM:SS] timestamps. If no speech is present, reply with [Demonstration without speech].

2. Stage 2 Prompt — Multimodal Roster Attribution & Weekly Log vs. Live Demo Cross-Examination (`gemini-4-argon` / `gemini-3.8-flash`)

In Stage 2, functions/ai_flows/projectMeetingFlows.js packages both the snapped image/jpeg meeting photos (meetingPhotos.slice(0, 8)) and the 720p video/mp4 / video/webm mobile demo clips (mobileDemoClips.slice(0, 3)) into reasoningMediaParts alongside the text prompt so Gemini 4 Argon (gemini-4-argon) or Gemini 3.8 Flash (gemini-3.8-flash) visually inspects the student's actual hardware/code demonstrations as well as their spoken answers:

// Source: functions/ai_flows/projectMeetingFlows.js (Stage 2 Multimodal Media Assembly)
const reasoningMediaParts = [];
for (const photo of meetingPhotos.slice(0, 8)) {
  const pGsUri = resolveGsUri(photo.storagePath || photo.downloadUrl, bucketName);
  if (pGsUri) {
    reasoningMediaParts.push({ media: { url: pGsUri, contentType: 'image/jpeg' } });
  }
}
for (const vClip of mobileDemoClips.slice(0, 3)) {
  const vGsUri = resolveGsUri(vClip.storagePath || vClip.downloadUrl, bucketName);
  if (vGsUri) {
    const vContentType = vClip.mimeType?.includes('mp4') ? 'video/mp4' : 'video/webm';
    reasoningMediaParts.push({ media: { url: vGsUri, contentType: vContentType } });
  }
}

const { response: reasoningResponse, modelUsed: reasoningModelUsed } = await generateWithResilience(
  {
    prompt: [{ text: reasoningPrompt }, ...reasoningMediaParts],
    config: {
      temperature: 0.1,
      maxOutputTokens: 65536,
      responseMimeType: 'application/json',
    },
  },
  reasoningModel
);

Here is the Stage 2 reasoningPrompt sent with ...reasoningMediaParts:

You are an expert academic project supervisor and technical meeting analyst for higher education Capstone / Final Year Project (FYP) weekly supervision meetings in Hong Kong.
Speakers in the meeting code-switch naturally between Cantonese, Mandarin, and English technical terminology (e.g., `React`, `Firestore`, `Docker`, `Cloud Run`, `CI/CD`, `API`, `Git`, `PR`).

## Meeting Metadata
- Class ID: {{classId}}
- Project Group: {{groupName}}
- Project Topic: {{projectTopic}}
- Week Number: Week {{weekNumber}}
- Supervisor: {{supervisorName}} ({{supervisorEmail}})

## Enrolled Group Student Roster
{{rosterLines}}

## Deterministic Speaker & Multimodal Evidence Anchors (1-Tap Supervisor Voice Tags, 720p Mobile Demo Videos & Snapped Meeting Photos)
### 1-Tap Supervisor "Speaking Now" Anchors (100% Ground Truth Speaker Identity at Timestamp):
{{voiceAnchorLines}}

### Visual Demonstration Clips & Meeting Photo Artifacts:
#### Meeting Photo Artifacts Captured Live by Supervisor:
{{meetingPhotoLines}}

#### Demonstration Video Clips (720p Mobile Camera):
{{screenClipLines}}

## Prior Week's Assigned Action Items (for Continuity Check)
{{priorWeekContext}}

## Students' Submitted Weekly Log Sheet Entries for Week {{weekNumber}} (for Log Quality Grading & Fake-Log Verification)
{{weeklyLogsLines}}

## Stage 1 Diarized Audio & Demo Video Transcript (from gemini-3.5-transcribe-preview / gemini-3.8-flash)
"""
{{combinedDiarizedTranscript}}
"""

## Task Instructions
Using the Stage 1 Diarized Audio & Demo Video Transcript, the Deterministic Speaker & Multimodal Evidence Anchors, the attached Meeting Photo Artifacts and 720p Mobile Demo Videos, and the Students' Submitted Weekly Logs above:
1. Resolve anonymous speaker labels (Speaker 1, Speaker 2, etc.) to the exact students in the Enrolled Group Student Roster or the Supervisor using opening self-introductions, 1-tap voice anchors, discussed project tasks, and live 720p mobile camera demonstrations.
2. Determine the Attendance Record for every student in the roster ('present', 'late', or 'absent'), their speakingContribution ('high', 'moderate', 'low', 'none'), a concise evidence string with [MM:SS] timestamp, and firstHeardTimestamp ('MM:SS').
3. Extract Key Meeting Insights (executiveSummary, overallGroupProgress: 'on_track' | 'ahead' | 'at_risk' | 'behind', keyDecisions[], blockersAndRisks[], supervisorFeedback[], and weeklyGroupFeedback).
4. Cross-verify each student's **Submitted Weekly Log** against what they **actually discussed/demonstrated in the meeting**, including any attached **Meeting Photo Artifacts** (whiteboard diagrams, hardware wiring, terminal code) and **720p Mobile Camera Demonstration Videos**:
   - Flag **vague/useless logs** (e.g. "done", "thinking") as `"vague_useless"` and **AI-generated/fake logs** that were NOT backed up by real understanding, code, or visual demonstration in the meeting as `"suspected_ai_unverified"` or `"contradicted_in_meeting"`. If verified by their spoken explanation, 720p demo video, or photo artifacts, mark `"genuine_verified"`.
   - Grade `logQualityScore` (0 to 10) and `realWorkScore` (0 to 10). Cite specific visual evidence from photos or 720p demo clips when available.
   - Provide 1-2 `challengeQuestions` and a constructive 2-3 sentence `weeklyFeedback` for the student covering both their real project work and how to improve their weekly log quality.
5. Output speaker-attributed transcriptSegments[] with startTime ('MM:SS'), endTime ('MM:SS'), speaker (resolved student name or Supervisor), and text.

Production Code 3: Stage 2 Deterministic Roster Reconciliation (`reconcileMeetingRosterResults`)

In Stage 2, Gemini 4 Argon (gemini-4-argon) or Gemini 3.8 Flash (gemini-3.8-flash) cross-examines the diarized transcript, the 720p mobile demo videos, the snapped photos, and the students' written weekly logs. It evaluates whether a student's live demo and spoken answers verified or contradicted their written log (genuine_verified, contradicted_in_meeting, suspected_ai_fake, vague_useless, inactive) and assigns two separate marks out of 10:

logQualityScore (/10): Quality, specificity, and honesty of the written weekly logbook.
realWorkScore (/10): Verified engineering progress demonstrated live via the 720p demo video, photos, and technical Q&A.

However, what if a 4-member group has a silent free-rider who never spoke a word during the meeting, causing the LLM to omit them from its output array? Or what if the supervisor manually tapped 🗣️ Speaking Now for a student whose voice was too soft for the microphone?

In functions/ai_flows/projectMeetingFlows.js, reconcileMeetingRosterResults deterministically reconciles the AI's JSON output against the authoritative groupData.members roster:

// Source: functions/ai_flows/projectMeetingFlows.js
export function reconcileMeetingRosterResults({
  members = [],
  existingAttendance = [],
  aiAttendance = [],
  aiContributions = [],
  voiceAnchors = [],
  screenClips = [],
}) {
  // Maps existing manual overrides, AI attendance, and AI contributions by email/uid/name
  // ...
  const attendanceRecord = normalizedMembers.map((member) => {
    const email = (member.email || '').toLowerCase().trim();
    const uid = member.uid || email;
    const name = member.name || member.studentName || email;
    const existing = existingByKey.get(email) || existingByKey.get(uid.toLowerCase());
    const aiAtt = aiAttByKey.get(email) || aiAttByKey.get(uid.toLowerCase()) || aiAttByKey.get(name.toLowerCase()) || null;

    // Match 1-tap supervisor 'Speaking Now' voice anchors & demo clips for this student
    const studentAnchors = (Array.isArray(voiceAnchors) ? voiceAnchors : []).filter(
      (a) => (a.email && a.email.toLowerCase().trim() === email) || (a.studentUid && a.studentUid === uid)
    );
    const studentScreens = (Array.isArray(screenClips) ? screenClips : []).filter(
      (s) => (s.studentEmail && s.studentEmail.toLowerCase().trim() === email) || (s.studentUid && s.studentUid === uid)
    );

    const hasAnchorOrScreen = studentAnchors.length > 0 || studentScreens.length > 0;
    let status = aiAtt?.status || (hasAnchorOrScreen ? 'present' : 'absent');
    if (hasAnchorOrScreen && status === 'absent') {
      status = 'present'; // Supervisor voice anchor or demo clip overrides false-negative absence
    }

    let teacherVerified = false;
    if (existing && existing.teacherVerified) {
      status = existing.status || status; // Honor explicit teacher attendance badge override
      teacherVerified = true;
    } else if (studentAnchors.length > 0) {
      teacherVerified = true;
    }

    return {
      uid,
      email,
      name,
      role: member.role || 'Member',
      status,
      detectedByAi: Boolean(aiAtt?.status === 'present' || aiAtt?.status === 'late' || hasAnchorOrScreen),
      teacherVerified,
      speakingContribution: aiAtt?.speakingContribution || (hasAnchorOrScreen ? 'high' : status === 'present' ? 'moderate' : 'none'),
      evidence: aiAtt?.evidence || (status === 'present' ? 'Detected in meeting discussion.' : 'No spoken turn or demo detected.'),
    };
  });

  // Also normalizes studentContributions[], clamping logQualityScore & realWorkScore to [0, 10]
  return { attendanceRecord, studentContributions };
}

6. 1-Click Class-Wide HTML Email Dispatch & Longitudinal Student Portfolios

Once the supervisor reviews the generated report (or edits any score/feedback inline), clicking 📧 Email All Week W Reports invokes batchEmailProjectMeetingReports in functions/ai_flows/projectMeetingFlows.js.

Using SMTP credentials mounted strictly at runtime from Google Cloud Secret Manager (SMTP_PASSWORD), the function formats a responsive, badge-styled HTML supervision report containing:

An optional highlighted 👨‍🏫 Supervisor Note,
The Executive Summary & Key Architectural Decisions,
The Attendance & Speaking Contribution Table,
Each student's Written Weekly Log vs. Live Verified Evaluation (Log Quality /10, Real Work /10, Status Badge, Work Completed, Next Week Action Items, and Challenge Questions),
Dispatches the email simultaneously to all group members (To) and the supervisor (Cc), and writes an immutable emailDispatch audit timestamp onto the Firestore meeting document.

At any point during the semester, clicking 🎓 Student Profiles & History opens the longitudinal Student Project Portfolio View, charting each student's attendance rate, average Log Quality vs. Real Work scores across Weeks 1–16, and flags chronic free-riders before final capstone grading!

What's Next in Part 9?

In Part 9, we turn to live classroom engagement and formal assessment security—exploring how we engineered Gamified "Classroom Bingo" Active Presence Verification (backed by a Gemini 4 Argon Question Bank Studio and a Non-Punitive Two-Strike Google Cloud Tasks State Machine) alongside Zero-Trust Exam Mode Lockdown and Court-Ready Microsoft Word (.docx) Incident Dossiers.

Google AI Classroom Part 7: Serverless Map-Reduce-Map Video Intelligence & Rubric-Driven Practical Task Auto-Grading with Gemini 4 Argon

Cyrus Wong — Wed, 11 Nov 2026 03:29:14 +0000

Author: Cyrus Wong (黃俊彥) — Google Developer Expert - Cloud (AI), Senior Lecturer at HKIIT

1. The 80-Hour Lab Grading Bottleneck & The HTTP 429 Quota Wall

In hands-on cloud computing, DevSecOps, and full-stack software engineering courses, students spend 2 to 3 hours building infrastructure in the AWS/GCP Console, writing Terraform modules in VS Code, and debugging container deployments in the terminal. Reviewing a single 40-student cohort's 2-hour screen recordings manually requires 80 hours of instructor time per lab.

However, when our engineering team first attempted to automate post-class video grading by firing 40 concurrent multimodal video analysis calls against the Gemini API at the end of a lab session, two hard distributed systems walls appeared immediately:

The Serverless Execution Timeout Wall (540s): Even a 2nd-Gen Cloud Run-backed Cloud Function caps synchronous execution at 540 seconds (9 minutes). Sequentially evaluating 40 multi-megabyte lab recordings inside a single monolithic worker inevitably times out mid-cohort, leaving half the class ungraded and corrupting aggregate progress counters.
Gemini API HTTP 429 RESOURCE_EXHAUSTED Rate-Limit Spikes: Sending 40 uncompressed 30-FPS screen recordings simultaneously floods regional Tokens-Per-Minute (TPM) and Requests-Per-Minute (RPM) quotas, triggering cascading HTTP 429 rate-limit errors.
LLM Scoring Drift & Hallucinated Arithmetic: When evaluating multi-step practical exams, raw LLM JSON responses occasionally award 25/20 points on a rubric step, omit a milestone when a student stops early, or miscalculate the sum of individual step scores.

In Part 7 of our 10-part Google Developer Expert engineering series, we walk through how we solved all three challenges by combining:

Cloud Run FFmpeg 1-FPS H.264 Timelapse Compilation (shrinking a 2-hour lab into a <20 MB forensic MP4 and slashing multimodal video token consumption by 90%+).
Google Cloud Tasks Rate-Limited Fan-Out (maxDispatchesPerSecond: 2, maxConcurrentDispatches: 4) for zero-timeout, zero-429 distributed video processing.
Two-Stage Map-Reduce-Map Cohort Discovery (processVideoAnalysisJob $\rightarrow$ analyzeSingleVideoTask $\rightarrow$ generateLabTaskPrompt), enabling teachers to select 🧠 Gemini 4 Argon (gemini-4-argon), ⚡ Gemini 3.8 Flash (gemini-3.8-flash), or 🪶 Gemini 3.5 Flash-Lite (gemini-3.5-flash-lite) to automatically discover what milestones a class achieved and synthesize a tool-calling grading rubric.
A 4-Tab Practical Task & Deterministic Rubric Grading Engine (extractTaskDemoSteps.js, evaluateTaskSubmission.js, TaskEditorModal.jsx, and TaskGradingMatrixView.jsx) with Ultrafast Stream-Copy Retroactive Video Slicing (-c copy) and a deterministic mathematical guardrail (normalizeRubricEvaluation / reconcileStepEvaluationsWithRubric).

2. Diagram 1: From Periodic Screenshots to Forensic 1-FPS H.264 MP4s (80% Storage & 90% Token Reduction)

Why record a 30-FPS desktop video stream when a student coding in VS Code or configuring a VPC in the browser changes meaningful visual state every 5 to 15 seconds?

Before Gemini ever evaluates a student's work, StudentView.jsx and our high-memory Cloud Run worker processVideoJob.js (2 vCPU, 8 GiB RAM, 540s timeout) transform periodic screenshots into a razor-sharp, tamper-evident 1-FPS H.264 MP4 timelapse that is 80% smaller than continuous screen recordings:

Engineering Highlights of the Cloud Run FFmpeg Pipeline

Single-Pass Client Geometric Scaling: On 4K Retina displays, raw canvas captures can exceed 3 MB. Instead of an expensive multi-pass compression loop in the browser, StudentView.jsx computes a single-pass geometric scaling ratio: $$\text{scale} = \sqrt{\frac{\text{maxImageSize}}{\text{blob.size}}} \times 0.9$$ This guarantees every uploaded frame stays well beneath the 2 MB storage.rules threshold while preserving crisp terminal text legibility at up to 1920px (1080p).
Memory-Bounded Sharp SVG Watermarking (BATCH_SIZE = 15): Inside processVideoJob.js, downloading and compositing 720 high-resolution frames concurrently via Promise.all() would spike heap memory and crash the container. We process frames in chunks of BATCH_SIZE = 15, using sharp to burn a 40px forensic SVG banner onto every frame (YYYY-MM-DD HH:mm:ss | Class ID | Student Email | [SCREEN]/[WEBCAM]) and rounding dimensions to even integers (w - (w % 2)) required by the H.264 yuv420p planar chroma format.
Why Burn-In Timestamps Matter for Gemini 4: Because the compiled MP4 plays back at 1 frame per second (where 1 second of video equals 15 seconds of real classroom time), a naive LLM looking only at the MP4 container's playback clock would miscalculate how long a student spent on a task by a factor of $15\times$. Burning the real wall-clock timestamp directly into every frame allows Gemini 4 Argon (gemini-4-argon) and Gemini 3.8 Flash (gemini-3.8-flash) to read the exact on-screen clock via optical character recognition!

3. Diagram 2: Two-Stage Map-Reduce-Map Cohort Video Discovery with Cloud Tasks & Gemini 4 Argon

What happens when an instructor runs an open-ended hands-on cloud lab—where students explore AWS/GCP services at their own pace—and wants AI to:

Watch all 40 student timelapse videos without hitting rate limits (Phase 1: Map),
Discover the common coursework milestones, point weights, and technical blockers encountered across the entire cohort (Phase 2: Reduce), and
Re-evaluate every student against that newly synthesized rubric to populate a live Coursework Milestone Matrix (Phase 3: Second Map)?

In functions/ai_flows/index.mjs, generateLabTaskPrompt.js, VideoLibrary.jsx, and VideoAnalysisJobs.jsx, we built a Serverless Map-Reduce-Map Pipeline where teachers can select gemini-4-argon, gemini-3.8-flash, or gemini-3.5-flash-lite at both the Map and Reduce stages:

Production Code 1: The Reduce Stage — Synthesizing a Cohort Grading Rubric from 40 Student Videos (`generateLabTaskPrompt.js`)

Let's examine functions/ai_flows/generateLabTaskPrompt.js. When Phase 1 (the exploratory Map pass) finishes, each student's raw activity observation is stored in aiJobs/{jobId}. Because Firestore's in query operator accepts at most 30 document IDs per query, generateLabTaskPrompt fetches the completed child jobs in 30-document chunks, compiles them into a cohort-wide activity corpus, and invokes generateWithResilience using the teacher's chosen model (such as gemini-4-argon or gemini-3.8-flash):

// Source: functions/ai_flows/generateLabTaskPrompt.js
export const generateLabTaskPrompt = onCall(
  {
    region: FUNCTION_REGION,
    cors: CORS_ORIGINS,
    timeoutSeconds: 120,
    memory: '1GiB',
  },
  async (request) => {
    if (!request.auth) {
      throw new HttpsError('unauthenticated', 'The function must be called while authenticated.');
    }

    const { jobId, model } = request.data || {};
    const masterJobRef = db.collection('videoAnalysisJobs').doc(jobId);
    const jobDoc = await masterJobRef.get();
    const jobData = jobDoc.data();
    const classId = jobData.classId || 'unknown_class';
    const aiJobIds = jobData.aiJobIds || [];

    let completedJobs = [];

    // Fetch child aiJobs via 30-document batched FieldPath.documentId() queries
    if (aiJobIds.length > 0) {
      for (let i = 0; i < aiJobIds.length; i += 30) {
        const batchIds = aiJobIds.slice(i, i + 30);
        const querySnapshot = await db
          .collection('aiJobs')
          .where(FieldPath.documentId(), 'in', batchIds)
          .get();

        querySnapshot.forEach((doc) => {
          const data = doc.data();
          if (data.status === 'completed' && data.result && typeof data.result === 'string' && data.result.trim()) {
            completedJobs.push(data);
          }
        });
      }
    }

    // Compile observations across all students in the cohort
    const compiledObservations = completedJobs
      .map((job, idx) => {
        const studentIdentifier = job.studentEmail || job.studentUid || `Student ${idx + 1}`;
        return `### Student ${idx + 1}: ${studentIdentifier}\n${job.result.trim()}`;
      })
      .join('\n\n---\n\n');

    const preferredModel = model || 'gemini-3.8-flash';
    const { response, modelUsed } = await generateWithResilience(
      {
        prompt: [{ text: metaPrompt }],
        config: { temperature: 0.2 },
      },
      preferredModel
    );

    const usage = response?.usage || response?.usageMetadata || {};
    const cost = calculateCost(usage, modelUsed || preferredModel);
    await logJob({
      classId,
      jobType: 'generateLabTaskPrompt',
      status: 'completed',
      promptText: metaPrompt,
      mediaPaths: [],
      usage: {
        inputTokens: usage.inputTokens ?? usage.promptTokenCount ?? 0,
        outputTokens: usage.outputTokens ?? usage.candidatesTokenCount ?? 0,
      },
      cost,
      modelUsed: modelUsed || preferredModel,
      result: generatedPrompt.trim().substring(0, 300),
      masterJobId: jobId,
    });

    return {
      generatedPrompt: generatedPrompt.trim(),
      modelUsed,
      summaryCount: completedJobs.length,
      classId,
      jobId,
    };
  }
);

The 3 Prompts Powering the Map-Reduce-Map Video Pipeline

To make this architecture concrete and reproducible, here are the exact prompts executed at each stage of the Map-Reduce-Map workflow:

1. Phase 1 (Exploratory Map Prompt) — Unsupervised Activity Discovery (`Summarize Student's Work.md`)

In Phase 1, the lecturer does not need to write a lab rubric in advance. Each Cloud Tasks worker (analyzeSingleVideoTask $\rightarrow$ analyzeSingleVideoFlow in functions/ai_flows/analysisFlows.js) wraps the exploratory template (admin/prompts/videos/Summarize Student's Work.md) with the class's time-lapse capture interval (captureInterval = 15s) and dispatches it in parallel across all 40 student videos:

# Summarize Student's Activity

**You are an AI teaching assistant. Your task is to summarize the activity of each student based on their video.**

## Video Context
* The video is a fast-forwarded time-lapse of screen captures, not a real-time recording (compiled at 1 frame per second from screenshots captured roughly every {{captureInterval}} seconds).
* Each frame includes a burned-in timestamp (date and time). All calculations of duration (e.g., for attendance, concentration time) **MUST** be based on these in-frame timestamps.
* **Do not use the video's playback length for any time-based calculations.**

## Attendance & Engagement Analysis
* Compare the timeframe of the student's activity (derived from the in-frame timestamps) with the scheduled lesson time ({{startTime}} to {{endTime}}) to estimate attendance. Note any late starts, early leaves, or gaps in screen sharing.
* Based on the student's on-screen activity and focus, estimate the total "concentration minutes" during the observed period using the in-frame timestamps, and call the `recordActualWorkingTime` tool.

## Analysis Guidelines
* Identify the main tasks, cloud consoles, CLI commands, and code files the student is working on throughout the video.
* Describe the student's pace, milestones achieved, and any test scores or verification outputs visible on screen.
* Note any periods of inactivity, error messages, or technical struggle.

## Actions
* Call `recordActualWorkingTime` to record your estimate of the student's concentration minutes.
* Call `recordLessonSummary` to record a concise, informative summary of their activity, progress, and blockers.
* Provide the exact same summary as your final text response.

2. Phase 2 (Reduce Meta-Prompt) — Synthesizing a Cohort Rubric from 40 Observations (`Two-Stage Map-Reduce Lab Rubric Synthesizer.md`)

Once the Phase 1 Map workers finish, clicking ✨ Generate Lab Task Prompt in VideoAnalysisJobs.jsx invokes generateLabTaskPrompt.js, which injects all 40 student summaries into {{compiledObservations}} inside admin/prompts/videos/Two-Stage Map-Reduce Lab Rubric Synthesizer.md:

You are an expert Computer Science university curriculum designer and AI prompt engineer for classroom invigilation and automated lab assessment.

Below are student activity observations collected from an initial video analysis across a class session (Class ID: "{{classId}}"):

=== BEGIN OBSERVED STUDENT ACTIVITIES ===
{{compiledObservations}}
=== END OBSERVED STUDENT ACTIVITIES ===

Based on these actual classroom observations, analyze the student activities and generate an optimized, comprehensive, ready-to-use Markdown task prompt for a high-precision second-pass evaluation of this lab.

The generated prompt MUST strictly follow this Markdown structure:

# [Descriptive Lab Title Based on Discovered Coursework]

**You are an AI teaching assistant evaluating student screen recording time-lapses for [Lab Topic/Subject].**

## Video Timing Rules
* The video is a fast-forward time-lapse with on-screen timestamps.
* All duration, attendance, and task timing calculations **MUST** be derived from the in-frame date/time stamps, NOT the video playback length.

## Coursework Tasks to Evaluate
[Break down the lab into 3-5 clearly numbered, specific tasks discovered from the student observations. Include specific platforms (e.g. AWS, Azure, Cloud Shell, VS Code), repositories, scripts, resource names, and milestones].

## Milestones, Scores & Rubrics
[Detail any test numbers, scoring points (e.g., 10 pts, 20 pts), or completion verifications discovered from the observations].

## Known Blockers & Technical Obstacles to Watch For
[List the common technical errors, blockers, or pitfalls students encountered, such as backend errors, incorrect regions, or setup issues].

## Required Tool Actions
1. **Working Time**: Call 'recordActualWorkingTime' with the total estimated active concentration minutes (clamped to lesson duration).
2. **Task Durations**: For each identified task above, call 'recordTaskDuration' with studentUid, classId, taskName, and estimated durationMinutes spent on that task.
3. **Student Summary**: Call 'recordLessonSummary' with a structured bulleted summary of their progress across each task, points earned, blockers encountered, and engagement verdict.
4. **Final Response**: Output the exact same structured summary text provided to 'recordLessonSummary'.

Return ONLY the clean Markdown prompt text. Do not wrap in markdown code fence blocks.

3. Phase 3 (Second-Map Evaluation Prompt) — Example Output Synthesized by Gemini 4 Argon

Here is an actual example of the Markdown prompt synthesized by Phase 2 from a 40-student cloud infrastructure lab, which the teacher reviews in the modal and dispatches back to Cloud Tasks (analyzeSingleVideoTask) for the Second Map pass:

# Serverless Cloud Run & Terraform Infrastructure Lab Evaluation

**You are an AI teaching assistant evaluating student screen recording time-lapses for Cloud Infrastructure & Serverless Container Deployment.**

## Video Timing Rules
* The video is a fast-forward time-lapse with on-screen timestamps (`YYYY-MM-DD HH:mm:ss`).
* All duration, attendance, and task timing calculations **MUST** be derived from the in-frame date/time stamps, NOT the video playback length.

## Coursework Tasks to Evaluate
1. **Task 1: Environment Setup & Repository Cloning**: Cloning the starter repo in VS Code / Cloud Shell, configuring `.env` variables, and running `npm install`.
2. **Task 2: Docker Containerization & Local Verification**: Writing the multi-stage `Dockerfile`, building the container image via `docker build` / `gcloud builds submit`, and verifying `localhost:8080/healthz` returns `HTTP 200 OK`.
3. **Task 3: Terraform Cloud Run & Firestore Provisioning**: Writing `main.tf` (`google_cloud_run_v2_service` and `google_firestore_database`), running `terraform init && terraform plan`, and applying infrastructure via `terraform apply -auto-approve`.
4. **Task 4: End-to-End Automated Grading Script Verification**: Executing `./run-lab-tests.sh` in the terminal and verifying all 5 integration assertions pass against the live `.run.app` endpoint.

## Milestones, Scores & Rubrics
* **Task 1 (15 pts)**: Starter repository cloned and dependencies installed without missing package errors.
* **Task 2 (25 pts)**: Container image built and local health check (`200 OK`) verified in terminal or browser.
* **Task 3 (35 pts)**: `terraform apply` completes with `Apply complete! Resources: 4 added, 0 changed, 0 destroyed` and outputs the live Cloud Run URL.
* **Task 4 (25 pts)**: Automated test script outputs `SCORE: 100/100 (5/5 tests passed)`.

## Known Blockers & Technical Obstacles to Watch For
* **Port Mismatch (`PORT=8080`)**: Container failing Cloud Run startup probe because the Express server listened on port `3000` instead of `process.env.PORT`.
* **IAM Permission Denied (`403`)**: `terraform apply` failing on Cloud Run IAM binding due to missing `roles/run.admin` or unenabled `run.googleapis.com` API.
* **Region Mismatch**: Deploying Firestore in `us-central1` while Cloud Run is configured in `asia-east2`.

## Required Tool Actions
1. **Working Time**: Call `recordActualWorkingTime` with the total estimated active concentration minutes (clamped to lesson duration).
2. **Task Durations**: For each identified task above (`Task 1`, `Task 2`, `Task 3`, `Task 4`), call `recordTaskDuration` with `studentUid`, `classId`, `taskName`, and estimated `durationMinutes` spent on that task.
3. **Student Summary**: Call `recordLessonSummary` with a structured bulleted summary of their progress across each task, points earned (out of 100), blockers encountered, and engagement verdict.
4. **Final Response**: Output the exact same structured summary text provided to `recordLessonSummary`.

When the teacher runs Phase 3 (the Milestone Evaluation Map) using that synthesized prompt, each Cloud Tasks worker invokes Genkit tools (recordActualWorkingTime, recordTaskDuration, and recordLessonSummary) that populate VideoAnalysisJobs.jsx and TaskGradingMatrixView.jsx—giving the lecturer a color-coded cohort progress matrix showing exactly how many minutes each student spent on Task 1, Task 2, Task 3, and Task 4, and where they got stuck!

4. Diagram 3: 4-Tab Practical Tasks, Retroactive Post-Class Slicing & Deterministic Rubric Reconciliation

While Map-Reduce-Map excels at exploratory cohort discovery, graded practical assignments and lab tests (📋 Tasks & Homework tab) require a strict, reproducible rubric where every student is graded against an identical point breakdown, with timestamped visual proof and full teacher override authority.

To support this, we engineered a 4-Tab Practical Task & Rubric Grading System across TaskEditorModal.jsx, TaskGradingMatrixView.jsx, extractTaskDemoSteps.js, and evaluateTaskSubmission.js.

Teachers have two independent AI model selection control points supporting 🧠 Gemini 4 Argon (gemini-4-argon), ⚡ Gemini 3.8 Flash (gemini-3.8-flash), and 🪶 Gemini 3.5 Flash-Lite (gemini-3.5-flash-lite):

Task Editor Modal (#task-ai-model-select in TaskEditorModal.jsx): Chooses the default AI model stored on classes/{classId}/tasks/{taskId}.model and used by extractTaskDemoSteps when watching a teacher's reference demonstration video to synthesize the step-by-step rubric.
Grading Matrix Toolbar (Grading AI Model selector in TaskGradingMatrixView.jsx): Allows the teacher to override the evaluation model on the fly when running single-student re-evaluations (handleTriggerEvaluate) or batch cohort grading (handleBatchEvaluatePending via batchEvaluateTaskSubmissions). For example, a teacher can grade formative practice attempts with gemini-3.5-flash-lite and switch the toolbar selector to gemini-4-argon for final high-stakes practical exam grading!

5. Deep Dive into the Production Code: Rubric Extraction, Zero-Race Stream-Copy Slicing & Deterministic Score Reconciliation

Production Code 2: Multimodal Reference Video Rubric Synthesis (`extractTaskDemoSteps.js`)

Instead of writing complex JSON rubrics by hand, an instructor can upload a 3-minute demonstration video of the solution (or pick an existing Teacher Lecture Recording or paste a YouTube URL) in Tab 2 of TaskEditorModal.jsx and click ✨ Extract Steps & Rubric with AI.

In functions/ai_flows/extractTaskDemoSteps.js, handleExtractTaskDemoSteps passes the video URI alongside any teacher marking notes to generateWithResilience(generateConfig, preferredModel) and enforces a strict JSON schema containing stepNumber, title, description, expectedEvidence, gradingCriteria, and points:

// Source: functions/ai_flows/extractTaskDemoSteps.js
export const handleExtractTaskDemoSteps = async ({
  classId,
  demoVideoPath,
  promptGuidelines,
  promptText,
  maxScore = 100,
  model,
}) => {
  const storage = getStorage();
  const bucketName = storage.bucket().name;

  const isYouTube = demoVideoPath
    ? /(?:youtu\.be\/|youtube\.com\/(?:embed\/|v\/|watch\?v=|watch\?.+&v=|live\/))([a-zA-Z0-9_-]{11})/i.test(demoVideoPath)
    : false;
  let resolvedVideoUri = null;
  if (demoVideoPath) {
    if (isYouTube || demoVideoPath.startsWith('http://') || demoVideoPath.startsWith('https://')) {
      resolvedVideoUri = demoVideoPath;
    } else if (demoVideoPath.startsWith('gs://')) {
      resolvedVideoUri = demoVideoPath;
    } else {
      resolvedVideoUri = `gs://${bucketName}/${demoVideoPath.replace(/^\/+/, '')}`;
    }
  }

  const preferredModel = model || 'gemini-3.8-flash';
  const promptParts = [{ text: metaPrompt }];
  if (resolvedVideoUri) {
    promptParts.push({ media: { url: resolvedVideoUri, contentType: 'video/mp4' } });
  }
  const generateConfig = {
    prompt: promptParts,
    config: {
      temperature: 0.2,
      responseMimeType: 'application/json',
    },
  };

  const { response, modelUsed } = await generateWithResilience(generateConfig, preferredModel);
  const parsed = JSON.parse(cleanJsonFence(response.text || ''));

  if (Array.isArray(parsed?.steps)) {
    parsed.steps = parsed.steps.map((s, idx) => ({
      stepNumber: Number(s.stepNumber) || idx + 1,
      title: s.title || `Milestone ${idx + 1}`,
      description: s.description || '',
      expectedEvidence: s.expectedEvidence || '',
      gradingCriteria: s.gradingCriteria || '',
      points: Number(s.points ?? s.weight ?? 20),
    }));
  }

  return { ...parsed, modelUsed, cost: calculateCost(response?.usage, modelUsed || preferredModel) };
};

Here is the metaPrompt inside extractTaskDemoSteps.js (backed by admin/prompts/rubrics/Practical Task Demo Step Extractor.md) that extracts a chronological JSON rubric from a teacher's demo video:

You are an expert technical educator and curriculum designer creating a structured grading rubric for a hands-on practical assignment or lab test.
Analyze the visual actions, commands, code editor steps, browser verifications, and results shown in the demonstration video.

Extract a structured, chronological step-by-step assessment rubric (between 3 and 8 distinct milestones).
For each milestone:
1. Provide a concise, clear title.
2. Describe the specific action or requirement (e.g. terminal command, file created, setting configured).
3. Specify the expected visual proof/evidence that an AI evaluator should search for in a student's screen recording (e.g. terminal output text, open ports, browser UI, green test checks).
4. Provide explicit marking / partial-credit criteria (how full marks vs. partial marks vs. 0 marks should be awarded).
5. Assign a point weight (total across all steps must equal 100 points).

You MUST respond strictly with valid JSON conforming to this schema:
{
  "title": "A concise, descriptive title for this practical task",
  "description": "A clear 2-3 sentence overview of what the student is required to reproduce",
  "suggestedMaxScore": 100,
  "steps": [
    {
      "stepNumber": 1,
      "title": "Short Step Title",
      "description": "Specific action performed by the student",
      "expectedEvidence": "Specific visual evidence, keywords, or UI elements on screen",
      "gradingCriteria": "Full marks if completed and verified; partial marks if configured with minor syntax errors",
      "points": 20
    }
  ]
}

Production Code 3: Eliminating the Post-Lesson Screenshot Deletion Race Condition with Ultrafast Stream-Copy Slicing (`slicePendingTaskSubmissions`)

A subtle race condition occurs during In-Class Practical Tasks:

At the end of a scheduled lesson, processVideoJob.js compiles all raw screenshots from screenshots/{classId}/{studentUid}/ into the master class timelapse video (videos/{classId}/{videoId}.mp4), and then deletes the raw screenshots to save 80% of Cloud Storage costs.
What if a student clicked Submit Task right as class ended, or what if a teacher realized after class that they forgot to create a task beforehand and wants to retroactively slice minutes 30:00 to 75:00 from all 40 students' compiled class videos?

In functions/media_processing/processVideoJob.js, slicePendingTaskSubmissions solves both problems simultaneously. Instead of re-downloading raw screenshots, it maps the task's [taskStartMs, taskEndMs] window (or tagged activeTaskId frame indices) to exact 1-FPS video offsets (startSec, durSec) and slices attempt_${attemptNumber}.mp4 directly out of the compiled class MP4 using FFmpeg lossless stream-copying (-c copy) in less than 0.5 seconds!

// Source: functions/media_processing/processVideoJob.js
export async function slicePendingTaskSubmissions({
  classId,
  studentUid,
  studentEmail,
  classStartTime,
  captureInterval = 15,
  outputVideoPath,
  screenshots = [],
  db,
  bucket,
  retry,
}) {
  // Identify exact 1-FPS start frame index and duration from tagged screenshots or timestamp math
  let firstIndex = screenshots.findIndex((s) =>
    (s.activeTaskId && s.activeTaskId === taskId) || (!isNaN(toMs(s.timestamp)) && toMs(s.timestamp) >= taskStartMs)
  );
  // ... (calculates startSec and durSec)

  const taskClipPath = path.join(os.tmpdir(), `${taskJobDoc.id}_clip.mp4`);
  await new Promise((resolve, reject) => {
    ffmpeg(outputVideoPath)
      .setStartTime(startSec)
      .setDuration(durSec)
      .outputOptions([
        '-c', 'copy',
        '-avoid_negative_ts', 'make_zero',
        '-movflags', '+faststart'
      ])
      .output(taskClipPath)
      .on('end', resolve)
      .on('error', () => {
        // Automatic fallback to ultrafast libx264 re-encode if keyframe boundary requires it
        ffmpeg(outputVideoPath)
          .setStartTime(startSec)
          .setDuration(durSec)
          .outputOptions(['-c:v', 'libx264', '-pix_fmt', 'yuv420p', '-preset', 'ultrafast', '-crf', '28', '-movflags', '+faststart'])
          .output(taskClipPath)
          .on('end', resolve)
          .on('error', reject)
          .run();
      })
      .run();
  });

  const destinationPath = `classes/${classId}/tasks/${taskId}/submissions/${studentUid}/attempt_${attemptNumber}.mp4`;
  await retry(() => bucket.upload(taskClipPath, { destination: destinationPath, /* metadata */ }));
}

And if a teacher uses ⏪ Post-Class Retroactive Slicing inside TaskGradingMatrixView.jsx, our retroactive slicing Cloud Function (functions/media_processing/extractRetroactiveTaskSubmissions.js) slices the specified [startOffsetMinutes, durationMinutes] window across all 40 class videos in parallel and immediately pipelines the sliced clips into the Gemini grading queue!

Production Code 4: Deterministic Rubric Reconciliation & Mathematical Guardrails (`evaluateTaskSubmission.js`)

Even frontier models like Gemini 4 Argon (gemini-4-argon) are probabilistic token generators—we should never trust raw LLM arithmetic for official academic gradebooks.

In functions/ai_flows/evaluateTaskSubmission.js (mirrored on the frontend by reconcileStepEvaluationsWithRubric in TaskGradingMatrixView.jsx), normalizeRubricEvaluation acts as a deterministic post-processing barrier between Gemini's JSON output and Firestore:

1-to-1 Step Alignment: It iterates over the authoritative effectiveSteps array from the task definition, matching AI step results by stepNumber, case-insensitive title, or positional index.
Strict [0, maxPoints] Clamping: Every step's scoreAwarded is clamped via Math.max(0, Math.min(maxPoints, ...)) so the AI can never award negative points or exceed a step's point ceiling.
Zero-Defaulting Omitted Steps: If a student stopped halfway through the lab and the LLM omitted Steps 4 and 5 from its JSON array, normalizeRubricEvaluation deterministically populates Steps 4 and 5 with scoreAwarded: 0, status: 'missing', and explicit feedback (No evidence of "${stepTitle}" was observed in the submitted screen recording.).
Exact Bottom-Up Summation & Multi-Attempt Scoring Strategy: finalScore is computed strictly as the sum of reconciled scoreAwarded values across all steps, and the student's root submission document automatically updates bestScore, latestScore, averageScore, and effectiveScore based on the task's configured scoring strategy (highest, latest, or average).

First, here is the strict rubric-driven evaluation prompt dispatched by evaluateTaskSubmission.js:

You are an expert technical invigilator and automated practical exam evaluator.
Evaluate the student's continuous screen recording time-lapse video strictly against the following practical assignment rubric.

=== ASSIGNMENT DETAILS ===
Title: "{{taskData.title}}"
Overview: "{{taskData.description}}"
Total Possible Score: {{maxScore}} points

=== REQUIRED RUBRIC MILESTONES (STRICT MARK BREAKDOWN) ===
{{effectiveStepsJson}}

=== INSTRUCTIONS FOR RUBRIC-DRIVEN EVALUATION ===
1. Watch the student's screen actions chronologically from start to finish.
2. You MUST evaluate EVERY SINGLE milestone listed in REQUIRED RUBRIC MILESTONES above in exact order:
   - Determine whether the step was "completed", "partial", "error", or "missing".
   - Record the approximate "mm:ss" timestamp ("timestampInVideo") where the step was attempted or completed (or null if missing).
   - Describe the concrete visual proof observed on the student's screen ("evidenceObserved", e.g., terminal command output, code written, browser state, or what was missing).
   - Award marks ("scoreAwarded") strictly between 0 and the step's allocated "points" ("maxPoints") based on the step's "expectedEvidence" and "gradingCriteria".
   - Provide a clear, detailed per-step explanation ("feedback") for the student and teacher explaining WHY those exact marks were awarded or deducted according to the rubric.
3. Highlight 2-3 specific technical strengths demonstrated by this student in the video.
4. Highlight any errors, warnings, configuration oversights, or suspicious anomalies (e.g. abrupt paste of whole codebase, unhandled exceptions).
5. Calculate "finalScore" as the exact sum of "scoreAwarded" across all rubric steps (out of {{maxScore}}).

And here is how normalizeRubricEvaluation deterministically reconciles Gemini's JSON response against the task rubric:

// Source: functions/ai_flows/evaluateTaskSubmission.js
export const normalizeRubricEvaluation = (rawEvaluation = {}, effectiveSteps = [], maxScore = 100) => {
  const normalizedMaxScore = Number(maxScore) || 100;
  const rawSteps = Array.isArray(rawEvaluation.stepResults)
    ? rawEvaluation.stepResults
    : Array.isArray(rawEvaluation.rubricScores)
    ? rawEvaluation.rubricScores
    : [];

  let reconciledSteps = [];

  if (Array.isArray(effectiveSteps) && effectiveSteps.length > 0) {
    const defaultStepPoints = Math.round(normalizedMaxScore / effectiveSteps.length);
    reconciledSteps = effectiveSteps.map((rubricStep, idx) => {
      const stepNum = Number(rubricStep.stepNumber ?? idx + 1);
      const maxPoints = Number(rubricStep.points ?? rubricStep.weight ?? defaultStepPoints) || 0;
      const stepTitle = rubricStep.title || rubricStep.criterion || `Step ${stepNum}`;

      const matched =
        rawSteps.find((s) => Number(s?.stepNumber) === stepNum) ||
        rawSteps.find(
          (s) =>
            s?.title &&
            stepTitle &&
            String(s.title).trim().toLowerCase() === String(stepTitle).trim().toLowerCase()
        ) ||
        rawSteps[idx] ||
        null;

      if (matched) {
        const rawScore = Number(
          matched.scoreAwarded ?? matched.score ?? (matched.status === 'completed' ? maxPoints : 0)
        );
        // Strictly clamp scoreAwarded to [0, maxPoints]
        const scoreAwarded = Math.max(
          0,
          Math.min(maxPoints, Number.isFinite(rawScore) ? Math.round(rawScore * 100) / 100 : 0)
        );
        const status =
          matched.status ||
          (scoreAwarded >= maxPoints && maxPoints > 0 ? 'completed' : scoreAwarded > 0 ? 'partial' : 'missing');

        return {
          stepNumber: stepNum,
          title: stepTitle,
          maxPoints,
          scoreAwarded,
          status,
          timestampInVideo: matched.timestampInVideo || matched.timestamp || null,
          evidenceObserved: matched.evidenceObserved || matched.evidence || '',
          feedback:
            matched.feedback ||
            matched.rationale ||
            matched.explanation ||
            (status === 'completed'
              ? `Completed all requirements for ${stepTitle}.`
              : `Incomplete or missing requirements for ${stepTitle}.`),
          expectedEvidence: rubricStep.expectedEvidence || '',
          gradingCriteria: rubricStep.gradingCriteria || '',
        };
      }

      // Deterministic fallback if the AI omitted a rubric step
      return {
        stepNumber: stepNum,
        title: stepTitle,
        maxPoints,
        scoreAwarded: 0,
        status: 'missing',
        timestampInVideo: null,
        evidenceObserved: 'Not observed in submission recording.',
        feedback: `No evidence of "${stepTitle}" was observed in the submitted screen recording.`,
        expectedEvidence: rubricStep.expectedEvidence || '',
        gradingCriteria: rubricStep.gradingCriteria || '',
      };
    });
  }

  // Exact bottom-up summation of clamped step scores
  const sum = reconciledSteps.reduce((acc, s) => acc + (Number(s.scoreAwarded) || 0), 0);
  const computedFinalScore = Math.max(0, Math.min(normalizedMaxScore, Math.round(sum * 100) / 100));

  return {
    ...rawEvaluation,
    finalScore: computedFinalScore,
    maxScore: normalizedMaxScore,
    completionPercentage: Math.round((computedFinalScore / normalizedMaxScore) * 100),
    stepResults: reconciledSteps,
  };
};

6. Architectural Comparison: Choosing the Right Gemini Model for Video Intelligence & Rubric Grading

Because instructors can select the AI model per job in VideoLibrary.jsx, TaskEditorModal.jsx (#task-ai-model-select), and TaskGradingMatrixView.jsx (Grading AI Model), here is how our lecturers allocate models across different assessment workloads:

Assessment Workload	Recommended Model	Why It Excels	Configured Rate in `cost.js` (Input / Output per 1M Tokens)
Daily Formative Lab Practice & Self-Check Attempts	🪶 Gemini 3.5 Flash-Lite (`gemini-3.5-flash-lite`)	Fast feedback after submission; lowest token rate for high-frequency self-checks.	`$0.30 / $2.50`
Standard Weekly Coursework & Map-Reduce Cohort Discovery	⚡ Gemini 3.8 Flash (`gemini-3.8-flash`)	Balanced multimodal OCR on dense terminal logs and IDE split-panes.	`$0.75 / $3.75`
High-Stakes Practical Exams & Complex Multi-Step Rubric Synthesis	🧠 Gemini 4 Argon (`gemini-4-argon`)	Frontier visual reasoning across code diffs, architecture diagrams, and multi-terminal debugging.	`$2.00 / $10.00`

In TaskGradingMatrixView.jsx, teachers see every student's per-step pill breakdown (S1: 30/30, S2: 62/70) directly in the matrix table, can click any [⏱️ 02:15] badge to seek the HTML5 video player to the exact frame where Gemini observed the evidence, override any individual step score or explanation with full audit trails (isTeacherEdited: true), and export a multi-sheet Excel Gradebook (.xlsx) or Google Drive Archive with one click.

What's Next in Part 8?

In Part 8, we move from individual lab screen recordings to face-to-face engineering capstone project consultations—showing how we turned the project supervisor's smartphone into a Streamlined Single-Device Multimodal Supervision Studio (continuous voice + zero-pause 0.1s photo snap + auto-sliced 720p mobile demo video) backed by self-healing Google Sheets weekly log sync, pre-meeting fake-log AI auditing, and a Two-Stage gemini-3.5-transcribe-preview + gemini-4-argon Cross-Examination Pipeline.

Google AI Classroom Part 6: Low-Bandwidth Classroom Screen Broadcasting & Single-Pass Whole-Lecture Subtitles with Gemini 4 Argon

Cyrus Wong — Wed, 11 Nov 2026 01:27:06 +0000

Author: Cyrus Wong (黃俊彥) — Google Developer Expert - Cloud (AI), Senior Lecturer at HKIIT

1. Problem Statement: Breaking Two Hard Ceilings in Lecture Hall Broadcasting & Post-Lecture Publishing

When broadcasting a live coding demonstration to a 30-to-80 student lab or up to a 600-seat multi-cohort lecture hall—and publishing the recorded 3-hour lecture with multilingual closed captions to YouTube Studio afterward—conventional web architectures hit two catastrophic walls:

The WebRTC 6-Peer CPU Wall & Firebase Anonymous Auth 100/hr NAT IP Ceiling:
- Peer-to-peer WebRTC screen sharing encodes a separate VP8/H.264 video stream per connected viewer ($O(N)$ CPU complexity), pegging the presenter's laptop CPU at 100% after just 6–8 students.
- Worse, when dozens of students in a lecture hall scan a live broadcast QR code while connected to the same campus Wi-Fi NAT gateway IP, calling signInAnonymously() immediately triggers Firebase Auth's hard 100-signups-per-hour-per-IP rate limit (auth/too-many-requests), locking out attendees.
The Chunked Speech-Preview VAD Cutoff & Timescale Drift Bugs:
- Early speech-to-text preview models enforced a 45,000 audio token ceiling (~30 minutes) and mistook natural 10-second classroom pauses—while the lecturer writes code in VS Code or draws a diagram on the whiteboard—as end-of-speech (finishReason: STOP), silently dropping the remaining 75% of the lecture's captions.
- Furthermore, feeding raw MediaRecorder WebM Opus chunks into long-context multimodal models caused 1.48×–1.68× timescale compression drift (subtitles running faster than wall-clock video playback) and wasted 90% of the token budget on redundant video frames.

In this post, we walk through how we solved both problems using:

A 32×18 RGB Pixel-Delta Screen Broadcaster (<2% presenter CPU) paired with Zero-Auth Firestore PIN Security Rules (/live/:classId?pin=XXXX) that scale effortlessly from daily 30-student labs up to 600-student multi-cohort lecture halls.
A Crash-Tolerant 1-Minute Segmented Recorder backed by IndexedDB recovery and a Cloud Run FFmpeg Merger guarded by a strict 5-Point Pre-Deletion Verification Gate.
A Single-Pass Audio-Only Whole-Lecture Subtitle & YouTube Studio Pipeline powered by Gemini 4 Argon (gemini-4-argon), Gemini 3.8 Flash (gemini-3.8-flash), and Gemini 3.5 Flash-Lite (gemini-3.5-flash-lite) with 48kHz CBR MP3 normalization, linear timescale calibration, and a Chromium zero-duration cue guard.

2. Architectural Deep-Dive: Why This Architecture Scales ($O(1)$ vs. $O(N)$ Complexity)

In a programming lecture, the presenter's screen (slides, IDE code, terminal output) needs crisp 1080p / 1440p text legibility, not 60 FPS motion video. And when the lecturer talks over a slide for 45 seconds without scrolling, re-uploading identical frames wastes bandwidth and compute.

Diagram 1: Why Serverless Screen Broadcast Scales — WebRTC $O(N)$ CPU Collapse vs. $O(1)$ Firestore Fanout

Why WebRTC Mesh Fails in Classrooms vs. How Our $O(1)$ Design Solves It

The WebRTC Mesh $O(N)$ Trap: In peer-to-peer WebRTC, the presenter's browser must establish an independent peer connection and video encoder pipeline for each connected student. For $N$ students, the presenter's CPU and upstream bandwidth scale as $O(N)$. On typical laptops, CPU load hits ~75% at 8 peers and locks up at 12–15 peers with dropped frames and audio glitching.
Our Serverless Single-Write $O(1)$ Presenter Workload:
- The teacher's browser runs an inline Web Worker timer and a 32×18 thumbnail canvas to compute pixel deltas.
- If the screen is static, 0 frames are uploaded. When slide or code changes, it encodes one single JPEG frame.
- It writes to exactly one Firestore document: classes/{classId}/screenBroadcast/liveFrame.
- The presenter's machine workload is strictly $O(1)$—taking <2% CPU whether 5 students or 600 students are watching.
Google Cloud Firestore Real-Time Fanout:
- Google Cloud's distributed Firestore infrastructure handles the fanout to all student onSnapshot() listeners.
Zero-Auth Campus NAT Gate:
- Standard Firebase Auth anonymous sign-ins share the campus Wi-Fi public IP and quickly hit the 100 signups/hr per-IP limit.
- We validate a 4-digit session PIN directly inside firestore.rules without invoking Firebase Auth, allowing entire student cohorts to connect simultaneously without authentication throttling.

Note

Architectural Boundary & Sizing: This pipeline is intentionally designed for slide and code demonstration at 0.5–1 FPS (1 frame every 1–3s) where text sharpness is paramount. It is not intended for 30 FPS video streaming. For higher education computer labs (30–80 students) and peak joint multi-cohort lecture halls (up to 600 seats), this zero-infrastructure serverless pattern delivers rock-solid stability without maintaining expensive media servers. Beyond 600 seats (e.g. multi-thousand public MOOCs or stadium keynotes), an institution would transition to CDN-backed HLS/DASH media distribution.

Implementation Walkthrough: 32×18 RGB Pixel Diffing, Hardware `ImageCapture`, & 950 KB Payload Guard

In web-app/src/hooks/useTeacherScreenBroadcast.js, our broadcaster solves four gnarly browser compositor bugs:

Inline Web Worker Tick Loop: When a lecturer switches focus from Chrome to VS Code or PowerPoint, Chromium throttles standard main-thread setInterval timers to 1 tick per minute. We instantiate an inline Blob Web Worker (worker.postMessage({ action: 'start', interval })) whose ticks are completely immune to background tab throttling.
1000ms ImageCapture.grabFrame() Watchdog & GPU Texture Release: We read frames directly from the hardware media track via reusableImageCapture.grabFrame(). However, on Windows/macOS, grabFrame() can hang indefinitely when the desktop compositor has zero damaged rectangles. We race grabFrame() against a 1000ms timeout—falling back to an attached 1920x1080 offscreen <video> element—and always call frameSource.close() in finally so ImageBitmap GPU textures never leak.
32×18 Downsampled RGB Delta: Before encoding a high-resolution JPEG, we draw the frame onto a tiny 32×18 canvas (576 pixels) and compare the R, G, B channels against lastDiffDataRef.current. We only encode and upload a full-resolution frame if more than 2 pixels changed by $>10$ intensity units (or every 2000ms as a keep-alive heartbeat).
Automatic Firestore 1 MB Ceiling Guard: Firestore documents have a strict 1,048,576 byte limit. If a complex 1440p IDE screenshot produces a Base64 JPEG string exceeding 950,000 characters, we dynamically step down the JPEG quality and canvas scale (0.85×) before calling setDoc:

// From web-app/src/hooks/useTeacherScreenBroadcast.js
// 1. Draw to low-res 32x18 thumbnail canvas to check RGB pixel delta
const diffCtx = diffCanvasRef.current.getContext('2d', { willReadFrequently: true });
diffCtx.drawImage(frameSource, 0, 0, 32, 18);
const currentDiffData = diffCtx.getImageData(0, 0, 32, 18).data;

let hasChanged = force || !lastDiffDataRef.current;
if (!hasChanged && lastDiffDataRef.current) {
  let diffPixels = 0;
  for (let i = 0; i < currentDiffData.length; i += 4) {
    if (
      Math.abs(currentDiffData[i] - lastDiffDataRef.current[i]) > 10 ||
      Math.abs(currentDiffData[i + 1] - lastDiffDataRef.current[i + 1]) > 10 ||
      Math.abs(currentDiffData[i + 2] - lastDiffDataRef.current[i + 2]) > 10
    ) {
      diffPixels++;
      if (diffPixels > 2) {
        hasChanged = true;
        break;
      }
    }
  }
}

// 2. Heartbeat emission every 2s so terminal cursors and newly joined viewers never freeze
const now = Date.now();
if (!hasChanged && now - frameStatsRef.current.lastEmittedTime > 2000) {
  hasChanged = true;
}

if (hasChanged) {
  // ... [Draw to targetWidth x targetHeight captureCanvas with bicubic smoothing] ...
  let quality = currentPreset.quality || 0.82;
  let jpegDataUrl = captureCanvasRef.current.toDataURL('image/jpeg', quality);

  // 3. Guard against Firestore 1MB document limit (1,048,576 bytes)
  const MAX_FIRESTORE_PAYLOAD_LEN = 950000;
  if (jpegDataUrl.length > MAX_FIRESTORE_PAYLOAD_LEN) {
    quality = Math.max(0.60, quality - 0.12);
    jpegDataUrl = captureCanvasRef.current.toDataURL('image/jpeg', quality);
    if (jpegDataUrl.length > MAX_FIRESTORE_PAYLOAD_LEN) {
      const safeCanvas = document.createElement('canvas');
      safeCanvas.width = Math.round(targetWidth * 0.85);
      safeCanvas.height = Math.round(targetHeight * 0.85);
      const safeCtx = safeCanvas.getContext('2d');
      safeCtx.imageSmoothingEnabled = true;
      safeCtx.imageSmoothingQuality = 'high';
      safeCtx.drawImage(captureCanvasRef.current, 0, 0, safeCanvas.width, safeCanvas.height);
      jpegDataUrl = safeCanvas.toDataURL('image/jpeg', 0.78);
    }
  }

  const seq = ++frameSeqRef.current;
  await setDoc(doc(db, `classes/${classId}/screenBroadcast/liveFrame`), {
    frameData: jpegDataUrl,
    frameSeq: seq,
    width: targetWidth,
    height: targetHeight,
    resolution: currentRes,
    timestamp: serverTimestamp(),
  });
  lastDiffDataRef.current = currentDiffData;
}

Architectural Decision: Why In-Document Base64 Over Cloud Storage Bucket Links

A frequent architectural question is: why write Base64 image payloads directly to a single Firestore document (liveFrame) instead of uploading .jpg files to a Google Cloud Storage bucket and broadcasting the download link?

While Cloud Storage is the standard choice for large assets and long recordings, four critical real-world requirements led us to choose in-document Base64 for live screen broadcasts:

Sub-Second Real-Time Push vs. Multi-Hop HTTP Roundtrips:
- With Firestore Base64: The teacher's browser pushes a single setDoc() call over an existing, persistent bidirectional WebSocket/gRPC channel. All connected student listeners receive the raw image data immediately in a single push (~200–400 ms latency).
- With Cloud Storage: The teacher would need to (1) upload the JPEG blob via uploadBytes(), (2) wait for upload confirmation, (3) write the storage URL to Firestore, and (4) every student browser would receive the metadata and then fire an additional HTTP GET request to download the image from the bucket. This doubles or triples latency (~1,000–1,500 ms) and introduces network race conditions where frame #42 might finish downloading after frame #43 over student mobile Wi-Fi.
Fine-Grained Dynamic Security Rules (Cross-Checking PINs & Enrollment):
- In firestore.rules, we can dynamically cross-check whether the student is currently enrolled in classes/{classId}, or if a guest's 4-digit PIN matches the active lecture session document.
- Firebase Storage rules cannot query Firestore documents (there is no get() in storage.rules). Storage rules have no way to verify whether a public PIN in a URL matches a dynamic session document in Firestore, nor can they verify real-time class roster membership without a dedicated backend token broker.
Instant, Atomic Session Revocation:
- The moment an instructor clicks Stop Broadcast, Firestore sets isBroadcasting: false. firestore.rules instantly denies further reads to all listeners across the entire cohort.
- In contrast, Storage public URLs (getDownloadURL) contain persistent bearer tokens (?alt=media&token=...) that remain readable on Google Cloud CDN caches even after the lecture ends, creating potential privacy leaks.
Ephemeral Single-Document In-Place Overwrite & Zero Long-Term Database Storage:
- Never Accumulates Historical Frame Documents: During a broadcast, frames are never appended as a growing collection of documents. Every new frame calls setDoc() on the exact same single document path (classes/{classId}/screenBroadcast/liveFrame), completely overwriting the previous frame in-place. Whether a lecture runs for 15 minutes or 3 hours, the total Firestore storage footprint is bounded to at most 1 frame (~150–350 KB) at any moment.
- Immediate null Wipe on Broadcast Stop: The instant the lecturer stops sharing, stopBroadcast() in useTeacherScreenBroadcast.js wipes frameData to null—leaving 0 bytes of image data stored in Firestore long-term and avoiding the tens of thousands of orphaned .jpg files that sequential Cloud Storage uploads (frame_001.jpg, frame_002.jpg) would leave behind:

// From web-app/src/hooks/useTeacherScreenBroadcast.js — Wiping frame payload on stop
const liveFrameDocRef = doc(db, `classes/${classId}/screenBroadcast/liveFrame`);
await setDoc(liveFrameDocRef, {
  frameData: null,
  endedAt: serverTimestamp(),
}, { merge: true });

2.2 FinOps & Cost Breakdown: Firestore Document Fanout at Classroom Scale

Because each frame overwrites the single liveFrame document in-place and is wiped to null when the session ends, long-term Firestore disk storage cost is $0.00. Runtime billing is strictly limited to document operations ($0.18 per 100K writes, $0.06 per 100K reads in asia-east2) and network egress (~$0.12 per GiB). Here is the empirical cost model for a typical 1-hour lecture (~500 emitted frames, leveraging 32×18 pixel-delta suppression when slides are static):

Workload Scale	Active Attendees	Firestore Writes (~500 Frames)	Firestore Reads	Network Egress (~450 KB/frame)	Total Google Cloud Cost (1-Hour Session)
Typical Computer Lab	30 Students	500 writes ($0.0009)	15,000 reads ($0.009)	~6.75 GB (Within 10 GB Free Tier)	~$0.01 USD
Standard Lecture Hall	60 Students	500 writes ($0.0009)	30,000 reads ($0.018)	~13.5 GB (~$0.42 after free tier)	~$0.44 USD
Multi-Cohort Joint Lecture	150 Students	500 writes ($0.0009)	75,000 reads ($0.045)	~33.7 GB (~$2.84)	~$2.89 USD
Peak Auditorium / Keynote	600 Students	500 writes ($0.0009)	300,000 reads ($0.180)	~135 GB (~$15.00)	~$15.18 USD

Why Serverless Wins Over Dedicated Media Streaming Servers (SFU)

Traditional WebRTC SFU / Media Server: Hosting dedicated media servers (e.g. Janus, Mediasoup, or LiveKit clusters on 2–4 c2-standard-4 Cloud Compute VMs) incurs a fixed baseline cost of $250 – $500 USD per month 24/7—even during nights, weekends, and semester breaks when no classes take place.
Our Serverless Firestore Pattern: When no lecture is broadcasting, infrastructure costs drop strictly to $0.00. For daily classroom teaching (30–60 students), a complete 1-hour broadcast costs less than half a dollar, scaling up to ~$15 for an occasional 600-student symposium without provisioning a single server.

Bypassing the 100/hr NAT Ceiling with Zero-Auth Firestore PIN Rules

To let multi-cohort student audiences join /live/:classId?pin=XXXX simultaneously from the same campus Wi-Fi NAT IP—and allow speakers to share the QR code in advance—we store both publicPin and savedPublicPin in classes/{classId}/screenBroadcast/session (configurePublicAccess in useTeacherScreenBroadcast.js) and validate the PIN directly inside firestore.rules with zero Firebase Auth calls:

// From firestore.rules
function isPublicBroadcastActive(classId) {
  let sessionPath = /databases/$(database)/documents/classes/$(classId)/screenBroadcast/session;
  return exists(sessionPath) && (
    (get(sessionPath).data.get('isPublic', false) == true &&
     get(sessionPath).data.get('publicPin', '') != '') ||
    (get(sessionPath).data.get('savedIsPublic', false) == true &&
     get(sessionPath).data.get('savedPublicPin', '') != '')
  );
}

match /screenBroadcast/{docId} {
  allow read: if isPublicBroadcastActive(classId) || isTeacher() || isTeacherInClass(classId) || isStudentInClass(classId);
  allow write: if isTeacher() || isTeacherInClass(classId);
}

match /screenBroadcastViewers/{studentUid} {
  allow create, update: if (
      request.auth != null && request.auth.uid == studentUid &&
      (isTeacher() || isTeacherInClass(classId) || isStudentInClass(classId))
    ) || (
      isPublicBroadcastActive(classId) &&
      request.resource.data.get('pin', '') != '' && (
        request.resource.data.get('pin', '') == get(/databases/$(database)/documents/classes/$(classId)/screenBroadcast/session).data.get('publicPin', '') ||
        request.resource.data.get('pin', '') == get(/databases/$(database)/documents/classes/$(classId)/screenBroadcast/session).data.get('savedPublicPin', '')
      )
    );
}

3. Crash-Tolerant 1-Minute Segmented Recording & The 5-Point Cloud Run FFmpeg Safety Gate

Recording a 3-hour lecture into a single in-memory browser Blob is a ticking time bomb: if the browser tab crashes or the laptop battery dies at minute 179, the entire 3-hour recording is lost.

Diagram 2: Resilient 1-Minute Rolling Ingestion & 5-Point Pre-Deletion Safety Gate

In useLectureRecorder.js and functions/media_processing/mergeLectureRecordings.js, we record rolling 1-minute self-contained .webm segments backed by an IndexedDB crash-recovery queue (lectureRecoveryDb.js), and merge them post-lecture inside a 2 vCPU, 2GiB RAM Cloud Run container guarded by a 5-Point Pre-Deletion Integrity Gate:

Production Code: Mixed Audio Alignment, 48kHz CBR MP3 Extraction, & Pre-Deletion Gate

What happens if a lecturer starts recording their screen with the microphone unplugged for the first two 1-minute clips, and plugs in their USB headset at minute 3? A standard ffmpeg -f concat -c copy command will either fail or slide the audio from minute 3 backward to minute 0, ruining lip/slide sync for the entire 3-hour video!

Here is how functions/media_processing/mergeLectureRecordings.js aligns mixed audio/no-audio clips using lavfi anullsrc, extracts a normalized 48kHz CBR MP3 for Gemini, and enforces the Pre-Deletion Integrity Gate:

// From functions/media_processing/mergeLectureRecordings.js
// 1. Stream alignment across clips: if some clips have audio and others do not,
// pad silent clips with a 48kHz stereo Opus track so FFmpeg concat preserves sync.
const clipsWithAudio = downloadedFiles.map((f) => currentAudioChecker(f));
const anyHasAudio = clipsWithAudio.some(Boolean);
const allHaveAudio = clipsWithAudio.every(Boolean);

const alignedFiles = [];
if (anyHasAudio && !allHaveAudio && !overrides.ffmpegRunner) {
  for (let i = 0; i < downloadedFiles.length; i++) {
    const filePath = downloadedFiles[i];
    if (!clipsWithAudio[i]) {
      const paddedPath = path.join(workDir, `clip_${i}_padded.webm`);
      const padRes = spawnSync(ffmpeg_static, [
        '-i', filePath,
        '-f', 'lavfi', '-i', 'anullsrc=r=48000:cl=stereo',
        '-c:v', 'copy', '-c:a', 'libopus', '-shortest', '-y', paddedPath,
      ]);
      if (padRes.status === 0 && fs.existsSync(paddedPath)) {
        alignedFiles.push(paddedPath);
        continue;
      }
    }
    alignedFiles.push(filePath);
  }
} else {
  alignedFiles.push(...downloadedFiles);
}

// 2. Concatenate videos via stream copy (-c copy) with monotonic DTS/PTS generation
const combinedVideoPath = path.join(workDir, 'combined_lecture.webm');
const concatCmd = ffmpeg()
  .input(concatListPath)
  .inputOptions(['-f concat', '-safe 0'])
  .outputOptions(['-c copy', '-avoid_negative_ts make_zero', '-fflags +genpts'])
  .output(combinedVideoPath);
await currentFfmpegRunner(concatCmd);

// 3. Extract normalized 48kHz Constant Bitrate MP3 for Gemini speech recognition
// Eliminates MediaRecorder WebM timestamp resets & prevents 1.48x subtitle drift!
const combinedMp3Path = path.join(workDir, 'combined_audio_normalized.mp3');
const audioNormalizedCmd = ffmpeg(combinedVideoPath)
  .noVideo()
  .audioCodec('libmp3lame')
  .audioBitrate('128k')
  .audioChannels(2)
  .audioFrequency(48000)
  .outputOptions(['-avoid_negative_ts make_zero', '-fflags +genpts'])
  .output(combinedMp3Path);
await currentFfmpegRunner(audioNormalizedCmd);

// ... [Uploads lecture.webm & lecture_audio_normalized.mp3 and commits Firestore doc] ...

// 4. Pre-Deletion Integrity Verification Gate:
// If ANY check fails, abort immediately so intermediate 1-min segments are NEVER deleted!
const destVideoFile = bucket.file(destVideoPath);
const [destExists] = await destVideoFile.exists();
if (!destExists) {
  throw new Error(`Integrity check failed: merged video '${destVideoPath}' does not exist in Cloud Storage. Segment deletion aborted.`);
}
const [metadata] = await destVideoFile.getMetadata();
const remoteSize = Number(metadata?.size || 0);
if (remoteSize > 0 && remoteSize < 1024) {
  throw new Error(`Integrity check failed: merged video is suspiciously small (${remoteSize} bytes). Segment deletion aborted.`);
}
if (durationSeconds <= 0 && videoStats.size < 1024) {
  throw new Error('Integrity check failed: merged video has 0 duration and invalid size. Segment deletion aborted.');
}

4. Single-Pass Audio-Only Whole-Lecture Subtitles with Gemini 4 Argon (`gemini-4-argon`)

Why send 1.2 GB of video frames into a multimodal LLM when closed captions and YouTube chapter markers come entirely from the lecturer's voice?

In ClassManagement.jsx and LectureRecordingsView.jsx, instructors can select their preferred lecture intelligence model—🧠 Gemini 4 Argon (gemini-4-argon), ⚡ Gemini 3.8 Flash (gemini-3.8-flash, Default), or 🚀 Gemini 3.5 Flash-Lite (gemini-3.5-flash-lite) (SUPPORTED_LECTURE_AI_MODELS in functions/ai_flows/config.js).

Diagram 3: Two-Stage Whole-Lecture Audio Pipeline, Timescale Calibration & YouTube Studio Packaging

Why We Split Whole-Lecture Subtitling into Stage 1 (Single-Pass Audio STT) + Stage 2 (Batched Translation)

When we initially asked the model to transcribe a 2-hour lecture and translate every sentence inline into 4 languages within a single JSON response, the output exceeded 65,536 completion tokens (maxOutputTokens) around minute 70, truncating the JSON!

In functions/ai_flows/processLectureSubtitles.js, we architected a Two-Stage Pipeline that handles up to 9-hour audio recordings without ever hitting the 65,536 output token ceiling:

Stage 1 — Single-Pass Whole-Audio Verbatim Transcription & Chaptering: We pass the normalized 48kHz CBR MP3 (gs://.../lecture_audio_normalized.mp3) to Gemini 4 Argon (gemini-4-argon) or Gemini 3.8 Flash (gemini-3.8-flash) with maxOutputTokens: 65536 and thinkingConfig: { thinkingBudget: 0 }. Because Stage 1 outputs only the original spoken cues and 4–10 YouTube chapter milestones, a full 3-hour lecture takes only ~18,000 output tokens and never cuts off on whiteboard pauses.
Stage 2 — Controlled-Concurrency Batched Translation: We slice the transcribed segments into batches of 50 sentences (BATCH_SIZE = 50, CONCURRENCY = 3) and translate each target language independently via buildLectureTranslationPrompt.

Solving Three Production Media & Subtitle Failure Modes

1. Fixing Long-Audio Timescale Drift (`normalizeLectureAudio` & `calibrateSubtitleTimeline`)

When Gemini processes long raw WebM Opus recordings in a single pass, WebM timestamp resets and internal audio framing cause the model's emitted timestamps to compress linearly (running ~1.48× to 1.68× faster than wall-clock video playback).

Our Fix: First, normalizeLectureAudio transcodes the audio track into a constant-bitrate 48kHz MP3 (-c:a libmp3lame -b:a 128k -ar 48000). Second, calibrateSubtitleTimeline compares the highest segment timestamp (rawMaxEnd) against the FFmpeg-probed totalDuration; if driftRatio > 1.05, it linearly stretches all segment start/end timestamps and chapter offsets into 1:1 synchronization with the video player:

// From functions/ai_flows/processLectureSubtitles.js
export function calibrateSubtitleTimeline(segments = [], chapters = [], totalDuration = 0) {
  if (!Array.isArray(segments) || segments.length === 0 || !totalDuration || totalDuration <= 0) {
    return { segments, chapters, driftRatio: 1.0 };
  }

  const rawMaxEnd = Math.max(...segments.map((s) => Number(s.end) || 0));
  if (rawMaxEnd <= 0 || totalDuration <= rawMaxEnd) {
    return { segments, chapters, driftRatio: 1.0 };
  }

  const driftRatio = totalDuration / rawMaxEnd;
  if (driftRatio <= 1.05) {
    return { segments, chapters, driftRatio: 1.0 };
  }

  const calibratedSegments = segments.map((seg) => ({
    ...seg,
    start: Math.round((Number(seg.start) || 0) * driftRatio * 1000) / 1000,
    end: Math.min(totalDuration, Math.round((Number(seg.end) || 0) * driftRatio * 1000) / 1000),
  }));

  const calibratedChapters = (Array.isArray(chapters) ? chapters : []).map((ch) => ({
    ...ch,
    timeSeconds: Math.min(Math.round(totalDuration), Math.round((Number(ch.timeSeconds) || 0) * driftRatio)),
  }));

  return { segments: calibratedSegments, chapters: calibratedChapters, driftRatio };
}

2. The Chromium `<track>` Zero-Duration Cue Drop Guard (`buildWebVTT` & `buildSRT`)

Occasionally, an LLM emits a subtitle cue where end == start (e.g., start: 142.5, end: 142.5). Both Chromium's native HTML5 <track> parser and YouTube Studio's .srt ingestion engine silently discard any cue where end <= start!

Our Fix: In buildWebVTT and buildSRT, we enforce a strict duration floor (end > start) proportional to the character length of the cue (Math.max(1.8, Math.min(5.0, text.length * 0.25))), clamped to the start of the next segment:

// From functions/ai_flows/processLectureSubtitles.js
export function buildWebVTT(segments = [], languageKey = 'original') {
  let vtt = 'WEBVTT\n\n';

  segments.forEach((seg, idx) => {
    const text = (
      languageKey === 'original'
        ? seg.original
        : seg.translations?.[languageKey] || seg.original || ''
    ).trim();

    if (!text) return;

    const start = Number(seg.start) || 0;
    let end = Number(seg.end) || 0;

    // Guard against 0 or negative duration cues that Chromium & YouTube discard:
    if (end <= start) {
      const nextSeg = segments[idx + 1];
      const nextStart = nextSeg ? Number(nextSeg.start) : Infinity;
      const minDisplayDur = Math.max(1.8, Math.min(5.0, text.length * 0.25));
      end = Math.min(nextStart > start ? nextStart : start + minDisplayDur, start + minDisplayDur);
    }

    vtt += `${idx + 1}\n`;
    vtt += `${formatTimestampToVTT(start)} --> ${formatTimestampToVTT(end)}\n`;
    vtt += `${text}\n\n`;
  });

  return vtt;
}

3. Fixing Unseekable `MediaRecorder` WebM Videos (`ensureIndexedLectureVideo`)

Files streamed directly from browser MediaRecorder lack an EBML SeekHead, Cues index table, and container Duration header—causing HTML5 <video> players to display 0:00 / Infinity and refuse to seek when a student clicks a chapter timestamp.

Our Fix: ensureIndexedLectureVideo runs a fast, zero-transcoding FFmpeg container remux (ffmpeg -y -i raw.webm -c copy indexed.webm) and stamps hasCuesIndex: 'true' into the Cloud Storage metadata so every lecture video seeks instantaneously.

5. Architectural Comparison: Legacy Video Broadcasting vs. Pixel-Delta & Audio-First Lecture Studio

Architecture / Pipeline Stage	Traditional WebRTC / Raw Video Approach	Our Gemini 4 / 3.8 Lecture Studio Architecture	Architectural Advantage
Live Screen Broadcast Transport	Peer-to-peer WebRTC video encoding per viewer	32×18 RGB Pixel-Delta + Firestore Single-Doc Fanout	Constant presenter workload regardless of viewer count
Campus Wi-Fi Student Onboarding	Blocked at 100 users/hr by Anonymous Auth NAT IP limit	Zero-Auth PIN Rules (`/live/:classId?pin=XXXX`)	Bypasses shared campus NAT IP throttling
Lecture Segment Merging	Full video transcoding on every merge	Zero-Transcode `ffmpeg -f concat -c copy` + 5-Point Safety Gate	Stream-copy concat without re-encoding video frames
Multimodal Input Payload to Gemini	Uploading full composite MP4/WebM video frames	Extracted 48kHz CBR MP3 Audio (`normalizeLectureAudio`)	Eliminates redundant video frame tokens for speech STT
Whole-Lecture Subtitle & Chapter Generation	Single monolithic prompt with video	Stage 1 Audio-Only STT + Stage 2 Batched Translation (50 cues/batch)	Prevents output token truncation on multi-hour lectures
Google Drive Archival	Proxying video buffers through Cloud Functions	Direct Browser $\rightarrow$ Google Drive OAuth Resumable Upload	$0.00 GCP Cloud Function Egress

6. Key Takeaways

Broadcast Slides & Code via Pixel-Delta Frames, Not Mesh WebRTC: A 32×18 RGB pixel-diffing canvas driven by an inline Web Worker timer and gated by Zero-Auth Firestore PIN rules scales 1080p/1440p classroom screen broadcasting across multi-seat cohorts at <2% presenter CPU.
Never Delete Raw Segments Without a Multi-Point Verification Gate: Recording in 1-minute rolling segments with IndexedDB recovery protects against browser crashes, while our 5-point Cloud Run FFmpeg check ensures intermediate clips are never purged unless the merged master video is verified healthy.
Send Pure Normalized 48kHz CBR MP3 to Gemini, Never Video Frames: Stripping video frames saves 90% of multimodal input tokens, while normalizing to 48kHz Constant Bitrate MP3 (normalizeLectureAudio) and running calibrateSubtitleTimeline eliminates the 1.48×–1.68× Opus timescale compression drift.
Decouple Verbatim Audio STT from Batched Multilingual Translation: Using Gemini 4 Argon (gemini-4-argon) or Gemini 3.8 Flash (gemini-3.8-flash) (thinkingBudget: 0, maxOutputTokens: 65536) for single-pass verbatim audio transcription + chaptering first, followed by 50-sentence batched translation and a zero-duration cue guard (end > start), guarantees 100% complete .vtt and YouTube Studio .srt packages for lectures up to 9 hours long.

Why Trucks Keep Hitting 11-Foot-8 Bridges: How I Built BridgeSpan AI with Gemma 4 & Google ADK to Stop Them

Robina Mirbahar — Tue, 10 Nov 2026 21:15:12 +0000

Author: Robina Mirbahar

AI/ML Google Developer Expert (GDE) | Google Cloud Champion Innovator | WTM Ambassador

Live Mobile-Responsive App: bridgespan-ai-mvhoucpfp2n5o25dewb2hi.streamlit.app
GitHub Open-Source Repo: github.com/RobinaMirbahar/bridgespan-ai
Kaggle Open Dataset (4,809 Bridges): BridgeSpan Bridge Strike, Load & Scour Dataset
Companion Kaggle Starter Notebook: BridgeSpan AI | 2024 FHWA NBI & Scour Starter

Everyone on the internet has watched viral videos of box trucks and semi-trailers getting their roofs peeled back like sardine cans by Durham's notorious 11-Foot-8+8 ("Can Opener") rail trestle, Boston's Storrow Drive, or Melbourne's Montague Street 3.0 m overpass.

Behind the memes sits a massive, expensive civil-engineering failure mode:

In the US and UK alone, thousands of low-clearance bridge strikes occur every year, halting commuter rail corridors above and costing millions in steel girder repairs.
Far less visible on camera, underwater hydraulic pier scour during flash-flood surges is the #1 cause of highway bridge collapse in the United States, accounting for over 50% of structural failures tracked in the US DOT FHWA National Bridge Inventory (Item 113) and HEC-18 Evaluating Scour at Bridges.
Meanwhile, heavy mobile cranes (42 t) and loaded semi-trailers (34.5 t) regularly cross historic trusses or stone-arch parkways rated for 15 t to 22.5 t.

If you ask a standard LLM navigation chatbot "Can my 13'-6\" truck clear that 3.89 m overpass?" or "Is Chain Bridge safe right now?", a single unit-conversion hallucination or ignored river-gauge surge turns into a physical catastrophe.

To solve this properly, I built and open-sourced BridgeSpan AI, a mobile-responsive global fleet and infrastructure safety platform powered by Google Gemma 4 (open weights), the Google Agent Development Kit (ADK), OSRM heavy-truck routing, and a 4,809-span real-world dataset assembled from the 2024 FHWA National Bridge Inventory, OpenStreetMap Overpass API, USGS Water Services, and Copernicus GloFAS.

Why Raw LLMs Fail on Physical Bridge Safety

When I tested unguarded prompts on real bridge-dispatch scenarios, three failure patterns appeared immediately:

Imperial (ft-in) vs. Metric (m) Arithmetic Traps: North American trailers declare height in feet and inches (13'-6" = 4.115 m), while urban underpasses in Europe, Australia, Azerbaijan, and Pakistan post signs in meters (3.95 m or 3.0 m). Safe civil clearance also requires a mandatory 0.15 m (6 in) dynamic suspension bounce and asphalt-overlay buffer. An LLM that compares 4.11 m against 4.15 m without the bounce buffer approves a roof strike at highway speed.
Silent Structural Overload: A 12'-0" mobile crane (42.0 metric tons) easily clears a 14'-2" historic steel truss vertically, so height-only checks wave it through even when the span's FHWA Item 064 Operating Rating is capped at 22.5 metric tons.
Blindness to Live River Pier Scour: During heavy rain or monsoon surges, a bridge deck looks dry and unobstructed on GPS maps while fast-moving water erodes riverbed sediment from beneath FHWA Item 113 scour-critical pier footings.

System Architecture: Gemma 4 + Deterministic 4-Gate Civil Interlocks

In safety-critical physical engineering, an LLM should orchestrate tools and explain telemetry, never hold unchecked authority over structural limits.

┌──────────────────────────────────────────────────────────────────────┐
│      BridgeSpan AI Mobile-Responsive Fleet Safety Command UI         │
│   8 Global Corridors (US • AU • AZ • PK) + Dark Leaflet OSRM Map     │
└──────────────────────────────────┬───────────────────────────────────┘
                                   │
                                   ▼
┌──────────────────────────────────────────────────────────────────────┐
│            Google ADK LlmAgent (Gemma 4 via LiteLLM)                 │
│  1. get_bridge_structural_profile() -> Clearance, Item 064, Item 113 │
│  2. get_live_river_scour_and_wind() -> USGS 15-min + GloFAS Gusts    │
│  3. compute_safe_bridge_dispatch()  -> Hard Civil Interlock Engine   │
└──────────────────────────────────┬───────────────────────────────────┘
                                   │
                                   ▼
┌──────────────────────────────────────────────────────────────────────┐
│          4-Gate Deterministic Civil Safety Interlocks (Python)       │
│  • GATE 1 (Clearance): parse_height_m + 0.15m bounce > posted_m      │
│  • GATE 2 (Axle Mass): gross_tons > FHWA Item 064 operating_tons     │
│  • GATE 3 (Pier Scour): Item 113 critical & surge >= +1.80m -> LOCK  │
│  • GATE 4 (Crosswind): high-sided & gust >= 65 km/h -> CLAMP 50km/h  │
└──────────────────────────────────┬───────────────────────────────────┘
                                   │
                                   ▼
┌──────────────────────────────────────────────────────────────────────┐
│     Live Overhead Highway LED Gantry Sign + Safe OSRM Truck Bypass   │
└──────────────────────────────────────────────────────────────────────┘

1. Killing the #1 Cause of Bridge Strikes: Deterministic `ft-in` to `m` Parsing

Instead of letting an LLM convert "13'-6\"" or "12'4\"" in token space, bridgespan_agent/guardrails.py parses every Imperial and Metric notation deterministically and adds the 0.15 m (6 in) dynamic suspension bounce margin before evaluating clearance:

import re

DYNAMIC_BOUNCE_BUFFER_M = 0.15  # Mandatory 6-inch suspension bounce & repaving buffer

def parse_height_to_meters(raw_height: str | float) -> float:
    """Deterministically parse '13'-6\"', '13ft 6in', '3.89 m', or '389 cm' into meters."""
    text = str(raw_height or "").strip().lower()

    imp_match = re.match(
        r"^\s*(\d+(?:\.\d+)?)\s*(?:'|ft|feet|foot)\s*[-\s]*(\d+(?:\.\d+)?)?\s*(?:\"|''|in|inch|inches)?\s*$",
        text,
    )
    if imp_match:
        feet = float(imp_match.group(1))
        inches = float(imp_match.group(2)) if imp_match.group(2) is not None else 0.0
        if inches >= 12.0:
            raise ValueError(f"Invalid inches ({inches}) in {raw_height!r}")
        return round((feet * 12.0 + inches) * 0.0254, 3)

    # Metric meters / centimeters handling ...

When a driver declares 13'-6" (4.115 m) approaching Durham's Gregson Street trestle (12'-4" / 3.76 m), the guardrail computes:

Effective Margin = 3.76 m - (4.115 m + 0.15 m bounce) = -0.505 m (STRIKE PREVENTED)

Gate 1 trips immediately (DIVERT_OVERHEIGHT_STRIKE_PREVENTED), updates the overhead LED highway gantry sign (OVERHEIGHT 13'-6" (4.12m) -> DIVERT RIGHT NOW), and reroutes the truck onto the verified OSRM high-clearance arterial bypass.

2. Enforcing Axle Mass, Pier Flood-Scour & Crosswind Rollover Clamps

Inside evaluate_bridge_interlocks(), all four civil-engineering invariants execute in strict safety priority order:

if scour_tripped:          # Item 113 scour-critical pier + >= 1.80 m river stage surge
    status = "CLOSE_BRIDGE_SCOUR_INTERLOCK"
    use_detour = True
elif height_tripped:       # Vehicle height + 0.15 m bounce > posted clearance
    status = "DIVERT_OVERHEIGHT_STRIKE_PREVENTED"
    use_detour = True
elif weight_tripped:       # Gross vehicle tons > FHWA Item 064 operating rating
    status = "DIVERT_OVERWEIGHT_LOAD_LIMIT"
    use_detour = True
elif wind_tripped:         # High-sided trailer + >= 65 km/h deck gust (e.g., Baku Khazri)
    status = "PROCEED_CLAMP_SPEED_50KMH"
    use_detour = False
    approved_speed = 50
else:
    status = "CLEAR_PRIMARY_ROUTE"

I also attached a Google ADK before_tool_callback (bridge_adk_before_tool_guardrail) directly to the LlmAgent so out-of-bounds tool arguments are sanitized before tool execution, and every final response is validated against the Pydantic v2 BridgeDispatchAdvisory schema.

3. Freight Decarbonization Impact (Carbone 4 & Sustainable Logistics Alignment)

Preventing bridge strikes and unscheduled corridor shutdowns is also a measurable freight-emissions win. As documented in heavy-transport decarbonization frameworks by Carbone 4 (Freight & Heavy Mobility Decarbonization) and the Smart Freight Centre GLEC Framework (ISO 14083):

A single low-clearance bridge strike triggers multi-hour highway and rail gridlock, forcing hundreds of diesel heavy-duty vehicles (~0.85 to 1.10 kg CO2e per vehicle-km) into stop-and-go idling.
Pre-trip deterministic OSRM routing onto a verified +1.07 km high-clearance arterial bypass eliminates catastrophic emergency backtracks (15+ km urban reversals), cranes, and steel girder replacement embodied carbon.
Pairing scale-to-zero serverless compute on Google Cloud Run with compact open-weights Gemma 4 models minimizes cloud inference energy overhead when no active dispatch queries are running.

4. Assembling a 100% Real Global Dataset (Zero Synthetic Toy Rows)

Rather than inventing fake bridges, I built an automated multi-source ingestion pipeline (collect_bridgespan_dataset.py) pulling from 5 authoritative public APIs and published both the dataset and a runnable starter notebook on Kaggle:

Dataset: BridgeSpan Bridge Strike, Load & Scour Dataset
Starter Notebook: BridgeSpan AI | 2024 FHWA NBI & Scour Starter

Table	Records	Real Public Source
`fhwa_nbi_2024_hazard_bridges.csv`	4,500	Official 2024 US DOT FHWA National Bridge Inventory (`NC`, `NY`, `MA`, `PA`, `DC`, `MD`) with exact clearances, tonnage ratings & Item 113 scour codes
`osm_global_low_clearance_bridges.csv`	309	OpenStreetMap Overpass API low-clearance spans across the US, UK, Australia, Pakistan, and Azerbaijan with verbatim `maxheight` tags
`usgs_live_river_scour_telemetry.csv`	1,436	USGS NWIS 15-minute instantaneous river stage (`00065`) & discharge (`00060`) time series
`global_glofas_flood_and_wind_daily.csv`	360	Copernicus GloFAS river discharge + Open-Meteo 10m crosswind gusts & rainfall across 8 global corridors
`bridgespan_dispatch_benchmark_80.csv`	80	Verified 80-scenario heavy-truck benchmark with real OSRM primary vs. safe bypass distances (`8/8` `pytest` suite verifies 100% agreement)

5. Designing a Mobile-Responsive Fleet Safety UI

Commercial dispatchers work on wide desktop monitors, while field supervisors and truck drivers check routes on phones. Taking inspiration from modern fleet telematics platforms, I designed app.py to be intuitive and mobile-responsive out of the box:

1-Tap Scenario Presets & Clean Vehicle Profiles: Test a 13'-6" semi-trailer, 42 t mobile crane, Potomac flood-scour surge, or 78 km/h Baku Khazri crosswind gale in a single tap.
At-a-Glance Progress Meters & Overhead Highway LED Gantry Sign: Visualizes exact clearance margin, axle load utilization, river surge, and OSRM bypass distance without raw debug JSON clutter.
Keyless Dark OpenStreetMap + OSRM Vector Map: Renders blocked bridges as dashed red lines and verified safe heavy-truck detours as bright emerald routes with zero paid map API keys required.

Try it live right in your browser or on your phone: bridgespan-ai-mvhoucpfp2n5o25dewb2hi.streamlit.app

6. Run Locally or Deploy Scale-to-Zero on Google Cloud Run

git clone https://github.com/RobinaMirbahar/bridgespan-ai.git
cd bridgespan-ai
pip install -r requirements.txt

# Verify all 80 benchmark scenarios & unit-conversion guardrails
pytest -v

# Launch the mobile-responsive operator console
streamlit run app.py

Deploying to Google Cloud Run with scale-to-zero ($0 idle cost) takes a single command:

gcloud run deploy bridgespan-ai \
  --source . \
  --region us-central1 \
  --allow-unauthenticated \
  --min-instances 0

Official Documentation & Standards Referenced

Google Gemma 4 Official Documentation: ai.google.dev/gemma/docs
Google Agent Development Kit (ADK) Docs & Callbacks: google.github.io/adk-docs
Google Cloud Run Serverless Containers: cloud.google.com/run/docs
US DOT FHWA National Bridge Inventory (NBI) Coding Guide: fhwa.dot.gov/bridge/nbi.cfm
FHWA HEC-18 Evaluating Scour at Bridges: fhwa.dot.gov/engineering/hydraulics
USGS National Water Information System (NWIS) Instantaneous Values API: waterservices.usgs.gov
Copernicus Emergency Management Service (GloFAS) & Open-Meteo Flood API: open-meteo.com/en/docs/flood-api
OpenStreetMap Overpass API (maxheight / maxweight tags): wiki.openstreetmap.org/wiki/Key:maxheight
OSRM HTTP Routing Engine API: project-osrm.org/docs/v5.24.0/api
Carbone 4 Low-Carbon Freight & Infrastructure Resilience: carbone4.com/en
Smart Freight Centre GLEC Framework (ISO 14083): smartfreightcentre.org

Explore All Project Links

Live Web App: bridgespan-ai-mvhoucpfp2n5o25dewb2hi.streamlit.app
GitHub Repository: github.com/RobinaMirbahar/bridgespan-ai
Kaggle Dataset: kaggle.com/datasets/robinamirbahar/bridgespan-global-bridge-strike-load-scour
Kaggle Starter Notebook: kaggle.com/code/robinamirbahar/bridgespan-ai-2024-fhwa-nbi-scour-starter

Google AI Classroom Part 5: Real-Time Multilingual Live Subtitles & Hong Kong Code-Switching with Gemini 4 / 3.8 & On-Device Gemma 4 E2B

Cyrus Wong — Tue, 10 Nov 2026 14:25:13 +0000

Author: Cyrus Wong (黃俊彥) — Google Developer Expert - Cloud (AI), Senior Lecturer at HKIIT

1. Problem Statement: Why Standard Speech-to-Text Breaks in Bilingual Technical Classrooms

In bilingual university and polytechnic classrooms across Hong Kong and Asia, lecturers rarely speak monolingual formal prose. Instead, we naturally code-switch between colloquial Cantonese, English, and domain-specific technical terminology within a single breath:

"大家留意呢個 React component，如果你喺 useEffect 入面唔記得 clean up 個 WebSocket listener，就會搞到 memory leak，甚至令個 Docker container OOM crash。"

When we tested standard commercial speech-to-text (STT) and machine translation pipelines on live Hong Kong lectures, they failed in three ways that actively confused students:

Homophone Hallucination: Generic STT engines phonetically force English programming keywords (useState, kubectl, DynamoDB, idempotent) into nonsensical Chinese characters (e.g., turning useState into "要死" or Docker into "多卡").
Colloquial Cantonese Calques: Standard translation APIs struggle with spoken Cantonese particles and idioms ("呢個", "點解", "搞掂", "睇下"), producing broken English or unnatural literal Mandarin translations.
Loss of Cross-Discipline Context: The word "port" means a TCP/UDP network port in Computer Science, a venous access port in Nursing & Healthcare, and a maritime harbor in Logistics. Without discipline-aware context and rolling conversational history, utterance-by-utterance translation loses pronoun referents ("it", "this hook") and domain accuracy.

To solve this for diverse international and mainland/local student cohorts, we engineered a Dual-Engine Real-Time Multilingual Live Subtitle Pipeline supporting 9 simultaneous target languages (zh-Hant, zh-Hans, en, ja, ko, es, fr, de, vi). Instructors can switch seamlessly between:

Server Mode (Recommended · Cross-Platform): Powered by Gemini 3.5 Flash-Lite (gemini-3.5-flash-lite), Gemini 3.8 Flash (gemini-3.8-flash), or Gemini 4 Argon (gemini-4-argon) via Genkit (translateTeacherSpeech in functions/ai_flows/subtitleFlows.js) with thinkingBudget: 0 for sub-second multi-language structured JSON generation.
Client Mode ($0.00 Cloud Cost · Desktop WebGPU/WASM): Powered by on-device LiteRT-LM Gemma 4 E2B (gemma-4-E2B-it-web.litertlm) running inside a dedicated Web Worker (litertGemma.worker.js) with automatic cloud fallback for zero dropped captions.

2. Architectural Deep-Dive: Dual-Engine Live Subtitle Pipeline

In TeacherSubtitleControlModal.jsx and useTeacherLiveSubtitles.js, instructors can toggle between Server Mode and Client Mode, change their spoken source language (zh-HK, en-US, zh-CN), or add/remove target languages on the fly without restarting their microphone stream.

Diagram 1: Dual-Engine Live Subtitle Architecture (Cloud Gemini 4/3.8/3.5 vs. $0.00 Edge Gemma 4 E2B)

2.1 How the Pipeline Captures and Translates Mixed Languages (Code-Switching)

A common question when designing multilingual live subtitles for bilingual classrooms is: how does the system actually handle mixed languages without breaking the ASR or the translation engine?

Our architecture splits this problem into two decoupled stages:

1. Ingestion: Capturing Spoken Mixed Speech (Cantonese + English Technical Terms)

In Hong Kong and international polytechnics, teachers speak Cantonese while using English for technical concepts ("大家留意呢個 React component，如果你喺 useEffect 入面唔記得 clean up 個 WebSocket listener...").

Primary Desktop Path (Web Speech API zh-HK): In useTeacherLiveSubtitles.js, Chromium's SpeechRecognition is initialized with recognition.lang = 'zh-HK'. Google's Hong Kong speech recognition acoustic models natively support conversational Cantonese interleaved with English technical loanwords and acronyms.
Audio Fallback Path (LiteRT Whisper & Cloud Gemini): When Web Speech is unavailable (e.g. specialized browsers or VAD silence flushes), 16kHz PCM audio buffers are transcribed via on-device LiteRT Whisper Web Worker or Firebase AI Logic (transcribeAudioWithFirebaseAI). The prompt instructs the engine to transcribe verbatim and retain technical vocabulary rather than forcing homophone substitutions.

2. Translation: Why LLM-Driven Translation Replaces Rule-Based MT for Code-Switching

Traditional machine translation engines (such as standard MT APIs) fail on code-switched technical speech because they expect grammatically formal, monolingual source text. When fed mixed Cantonese-English sentences:

They mistranslate colloquial Cantonese grammatical particles ("呢個", "點解", "搞掂", "睇下").
They attempt to literally translate programming identifiers (useEffect, useState, Docker, Kubernetes) into Chinese characters.
They lose pronoun referents ("佢" / "it") across short, choppy speech bursts.

To overcome this, our system forwards the raw bilingual transcript to an LLM (Cloud Gemini 3.5/3.8/4 or On-Device Gemma 4 E2B) configured with tailored system instructions:

Explicit Source Identification: Tells the model that the source is zh-HK containing colloquial Cantonese with English technical code-switching.
Preservation of Technical Terms: Explicitly instructs the model to preserve programming keywords, variable names, framework identifiers, and CLI commands in their original technical English form.
Colloquial-to-Formal Transformation: For Traditional Chinese (zh-Hant), transforms spoken Cantonese grammar into standard formal written Chinese (書面語); for English (en), translates the colloquial framing into fluent, idiomatic English rather than word-for-word calques.
Pronoun & Context Resolution via Rolling History: Injects the preceding 4 speech cues as read-only context (historyText), allowing the LLM to understand what "佢" or "this hook" refers to from the previous sentence.

3. Implementation Walkthrough: Code-Switching Translation with Cloud Gemini & Edge Gemma 4 E2B

3.1 Server Engine: `translateTeacherSpeech` with Rolling History & `thinkingBudget: 0`

When translating live speech in a lecture theatre, latency is paramount: if subtitles lag more than 1.5 seconds behind the lecturer's voice, students lose synchronization with the slide pointer.

In functions/ai_flows/subtitleFlows.js, translateTeacherSpeech minimizes multi-language translation latency while preserving technical terminology by combining three techniques:

Zero-Thinking Fast Path (thinkingConfig: { thinkingBudget: 0 }): Disables chain-of-thought reasoning tokens for direct lexical translation while keeping temperature: 0.1 for deterministic technical terminology preservation.
Rolling Conversational History (historyText): Passes the lecturer's preceding 4 utterances (historyBufferRef.current.slice(-4)) strictly as read-only context so the model resolves pronouns ("if we omit it" $\rightarrow$ "if we omit the dependency array") without re-translating old sentences.
Resilient Multi-Model Routing (generateWithResilience): Supports gemini-3.5-flash-lite (default high-frequency workhorse), gemini-3.8-flash, and gemini-4-argon (gemini-4-argon) with automatic region/model failover and per-class FinOps quota enforcement (checkQuota & logJob).

// From functions/ai_flows/subtitleFlows.js
export async function translateTeacherSpeech({
  classId,
  teacherUid,
  teacherEmail = '',
  text,
  sourceLang = 'zh-HK',
  targetLangs = ['zh-Hant'],
  context = '',
  customPrompt = '',
  historyText = null,
  model = null,
}) {
  const targetModel = model || AI_MODEL;
  const trimmedText = (text || '').trim();
  if (!trimmedText || isSubtitleNoiseOrRefusal(trimmedText)) {
    return { originalText: '', sourceLang, translations: {}, modelUsed: targetModel, cost: 0 };
  }

  const validTargets = Array.isArray(targetLangs) && targetLangs.length > 0
    ? targetLangs.filter((l) => Boolean(l && typeof l === 'string'))
    : ['zh-Hant'];

  const targetDesc = validTargets
    .map((code) => `${code} (${SUPPORTED_SUBTITLE_LANGUAGES[code] || code})`)
    .join(', ');

  const domainContext = context?.trim() || 'General Classroom Instruction';
  let historyBlock = '';
  if (Array.isArray(historyText) && historyText.length > 0) {
    const lines = historyText.map((l) => (typeof l === 'string' ? l.trim() : '')).filter(Boolean);
    if (lines.length > 0) {
      historyBlock = `\nPreceding Speech History (for conversational context, pronoun resolution, and terminology continuity only; DO NOT translate this section):\n${lines.map((l) => `- "${l}"`).join('\n')}\n`;
    }
  }

  const prompt = `You are a real-time lecture subtitle translator for higher education.
Context / Subject Matter: ${domainContext}
Spoken Source Language: ${sourceLang} (May include colloquial Cantonese and English code-switching)
Target Language(s) to produce: ${targetDesc}
${historyBlock}
Current Speech to Translate:
"${trimmedText}"

Guidelines:
1. FULL TRANSLATION MANDATORY: Translate the complete sentence and meaning accurately and naturally into each requested target language.
2. For English ("en"): Translate all spoken Chinese/Cantonese phrases into natural fluent English. NEVER leave untranslated Chinese characters or colloquialisms in the English translation.
3. For Simplified Chinese ("zh-Hans"): Translate Cantonese colloquialisms into standard written Simplified Chinese.
4. For Traditional Chinese ("zh-Hant"): Translate into standard written Traditional Chinese (書面語).
5. Technical Terms: Preserve domain-specific technical keywords, programming syntax, standard formula names, and brand names (e.g. "useEffect", "Firestore", "Docker", "Python", "Kubernetes") in their original technical spelling without unnatural literal translations.
6. Translate ONLY the "Current Speech to Translate", using the Preceding Speech History solely to infer context, resolve pronouns, and maintain terminology continuity.
7. Return a JSON mapping with the exact target language codes as keys.`;

  const translationsSchema = z.record(z.string(), z.string());

  const { response, modelUsed } = await generateWithResilience(
    {
      prompt,
      config: {
        temperature: 0.1,
        thinkingConfig: { thinkingBudget: 0 },
      },
      output: {
        schema: z.object({
          detectedSourceLang: z.string().optional(),
          translations: translationsSchema,
        }),
      },
    },
    targetModel
  );
  // ... [Normalizes language keys and logs exact token usage to classes/{classId}/aiCosts] ...
}

3.2 Client Engine: $0.00 On-Device `Gemma 4 E2B` in Web Worker with Streaming CacheStorage

For instructors running 3-hour daily lab sessions on desktop workstations who want $0.00 cloud AI expenditure, we integrated Gemma 4 E2B (gemma-4-E2B-it-web.litertlm) via @litert-lm/core inside web-app/src/workers/litertGemma.worker.js.

When the teacher clicks Download & Load Local Gemma 4 (~1.9 GB) in TeacherSubtitleControlModal.jsx, the worker streams the model weights with real-time progress reporting and caches the binary in the browser's persistent CacheStorage (webai-litert-gemma-v1), so subsequent lectures load from local SSD in under 2 seconds:

// From web-app/src/workers/litertGemma.worker.js
export function buildGemmaTranslationPrompt({
  transcript,
  sourceLang = 'zh-HK',
  targetLangs = ['en'],
  courseContext = '',
  customPrompt = '',
}) {
  const sourceName = langNames[sourceLang] || sourceLang;
  const targetList = targetLangs.map((code) => `- "${code}": ${langNames[code] || code}`).join('\n');
  const domainContext = courseContext?.trim() || 'higher education and classroom instruction';

  return `<start_of_turn>user
You are an expert real-time lecture translation assistant for ${domainContext}.
Translate the following spoken classroom transcript from ${sourceName} into the requested target languages:
${targetList}

CRITICAL RULES:
1. FULL TRANSLATION MANDATORY: Translate the entire meaning into each requested target language. For English ("en"), translate all spoken Chinese/Cantonese phrases into fluent natural English.
2. For Simplified Chinese ("zh-Hans" / "zh-CN"), translate into standard written Simplified Chinese. For Traditional Chinese ("zh-Hant"), translate into standard written Traditional Chinese.
3. Preserve discipline-specific terminology, proper nouns, formula/variable names, and standard technical abbreviations (e.g. "useEffect", "Docker", "Python") in their original technical form.
4. Return strictly a single valid JSON object mapping each target language code to its translated text.

Spoken transcript:
"${transcript}"
<end_of_turn>
<start_of_turn>model
`;
}

Crucially, in useTeacherLiveSubtitles.js, if on-device Gemma 4 E2B is still initializing, times out (>10s), or omits one of the requested target languages in its JSON output, translateTranscript automatically backfills only the missing language keys via the Cloud translateTeacherSpeech function—guaranteeing zero dropped captions:

// From web-app/src/hooks/useTeacherLiveSubtitles.js
const result = await gemmaPromise;
const gemmaTranslations = result?.translations || {};
const missingLangs = targetLanguages.filter((l) => !gemmaTranslations[l]);
if (missingLangs.length === 0) {
  return gemmaTranslations;
}

// Hybrid Partial Fallback: Only request missing languages from Cloud Gemini!
const callTranslate = httpsCallable(functions, 'translateTeacherSpeech');
const response = await callTranslate({
  classId,
  text: trimmed,
  sourceLang: speechLanguage,
  targetLangs: missingLangs,
  context: courseContext,
  customPrompt: customPromptText,
  historyText: recentHistoryStrings,
});
return { ...gemmaTranslations, ...(response.data?.translations || {}) };

4. Discipline Glossary Injection & Zero-Restart Dynamic Prompt Propagation

A single university deployment spans multiple faculties—from Computer Science & Software Engineering to Healthcare, Nursing & Medical Sciences, Business & Corporate Finance (EBITDA, IFRS), and Electrical & Mechanical Engineering.

Diagram 2: Discipline Glossary Injection & Zero-Restart Dynamic Prompt Propagation

When an instructor switches the Subject Domain or edits custom prompt instructions inside TeacherSubtitleControlModal.jsx (or ClassManagement.jsx), the update propagates to all active inference workers and student screens via Firestore onSnapshot with 0ms audio interruption:

4.1 Production Prompt Engineering Gallery: Code-Switching & Multi-Discipline Samples

To help educators and AI engineers adapt our subtitle pipeline to their own universities, below are the exact production prompt templates and assembled payloads from functions/ai_flows/subtitleFlows.js, web-app/src/constants/promptRegistry.js, and admin/prompts/translations/.

Sample 1: Full Assembled Server-Mode Prompt (`translateTeacherSpeech`) with 4-Turn Rolling History & 9-Language JSON Output

When a lecturer speaks in Hong Kong Cantonese mixed with English technical terms, translateTeacherSpeech injects the 4 most recent utterances (historyText) strictly for pronoun resolution ("佢" / "it" $\rightarrow$ useEffect cleanup function) and terminology continuity:

You are a real-time lecture subtitle translator for higher education.
Context / Subject Matter: Cloud Native Architecture, React Hooks & Container Orchestration (ITP4124)
Spoken Source Language: zh-HK (May include colloquial Cantonese and English code-switching)
Target Language(s) to produce: zh-Hant (繁體中文 (Traditional Chinese)), zh-Hans (简体中文 (Simplified Chinese)), en (English), ja (日本語 (Japanese)), ko (한국어 (Korean)), es (Español (Spanish)), fr (Français (French)), de (Deutsch (German)), vi (Tiếng Việt (Vietnamese))

Preceding Speech History (for conversational context, pronoun resolution, and terminology continuity only; DO NOT translate this section):
- "頭先我哋起咗個 Firestore onSnapshot listener 去 subscribe 個 liveSubtitles document。"
- "大家留意下，當個 student component unmount 嗰陣，個 WebSocket connection 其實仲未斷㗎。"
- "如果你喺 useEffect 入面唔記得 return 個 cleanup function 呢，"

Current Speech to Translate:
"佢就會一直食住個 browser memory，搞到 memory leak 甚至令個 container OOM crash 囉！"

Guidelines:
1. FULL TRANSLATION MANDATORY: Translate the complete sentence and meaning accurately and naturally into each requested target language.
2. For English ("en"): Translate all spoken Chinese/Cantonese phrases into natural fluent English. NEVER leave untranslated Chinese characters or colloquialisms in the English translation.
3. For Simplified Chinese ("zh-Hans"): Translate Cantonese colloquialisms into standard written Simplified Chinese.
4. For Traditional Chinese ("zh-Hant"): Translate into standard written Traditional Chinese (書面語).
5. Technical Terms: Preserve domain-specific technical keywords, programming syntax, standard formula names, and brand names (e.g. "useEffect", "Firestore", "Docker", "Python", "Kubernetes") in their original technical spelling without unnatural literal translations.
6. Translate ONLY the "Current Speech to Translate", using the Preceding Speech History solely to infer context, resolve pronouns (e.g. "it", "they", "this"), and maintain terminology continuity.
7. Return a JSON mapping with the exact target language codes as keys.

Structured JSON Response (gemini-3.5-flash-lite / gemini-4-argon with thinkingBudget: 0):

{
  "detectedSourceLang": "zh-HK",
  "translations": {
    "zh-Hant": "它便會持續佔用瀏覽器記憶體，導致 memory leak，甚至令該 container 發生 OOM crash！",
    "zh-Hans": "它就会一直占用浏览器内存，导致 memory leak，甚至让该 container 发生 OOM crash！",
    "en": "It will keep consuming browser memory, causing a memory leak or even triggering a container OOM crash!",
    "ja": "ブラウザのメモリを消費し続け、memory leakを引き起こし、さらにはcontainerのOOM crashを招いてしまいます！",
    "ko": "브라우저 메모리를 계속 점유하여 memory leak을 유발하고, 심지어 container OOM crash까지 발생시킬 수 있습니다!",
    "es": "¡Seguirá consumiendo memoria del navegador, provocando un memory leak o incluso un OOM crash del container!",
    "fr": "Il continuera à consommer la mémoire du navigateur, provoquant un memory leak voire un OOM crash du container !",
    "de": "Es belegt weiterhin Browser-Speicher, was zu einem memory leak oder sogar einem OOM crash des containers führt!",
    "vi": "Nó sẽ tiếp tục chiếm dụng bộ nhớ trình duyệt, gây ra memory leak hoặc thậm chí làm container bị OOM crash!"
  }
}

Sample 2: Cantonese-English Code-Switching Lecture Translator (`admin/prompts/translations/Cantonese-English Code-Switching Lecture Translator.md`)

Used when instructors want explicit transformation of Hong Kong spoken modal particles (㗎, 啦, 喎, 囉, 呢個, 點解, 搞掂) into formal written Chinese (書面語) and idiomatic English:

# Cantonese-English Code-Switching Lecture Translator

You are a bilingual simultaneous interpreter specializing in university classroom lectures delivered in colloquial Cantonese mixed with English technical jargon (Hong Kong higher education classroom context).

## Input Context
- Spoken Language: {{spokenLanguage}} (Spoken Cantonese / Hong Kong Colloquial Cantonese with English code-switching)
- Target Subtitle Language: {{targetLanguage}}
- Subject Domain: {{courseContext}}

## Spoken Speech Transcript
"""
{{speechText}}
"""

## Translation Guidelines
1. **Handle Code-Switching and Loanwords**:
   - Spoken Cantonese modal particles (e.g., 㗎, 啦, 喎, 囉, 呀, 嘅) and spoken colloquialisms (e.g., 呢個, 佢哋, 點解, 咁樣, 睇下) must be transformed into fluent, written grammar.
   - For Traditional Chinese (`zh-TW` / `zh-HK`) target: Translate into standard formal written Chinese (書面語), while retaining English technical terms where standard in Hong Kong industry.
   - For English (`en`) target: Provide clean, idiomatic English explanations without Cantonese grammatical calques.
2. **Technical Keyword Preservation**:
   - Retain industry acronyms, software names, and specialized domain terms in standard English (e.g., `pipeline`, `database migration`, `props`, `API endpoint`, `pull request`).
3. **Subtitling Pacing**:
   - Provide crisp, compact sentence segments that synchronize well with real-time video captions.

## Output Format
Return ONLY the translated sentence(s). No explanations, no pinyin/jyutping, and no markdown quotes.

Sample 3: Healthcare & Medical Sciences Clinical Lecture Translator (`admin/prompts/translations/Healthcare & Medical Sciences Clinical Lecture Translator.md`)

Used by nursing and health-science faculties so clinical dosages (mg/kg, b.i.d., q8h) and Hospital Authority (HA) medical abbreviations (BP, SpO2, ECG, PRN, STAT) are never mistranslated:

# Healthcare & Medical Sciences Clinical Lecture Translator

You are a certified medical interpreter and clinical lecture subtitle specialist assisting healthcare, nursing, and medical sciences students.

## Input Context
- Spoken Language: {{spokenLanguage}}
- Target Subtitle Language: {{targetLanguage}}
- Academic Discipline: Healthcare, Nursing & Clinical Sciences

## Spoken Speech Transcript
"""
{{speechText}}
"""

## Translation Guidelines
1. **Medical & Pharmacological Accuracy**:
   - Strictly preserve drug generic and brand names, pharmacological classifications, anatomical structures (ICD / MeSH standard), dosages (`mg/kg`, `mcg`, `b.i.d.`, `q8h`), and clinical abbreviations (`BP`, `SpO2`, `ECG`, `PRN`, `STAT`).
   - Do not approximate or mislabel clinical measurements or contraindications.
2. **Clinical Purity**:
   - If translating into Chinese: Use standard Hong Kong Hospital Authority (HA) / international clinical Chinese terminology.
   - If translating into English: Use standard GMC / AMA medical English.
3. **Pacing**:
   - Keep subtitle segments readable in under 2 seconds per line.

## Output Format
Return ONLY the translated clinical sentence without surrounding commentary or quotation marks.

Sample 4: Business & Financial Accounting Lecture Translator (`admin/prompts/translations/Business & Financial Accounting Lecture Translator.md`)

Used in corporate finance and accounting lectures to preserve IFRS, HKFRS, EBITDA, WACC, and CAPEX distinctions:

# Business & Financial Accounting Lecture Translator

You are an expert financial translator and business academic interpreter.

## Input Context
- Spoken Language: {{spokenLanguage}}
- Target Subtitle Language: {{targetLanguage}}
- Academic Discipline: Business, Finance & Accounting

## Spoken Speech Transcript
"""
{{speechText}}
"""

## Translation Guidelines
1. **Financial & Accounting Terminology Preservation**:
   - Preserve accounting standards (`IFRS`, `GAAP`, `HKFRS`), financial metrics (`EBITDA`, `ROE`, `CAPEX`, `WACC`, `DCF`, `P/E ratio`), and balance sheet classifications.
   - Do not confuse revenue, net profit, gross margin, or cash flow terminologies.
2. **Corporate & Economic Terminology**:
   - Use standard corporate finance and capital markets nomenclature for target languages.
3. **Pacing & Conciseness**:
   - Present translated subtitles cleanly for continuous multi-speaker business presentations and case studies.

## Output Format
Return ONLY the translated sentence(s). No markdown formatting, bullet points, or extraneous text.

Sample 5: Engineering & Applied Sciences Lecture Translator (`admin/prompts/translations/Engineering & Applied Sciences Lecture Translator.md`)

Used in electrical, civil, and mechanical engineering labs to preserve SI units (kW, kV, pF, MHz), dimensional tolerances (±0.05mm), and CAD/BIM terminology:

# Engineering & Applied Sciences Lecture Translator

You are an expert technical translator specializing in engineering, manufacturing, and applied sciences lectures.

## Input Context
- Spoken Language: {{spokenLanguage}}
- Target Subtitle Language: {{targetLanguage}}
- Academic Discipline: Engineering & Construction

## Spoken Speech Transcript
"""
{{speechText}}
"""

## Translation Guidelines
1. **Engineering Precision**:
   - Strictly preserve SI metric units, electrical notations (`kW`, `kV`, `pF`, `MHz`), tolerance symbols (`±0.05mm`), mechanical formulas, and CAD/BIM terminology.
   - Retain standard international engineering terms in English when used in global engineering practice.
2. **Clarity**:
   - Ensure safety protocols, machinery operational procedures, and mathematical steps are translated with zero ambiguity.

## Output Format
Return ONLY the direct translated subtitle text without markdown or commentary.

Sample 6: Client-Mode On-Device `Gemma 4 E2B` Prompt & Multimodal Audio Fallback STT Prompt

When running in Client Mode (litertGemma.worker.js), or when local Web Speech API / WASM Whisper encounters low-confidence audio and falls back to transcribeAudioWithFirebaseAI (Real-Time Speech-to-Text Transcriber.md), the exact prompts are:

# 1. On-Device LiteRT-LM Gemma 4 E2B Turn Prompt (web-app/src/workers/litertGemma.worker.js)
<start_of_turn>user
You are an expert real-time lecture translation assistant for Cloud Computing & DevOps.
Translate the following spoken classroom transcript from Cantonese (Hong Kong) into the requested target languages:
- "zh-Hant": Traditional Chinese
- "en": English
- "ja": Japanese

CRITICAL RULES:
1. FULL TRANSLATION MANDATORY: Translate the entire meaning into each requested target language. For English ("en"), translate all spoken Chinese/Cantonese phrases into fluent natural English.
2. For Simplified Chinese ("zh-Hans" / "zh-CN"), translate into standard written Simplified Chinese. For Traditional Chinese ("zh-Hant"), translate into standard written Traditional Chinese.
3. Preserve discipline-specific terminology, proper nouns, formula/variable names, and standard technical abbreviations (e.g. "useEffect", "Docker", "Python") in their original technical form.
4. Return strictly a single valid JSON object mapping each target language code to its translated text.

Spoken transcript:
"大家用 terraform apply 之前，記住行一次 terraform plan 睇下個 state diff 先。"
<end_of_turn>
<start_of_turn>model
{"zh-Hant":"大家在使用 terraform apply 之前，請記得先執行一次 terraform plan 查看 state diff。","en":"Before running terraform apply, everyone remember to run terraform plan first to inspect the state diff.","ja":"terraform apply を実行する前に、まず terraform plan を一度実行して state diff を確認してください。"}

# 2. Multimodal Audio Fallback Verbatim STT Prompt (transcribeAudioWithFirebaseAI)
You are a real-time classroom speech-to-text transcriber for Cloud Computing & DevOps.
The spoken language is zh-HK (with mixed domain and technical terms).
Transcribe the spoken audio verbatim. Keep discipline-specific terms, proper nouns, and technical vocabulary accurate for Cloud Computing & DevOps.
CRITICAL RULES:
- Output ONLY the exact words spoken in the audio.
- If the audio contains only silence, background noise, breathing, or no clear human speech, return an EMPTY string "".
- NEVER ask for audio, NEVER explain that no speech was detected, and NEVER add notes, quotes, or commentary.

5. Edge Cases & Production Failure Modes We Solved

Building a demo speech-to-text app takes an afternoon; making it survive 3-hour lectures across Windows, macOS, iPadOS, and Android hardware in noisy lecture halls required solving five subtle production bugs:

Diagram 3: Out-of-Tab Microphone Silence Watchdog & Mobile iOS Jetsam OOM Guard

Edge Case 1: The Mobile iOS WebKit 1.9 GB `Jetsam OOM` Crash Guard

iOS and iPadOS browsers enforce a strict per-tab WebContent memory ceiling (~1.4 GB–2.0 GB). If an instructor opens the Teacher Dashboard on an iPad Pro and the browser attempts to auto-download and instantiate the ~1.9 GB gemma-4-E2B-it-web.litertlm model in WASM/WebGPU memory, iOS WebKit immediately kills and reloads the page (Jetsam OOM).

Our Fix: In useTeacherLiveSubtitles.js, isMobileOrIosBrowser() detects iOS, Android, and iPadOS (navigator.platform === 'MacIntel' && navigator.maxTouchPoints > 1) and automatically locks engineMode to 'server', while on desktop browsers Gemma 4 E2B is never downloaded automatically—it requires an explicit teacher click on loadGemmaModel().

Edge Case 2: Whisper Silence Hallucinations (`"Thank you for watching"`, `"♪"`, Foreign Script Artifacts)

When a lecturer pauses for 5 seconds to write on the whiteboard, whisper-tiny frequently hallucinates YouTube training artifacts such as "Thank you.", "Thanks for watching.", "[BLANK_AUDIO]", or random Korean/Cyrillic/Arabic characters due to background HVAC fan noise. Worse, if a non-empty hallucination string is returned, a naive pipeline treats it as valid speech and skips the Cloud AI audio transcription fallback!

Our Fix: Both client (isWhisperNoiseArtifact in useTeacherLiveSubtitles.js) and server (isSubtitleNoiseOrRefusal in functions/ai_flows/subtitleFlows.js) strip these exact hallucination patterns and out-of-script Unicode blocks ([\uAC00-\uD7AF\u0600-\u06FF...]) before deciding whether to invoke transcribeAudioWithFirebaseAI or publish to Firestore:

// From web-app/src/hooks/useTeacherLiveSubtitles.js
export function isWhisperNoiseArtifact(text, speechLanguage = 'zh-HK') {
  const trimmed = (text || '').trim();
  if (!trimmed) return true;
  if (/^(\[.*\]|\(.*\)|♪+.*♪*|thank you\.?|thanks for watching\.?|you\.?)$/i.test(trimmed)) {
    return true;
  }
  if (
    /please provide the audio|audio recording or transcript|no speech detected|no clear speech|inaudible|ambient room noise|nothing to transcribe|blank_audio|^transcription\s*:/i.test(
      trimmed
    )
  ) {
    return true;
  }
  const normalizedLang = String(speechLanguage || 'zh-HK').toLowerCase();
  if (normalizedLang.startsWith('zh') || normalizedLang.startsWith('en')) {
    // Reject unexpected Hangul, Arabic, Hiragana/Katakana, Thai, or Cyrillic noise tokens
    if (/[\uAC00-\uD7AF\u1100-\u11FF\u0600-\u06FF\u3040-\u30FF\u0E00-\u0E7F\u0400-\u04FF]/.test(trimmed)) {
      return true;
    }
  }
  return false;
}

Edge Case 3: Out-of-Order Async Network Responses & Parallel STT Race Conditions

When running hybrid Web Speech API + WASM Whisper VAD in parallel, two race conditions happen:

Web Speech API finalizes an utterance in 150ms, and 4 seconds later the local WASM Whisper worker finishes processing the same PCM buffer and overwrites the newer subtitle.
Utterance #14 (a long 25-word sentence) takes 800ms to translate in the cloud, while Utterance #15 (a short 3-word phrase) finishes in 300ms. When Utterance #14's HTTP response arrives late, it overwrites Utterance #15 on student screens!
Our Fix:
- Whenever recognition.onresult emits a final transcript, it sets lastSpeechRecFinalAt = Date.now() and calls clearPendingVadBuffer(), suppressing any trailing Whisper VAD flush within 2.5 seconds.
- In publishSubtitle, we assign a monotonically increasing sequence number const seq = ++seqCounterRef.current before awaiting translateTranscript(). When the translation resolves, if seq < latestPublishedSeqRef.current, we discard the stale write from touching liveSubtitles/current while still preserving it in recentHistory sorted by seq.

Edge Case 4: iOS Safari WebKit `AudioContext.resume()` Deadlock & Zero-Gain Pull Node

In Chrome, Safari, and iOS WebKit, calling await audioCtx.resume() inside an async getUserMedia callback when transient user activation has expired returns a permanently pending Promise that deadlocks audio setup forever. Furthermore, on iOS WebKit, a MediaStreamAudioSourceNode connected only to an AnalyserNode or AudioWorklet will refuse to pull PCM frames unless the graph terminates at audioCtx.destination.

Our Fix: In useTeacherLiveSubtitles.js and TeacherSubtitleControlModal.jsx, we never await audioCtx.resume(); instead we fire-and-forget tryResumeAudioCtx() alongside passive pointerdown/keydown listeners, and connect the analyser to audioCtx.destination through a 0-gain silent GainNode (silentGain.gain.value = 0) so WebKit pulls hardware audio frames without causing speaker feedback.

Edge Case 5: The Wireless Mic Battery Trap (`isMicSilentWarning` Watchdog)

Lecturers frequently minimize the browser and spend 40 minutes coding in VS Code or presenting fullscreen PowerPoint slides. If their wireless lapel microphone battery dies or they accidentally hit the hardware mute switch on their headset, they have no idea that student captions have stopped.

Our Fix: A 2-second watchdog in useTeacherLiveSubtitles.js monitors both hardware track events (primaryAudioTrack.muted, enabled === false, readyState === 'ended') and RMS voice activity timestamps (lastVoiceActivityAtRef.current). If the hardware track is muted for $\ge 10\text{s}$ or zero speech energy is detected for $\ge 45\text{s}$, isMicSilentWarning triggers a high-visibility alert and native OS notification on top of fullscreen PowerPoint.

6. Architectural Comparison: Server Gemini Models vs. Edge Gemma 4 E2B

Here is how the dual-engine subtitle options compare across execution mode, model configuration, and device requirements:

Translation Engine	Model Identifier	Generation Configuration	Browser Memory Footprint	Cloud Token Cost (`cost.js`)
Server Mode (Default Workhorse)	`gemini-3.5-flash-lite`	`temperature: 0.1`, `thinkingBudget: 0`	Minimal (Stateless HTTPS Callable)	`$0.30` / `$2.50` per 1M tokens
Server Mode (High-Nuance Flash)	`gemini-3.8-flash`	`temperature: 0.1`, `thinkingBudget: 0`	Minimal (Stateless HTTPS Callable)	`$0.75` / `$3.75` per 1M tokens
Server Mode (Flagship Reasoning)	`gemini-4-argon`	`thinkingBudget: 0` (Legacy sampling keys stripped)	Minimal (Stateless HTTPS Callable)	`$2.00` / `$10.00` per 1M tokens
Client Mode (Edge WebGPU/WASM)	`gemma-4-E2B-it-web.litertlm`	On-Device Web Worker + Hybrid Partial Cloud Fallback	~1.9 GB (Cached in `CacheStorage`, Desktop Only)	$0.00 (for all locally translated languages)

7. Key Takeaways

Decouple STT from Multilingual Translation: Running lightweight hybrid STT on the edge (Web Speech API + LiteRT Whisper + noise artifact filtering) and sending clean text strings with a 4-turn rolling history to translateTeacherSpeech eliminates server audio-slicing overhead and keeps latency under 500ms.
Use thinkingBudget: 0 for Real-Time Captioning: When invoking Gemini 4 Argon (gemini-4-argon), Gemini 3.8 Flash, or Gemini 3.5 Flash-Lite for live subtitles, setting thinkingConfig: { thinkingBudget: 0 } and temperature: 0.1 delivers flagship lexical accuracy without reasoning-token latency spikes.
Hybrid Edge + Cloud Beats Pure Edge: On-device Gemma 4 E2B reduces cloud FinOps cost to $0.00 on desktop laptops, while automatic partial fallback to translateTeacherSpeech ensures mobile iPads never crash from WebKit memory limits and no target language is ever left blank.
Guard Every Browser Audio Quirk: Non-blocking AudioContext.resume(), 0-gain destination nodes on WebKit, Android SpeechRecognizer mic-seizure avoidance, and monotonic sequence numbers (seqCounterRef) are essential for rock-solid classroom reliability.

Zero Trust GKE Control Plane Access with DNS Endpoints

Azidin Shairi — Tue, 10 Nov 2026 08:18:33 +0000

Secure Your GKE Control Plane: Migrating from IP-Based Authorized Networks to IAM-Based DNS Endpoint Access

The GKE DNS-based endpoint provides a cluster-unique FQDN (Fully Qualified Domain Name) for the Kubernetes control plane, allowing for simplified, IAM-based access control instead of relying on static IP-based authorized networks. This guide outlines the steps to enable this feature and transition your access methods.

The DNS-Based Endpoint offers several advantages over the IP-Based Endpoint with Authorized Networks for accessing Google Cloud clusters. While IP-based access relies on source IP CIDR blocks and often requires bastion hosts or VPN for private clusters, the DNS-based approach uses Google Cloud IAM and is accessible from any network that can reach Google Cloud APIs. This provides a more flexible and dynamic access control model, integrated with IAM and VPC Service Controls for enhanced security, compared to the primarily network-layer security of IP-based endpoints.

This guide walks through the steps to enable the DNS endpoint on an existing GKE cluster and migrate the access to it. We'll use a GKE cluster named gke-test-cluster-1, deployed in the GCP Finland region, zone europe-north1-a.

Step 1: Enable the DNS-Based Endpoint

You can enable the DNS-based endpoint on an existing GKE Standard or Autopilot cluster using the gcloud CLI.

First, let's review the GKE Control Plane configuration:

gcloud container clusters describe gke-test-cluster-1 \
--zone europe-north1-a --format="yaml(controlPlaneEndpointsConfig)"

The output should look like this:

controlPlaneEndpointsConfig:
  dnsEndpointConfig:
    allowExternalTraffic: false
    enableK8sCertsViaDns: false
    enableK8sTokensViaDns: false
    endpoint: gke-CLUSTER_ID-PROJECT.europe-north1-a.gke.goog
  ipEndpointsConfig:
    authorizedNetworksConfig:
      cidrBlocks:
      - cidrBlock: ALLOWED_IP_ADDR
        displayName: office-network
      enabled: true
      gcpPublicCidrsAccessEnabled: true
    enablePublicEndpoint: true
    enabled: true
    privateEndpoint: PRV_IP_ADDR
    publicEndpoint: PUB_IP_ADDR

This is the current GKE Control Plane configuration. The IP-based Endpoint is enabled for both Public and Private IPs and Authorized Network is also enabled allowing "office-network" IP addresses.
The GKE DNS Endpoint FQDN follows this format:

gke-CLUSTER_ID-PROJECT.europe-north1-a.gke.goog

However, it is currently disabled.

So let's update the cluster to enable DNS-based access.

gcloud container clusters update gke-test-cluster-1 \
--zone europe-north1-a --enable-dns-access

It may take a few minutes for the GKE Control Plane to update with DNS Endpoint.
Once the operation is completed, you can run gcloud container clusters describe command again and should see allowExternalTraffic is enabled for dnsEndpointConfig .

Now, both IP-based and DNS-based endpoints are enabled. Both coexist at this point while we work on the next steps migrating the access from IP-based to DNS-based.

Step 2: Configure IAM Access

Access to the DNS endpoint is controlled by IAM. The specific IAM permission required for DNS-based access is container.clusters.connect
The permission container.clusters.connect is included in many predefined IAM roles such as "Kubernetes Engine Admin" and "Kubernetes Engine Developer". It's most likely your IAM user already has the permission through existing IAM roles, otherwise you were not able to update the GKE cluster in the previous step.

For other users however, your fellow developers, CI/CD pipelines and such might use other IAM roles or custom roles. In this case you need to update those roles to include the IAM permission.

As an example here if you need to add another GKE admin user, you can grant the IAM role as follow:

gcloud projects add-iam-policy-binding $PROJECT \
--member="user:$GKE_ADMIN" --role="roles/container.admin"

Step 3: Test and Migrate Client Access

Now that the GKE Control Plane DNS Endpoint is enabled and IAM roles are configured, it's time to update clients to use the FQDN.
To make use of the DNS Endpoint in kubectl you need to refresh the kubeconfig file by fetching the new GKE cluster's credentials:

gcloud container clusters get-credentials gke-test-cluster-1 \
--zone europe-north1-a --dns-endpoint

Take note of the --dns-endpoint parameter. This updates the kubeconfig and replaces the configuration for accessing your GKE Control Plane using FQDN instead of public IP address.
You can view the kubeconfig configuration using command:

kubectl config view --minify
Output:
apiVersion: v1
clusters:
- cluster:
    server: https://gke-CLUSTER_ID-PROJECT.europe-north1-a.gke.goog
  name: gke_PROJECT_europe-north1-a_gke-test-cluster-1

Now, run your favourite kubectl command as usual like kubectl get namespaces. If the command executed successfully it means our setup now works correctly.

Once you verified it's working, it's time to update all your CI/CD tools, pipelines to include --dns-endpoint parameter when fetching GKE cluster credentials, and inform your fellow GKE admins and developers to refresh their GKE cluster credentials.

Step 4: Disable IP-Based Authorized Networks

After you have successfully migrated all clients to using DNS Endpoint you can safely remove the IP-based authorized network along with the IP-based Endpoint. This helps keep your GKE clusters configuration clean.

There are 2 options here:

Disabling the Public IP Endpoint which removes the Public IP address from your GKE Control Plane.
Disable IP-based Endpoint completely which removes both Public and Private IP Endpoint. If you still have some other dependency on the Private IP Endpoint, you can just remove the Public IP Endpoint which improves your Control Plane security by not exposing it directly to the public.

Please note that the DNS Endpoint also works from your VPC.
Nonetheless to disable the Public IP Endpoint while keeping the Private IP Endpoint you can update the GKE cluster as follow:

gcloud container clusters update gke-test-cluster-1 \
--zone europe-north1-a --enable-private-endpoint

To disable IP-based Endpoint completely you can disable both Public IP and Private IP Endpoints as follow:

gcloud container clusters update gke-test-cluster-1 \
--zone europe-north1-a --no-enable-ip-access

Your GKE Control Plane Endpoint config will look like this after the clean up.

controlPlaneEndpointsConfig:
  dnsEndpointConfig:
    allowExternalTraffic: true
    enableK8sCertsViaDns: false
    enableK8sTokensViaDns: false
    endpoint: gke-CLUSTER_ID-PROJECT.europe-north1-a.gke.goog
  ipEndpointsConfig:
    enabled: false

With GKE Control Plane access now managed by IAM, you can use VPC Service Controls to add Context-Aware Access and further secure your GKE clusters.

Google AI Classroom Part 4: Defeating Proxy Attendance & Lab Keyloggers with WebAuthn Passkeys, Rotating HMAC QRs & GCIP Blocking Functions

Cyrus Wong — Tue, 10 Nov 2026 07:25:07 +0000

Author: Cyrus Wong (黃俊彥) — Google Developer Expert - Cloud (AI), Senior Lecturer at HKIIT

1. Problem Statement: The Proxy Account Sharing, "Fake Screen-Recording" & Keylogger Crisis in University Computer Labs & 200-Seat Lecture Theatres

When running hands-on programming classes and invigilated practical assessments across physical computer labs and 200-seat lecture halls at the Hong Kong Institute of Information Technology (HKIIT / VTC), our engineering team encountered three real-world identity and physical-presence exploits that traditional username-and-password authentication and campus IP checks cannot stop:

The "Give My Password to a Classmate" Proxy Lab Exploit (Fake Video Playback & Proxy Bingo): Even when campus CIDR IP restrictions (checkipaddress) block off-campus logins, an absent student simply texts their email and password to a friend sitting inside the computer lab. That single classmate sits in front of two adjacent lab workstations (or opens a second browser profile), logs into the absent student's account from a legitimate lab IP, starts screen sharing, and plays a pre-recorded coding video on loop in full-screen to fake live lab activity! Whenever a Classroom Bingo active-presence quiz pops up on that second monitor, the classmate simply reaches over and clicks an answer on their friend's behalf.
The "WhatsApp Proxy Attendance" Relay in 200-Seat Lecture Halls: When instructors project a static attendance QR code or 4-digit check-in code on the lecture theatre screen, a student sitting in the front row snaps a photo and forwards it to a 150-member WhatsApp or Discord group. Within five seconds, absent students still in bed—or sitting in the campus cafeteria—mark themselves "present."
The Shared Lab PC Keylogger Risk: Public university computer labs are high-turnover environments. Typing institutional passwords—especially an instructor's privileged account that can trigger high-quota Gemini 4 Argon (gemini-4-argon) cohort-wide video analysis or modify exam policies—on untrusted shared USB keyboards exposes faculty and students to both hardware inline keyloggers and resident TSR malware.

Why is a password or campus IP check completely insufficient here? Because a password can be texted over WhatsApp in 2 seconds, and a lab PC next to your friend shares the exact same campus subnet IP—whereas a student will never hand over their personal unlocked smartphone (Apple Secure Enclave / Android Titan M2 with Face ID, personal WhatsApp, and mobile banking) to a classmate for 3 hours!

To defeat all three exploits simultaneously without requiring dedicated biometric turnstiles or proprietary native mobile apps, we engineered a zero-trust, hardware-bound identity layer built on three pillars:

FIDO2 / WebAuthn Hardware-Bound Mobile Passkeys (@simplewebauthn/server) enforcing a strict 1-Phone = 1-Student Hardware Lock: One physical smartphone can only ever be bound to one student account (credentialID + deviceFingerprint). Even if a classmate logs into a second lab PC using a friend's shared password and plays a fake screen recording, they cannot pass the Mobile Passkey Bingo challenge or the Lab PC Passkey Enforcement Gate because their own phone is already cryptographically locked to their own student UID!
Google Cloud Identity Platform (GCIP) Blocking Functions (beforeUserCreated & beforeUserSignedIn) enforcing a 4-Tier Deterministic Role Hierarchy and schedule-aware Campus Lab CIDR IP Gating.
15-Second Rotating HMAC-SHA256 QR Codes for both 1.8-second Lecture Hall Biometric Roll Call and Zero-Keystroke Passwordless Lab PC Login, paired with a 3-Tier Emergency Bypass & Audit System.

2. Architectural Deep-Dive: 4-Tier Hybrid Role Resolution & Schedule-Aware CIDR Gating

On multi-campus educational domains, distinguishing faculty from students at the exact moment of registration is critical. In our platform, a user with role: 'teacher' can orchestrate Gemini 4 Argon (gemini-4-argon) multimodal grading jobs, inspect live student screen feeds, and reset hardware passkeys, whereas role: 'student' is strictly scoped to submitting their own telemetry and viewing authorized course materials. An accidental privilege escalation is both a data-privacy violation and a FinOps risk.

Diagram 1: 4-Tier Deterministic Role Hierarchy & GCIP Blocking Gate

In functions/auth_triggers/userManagement.js, functions/auth_triggers/ipRestriction.js, and functions/auth_triggers/config.js, our GCIP blocking functions intercept every signup and sign-in request before a Firebase JWT is ever minted:

Implementation Walkthrough: Multi-Domain Role Derivation & Pre-Enrolled Roster Linking

Why do we need four tiers instead of a simple @vtc.edu.hk suffix check? Because in real polytechnic and university environments:

Visiting industry lecturers or external examiners may be pre-enrolled by a course coordinator in classes/{classId}.teacherEmails before they ever log in for the first time (Tier 1).
Some institutions share a single apex domain (@school.edu) for both staff and students, distinguishing them only by username pattern (e.g., 9-digit student IDs vs. alphabetic staff handles—Tier 2).
In hierarchical domains (@stu.vtc.edu.hk vs. @vtc.edu.hk), a naive .endsWith('vtc.edu.hk') check would accidentally classify @stu.vtc.edu.hk as a teacher unless student subdomains take strict precedence (Tier 3).

Here is the production implementation from functions/ai_flows/config.js and functions/auth_triggers/userManagement.js:

// From functions/ai_flows/config.js
export function deriveUserRole(email) {
  if (!email || typeof email !== 'string' || !email.includes('@')) return null;
  const cleanEmail = email.trim().toLowerCase();
  const atIndex = cleanEmail.lastIndexOf('@');
  const username = cleanEmail.substring(0, atIndex);
  const domain = cleanEmail.substring(atIndex + 1);

  if (!username || !domain) return null;

  const matchesTarget = (targetDomain) =>
    targetDomain === '*' || domain === targetDomain || domain.endsWith('.' + targetDomain);

  const isStudentDomain = STUDENT_EMAIL_DOMAINS.some(matchesTarget);
  const isTeacherDomain = TEACHER_EMAIL_DOMAINS.some(matchesTarget);

  if (!isStudentDomain && !isTeacherDomain) return null;

  // Tier 2: Username Regex check takes precedence if configured
  if (STUDENT_USERNAME_REGEX && STUDENT_USERNAME_REGEX.test(username)) return 'student';
  if (TEACHER_USERNAME_REGEX && TEACHER_USERNAME_REGEX.test(username)) return 'teacher';

  // Tier 4 Guard: Check for same-domain or wildcard overlap
  const exactStudentDomain = STUDENT_EMAIL_DOMAINS.some((d) => d === '*' || domain === d);
  const exactTeacherDomain = TEACHER_EMAIL_DOMAINS.some((d) => d === '*' || domain === d);
  if (exactStudentDomain && exactTeacherDomain) {
    return DEFAULT_TO_STUDENT ? 'student' : null;
  }

  // Tier 3: Subdomain hierarchy (student subdomains win over parent teacher domains,
  // e.g., stu.vtc.edu.hk wins over vtc.edu.hk)
  if (isStudentDomain) return 'student';
  if (isTeacherDomain) return 'teacher';

  return DEFAULT_TO_STUDENT ? 'student' : null;
}

And during beforeUserCreated, we combine Tier 1 pre-enrollment checks with automatic profile hydration from the central studentDirectory in functions/auth_triggers/userManagement.js:

// From functions/auth_triggers/userManagement.js
export async function handleBeforeUserCreatedLogic(event, customDeps = {}) {
  const currentDb = customDeps.db || db;
  const currentDeriveRole = customDeps.deriveUserRole || deriveUserRole;
  const { uid, email } = event?.data || {};

  if (!email) {
    throw new HttpsError('invalid-argument', 'Email is required to sign up.');
  }

  const classesRef = currentDb.collection('classes');

  // Tier 1: Check if this email was pre-registered as a teacher in any existing class
  const teacherPreEnrollSnapshot = await classesRef
    .where('teacherEmails', 'array-contains', email)
    .limit(1)
    .get();
  const isPreEnrolledTeacher = !teacherPreEnrollSnapshot.empty;

  let derivedRole = isPreEnrolledTeacher ? 'teacher' : currentDeriveRole(email);
  if (!derivedRole) {
    throw new HttpsError('invalid-argument', 'Please use a valid institutional email address.');
  }

  // Return custom claims directly into the newly minted GCIP JWT
  return { customClaims: { role: derivedRole } };
}

Schedule-Aware Campus Lab CIDR Gating (`checkipaddress`)

During scheduled practical exams, we want to prevent remote students from logging into a class session from outside the physical computer lab subnet, without blocking students from reviewing their course materials from home in the evening.

In functions/auth_triggers/ipRestriction.js, checkipaddress (beforeUserSignedIn) evaluates the student's enrolled classes against their timezone-aware schedule.timeSlots using date-fns-tz and validates event.ipAddress against the class's IPv4 CIDR ranges (ip-address Address4.isInSubnet()):

// From functions/auth_triggers/ipRestriction.js
export const checkipaddress = beforeUserSignedIn(
  { region: FUNCTION_REGION, memory: '512MiB', cpu: 1 },
  async (event) => {
    const user = event.data;
    const ipAddress = event.ipAddress;
    if (!user.email || !ipAddress) return;

    // Teachers are never blocked by student lab IP restrictions
    if (user.customClaims && user.customClaims.role === 'teacher') return;

    const studentProfileSnap = await db.collection('studentProfiles').doc(user.uid).get();
    if (!studentProfileSnap.exists || !studentProfileSnap.data().classes?.length) return;

    const classSnapshots = await Promise.all(
      studentProfileSnap.data().classes.map((id) => db.collection('classes').doc(id).get())
    );

    let allRestrictions = [];
    classSnapshots.filter((s) => s.exists).forEach((doc) => {
      const classData = doc.data();
      // Only enforce IP restrictions if the class is currently in an active scheduled time slot
      if (isDuringScheduledTime(classData.schedule) && classData.ipRestrictions?.length > 0) {
        allRestrictions = allRestrictions.concat(classData.ipRestrictions);
      }
    });

    if (allRestrictions.length === 0) return; // Outside class hours -> allow home study access!

    const uniqueRestrictions = [...new Set(allRestrictions)];
    const ipAllowed = uniqueRestrictions.some((restriction) => {
      if (restriction.includes('/')) {
        return new Address4(ipAddress).isInSubnet(new Address4(restriction));
      }
      return ipAddress === restriction;
    });

    if (!ipAllowed) {
      throw new HttpsError(
        'permission-denied',
        'Login is only permitted from a school network during scheduled class times.'
      );
    }
  }
);

3. FIDO2 WebAuthn Hardware Lock: Enforcing `1-Phone = 1-Student` to Stop Password-Sharing & Fake Lab Recordings

Even when IP restrictions keep traffic inside the campus Wi-Fi or lab subnet, how do we stop Student A from logging into an adjacent lab PC with Student B's shared password, playing a pre-recorded screen video to fake lab work, and clicking Bingo answers for Student B?

The root flaw of password-based lab sessions is that credentials are zero-cost to share: Student B can give their password to Student A without losing access to anything. By contrast, no student will surrender their personal physical smartphone (and their Face ID / fingerprint) to a classmate for an entire morning. And when a class enables Mobile Passkey Bingo (mobile_passkey) or the Passkey Enforcement Gate (PasskeyEnforcementGate.jsx), every workstation must be unlocked and periodically verified by scanning a per-student challenge QR code with the student's paired hardware passkey phone.

To prevent Student A from registering both their own account and Student B's account on Student A's smartphone, we implemented a strict 1-Phone = 1-Student Hardware Lock in functions/ai_flows/passkeyFlows.js using @simplewebauthn/server. When a student pairs their smartphone:

We require authenticatorAttachment: 'platform', forcing the mobile browser to generate an asymmetric ECDSA P-256 keypair inside the phone's hardware Apple Secure Enclave (iOS Face ID / Touch ID) or Android Titan M2 module—non-exportable and immune to cloning.
During handleVerifyPasskeyRegistration, we check two orthogonal hardware identifiers across the entire studentPasskeys collection:
- The cryptographic WebAuthn credentialID.
- The persistent hardware deviceFingerprint (protecting against a student creating two different browser profiles, using Chrome + Safari, or clearing cookies on the same physical phone to register a classmate's account).
Once paired, students cannot self-unlink their smartphone. Unbinding a phone requires an instructor to invoke handleResetStudentPasskey, which records the previous credentialID, deviceModel, and teacher justification into an append-only passkeyAuditLogs collection. As a result, a classmate sitting at two lab monitors with a shared password and a looping screen recording is immediately caught on the very first hardware Passkey check!

Production Code: Hardware Collision Detection in `passkeyFlows.js`

// From functions/ai_flows/passkeyFlows.js
export async function handleVerifyPasskeyRegistration({
  pairingToken,
  attestationResponse,
  clientRpId,
  deviceModel,
  deviceFingerprint,
  auth,
}) {
  // ... [Challenge lookup & verifyRegistrationResponse via @simplewebauthn/server] ...
  const { credential } = verification.registrationInfo;
  const credentialID = credential.id;

  // =========================================================================
  // CRITICAL SECURITY ENFORCEMENT: 1 Phone = 1 Student Hardware Lock
  // 1. Check persistent deviceFingerprint: A phone cannot be shared across multiple students!
  //    EXCEPTION: Teachers and whitelisted testing accounts can share devices across roles.
  // 2. Check WebAuthn credentialID: Credential cannot be shared across multiple students!
  // =========================================================================
  const isIncomingWhitelisted =
    userRole === 'teacher' || isPasskeySharingWhitelisted(studentEmail);

  if (deviceFingerprint) {
    const existingDeviceSnap = await db
      .collection('studentPasskeys')
      .where('deviceFingerprint', '==', deviceFingerprint)
      .get();

    if (!existingDeviceSnap.empty) {
      for (const doc of existingDeviceSnap.docs) {
        if (doc.id !== studentUid) {
          const existingData = doc.data() || {};
          const boundEmail = existingData.studentEmail || doc.id;
          const existingRole = existingData.role || deriveUserRole(boundEmail) || 'student';
          const isExistingWhitelisted =
            existingRole === 'teacher' || isPasskeySharingWhitelisted(boundEmail);

          if (!isIncomingWhitelisted && !isExistingWhitelisted) {
            throw new HttpsError(
              'already-exists',
              `Hardware Lock: This physical phone is already bound to student account (${boundEmail}). Each mobile phone can only be used by one student.`
            );
          }
        }
      }
    }
  }

  const existingSnap = await db
    .collection('studentPasskeys')
    .where('credentialID', '==', credentialID)
    .get();

  if (!existingSnap.empty) {
    for (const doc of existingSnap.docs) {
      if (doc.id !== studentUid) {
        throw new HttpsError(
          'already-exists',
          'This physical mobile device is already registered to another user. Each phone can only be paired with one user account.'
        );
      }
    }
  }

  // Persist hardware public key, signature counter (replay protection), and fingerprint
  await db.doc(`studentPasskeys/${studentUid}`).set({
    studentUid,
    studentEmail,
    role: userRole,
    credentialID,
    credentialPublicKey: Buffer.from(credential.publicKey).toString('base64'),
    deviceFingerprint: deviceFingerprint || null,
    counter: credential.counter || 0,
    deviceModel: deviceModel || 'Mobile Device',
    transports: credential.transports || ['internal'],
    registeredAt: FieldValue.serverTimestamp(),
  });
}

4. Defeating WhatsApp Relay Attacks: 15-Second Rotating HMAC-SHA256 Projector QRs

In a 200-seat lecture hall where students do not sit at fixed lab desktops, how do we verify attendance in under 2 seconds while defeating photo-forwarding in WhatsApp group chats?

Diagram 2: 15-Second Rotating HMAC Projector QR & Challenge Creation Tolerance

Edge Case Solved: The "Mid-Biometric QR Rotation" Race Condition

When we first tested 15-second rotating QR codes with 150 students in a lecture theatre, we hit a subtle race condition:

A student scans the projector QR code at second t = 14.2s of window W_1.
getLecturePasskeyAuthOptions validates W_1 and triggers the phone's Face ID / fingerprint prompt.
Because the student's hands are damp or they adjust their glasses, their biometric scan takes 4.5 seconds—completing at t = 18.7s (now in window W_2).
If we shorten the grace window to block WhatsApp forwarding, a naive token check during verifyLecturePasskeyAuth rejects the legitimate in-room student because the projector QR rotated while Face ID was scanning!

We solved this in functions/ai_flows/passkeyFlows.js with Two-Phase Challenge Creation Tolerance:

Phase 1 (handleGetLecturePasskeyAuthOptions): When the camera scans the QR code, the server immediately validates the HMAC token using crypto.timingSafeEqual and writes a cryptographically random 2-minute WebAuthn challenge document (bingoRecords/{bingoId}/challenges/{challengeId}) stamped with createdAt: FieldValue.serverTimestamp().
Phase 2 (handleVerifyLecturePasskeyAuth): When the phone submits the signed hardware assertion a few seconds later, we verify whether the token is currently valid OR was valid at challengeCreatedAtMillis:

// From functions/ai_flows/passkeyFlows.js
export function computeLectureQrToken(sessionSecret, timeInterval) {
  if (!sessionSecret) return '';
  return crypto
    .createHmac('sha256', sessionSecret)
    .update(`lecture_qr_bingo_${timeInterval}`)
    .digest('hex')
    .substring(0, 16);
}

export function isValidLectureQrToken(
  sessionSecret,
  token,
  timestamp = Date.now(),
  rotationIntervalMs = DEFAULT_LECTURE_QR_ROTATION_INTERVAL_MS
) {
  if (!sessionSecret || !token || typeof token !== 'string' || token.length !== 16) {
    return false;
  }
  const intervalMs = Number(rotationIntervalMs) > 0 ? Number(rotationIntervalMs) : 15000;
  const currentInterval = Math.floor(timestamp / intervalMs);
  for (let i = 0; i <= LECTURE_QR_GRACE_INTERVALS; i++) {
    const expected = computeLectureQrToken(sessionSecret, currentInterval - i);
    if (
      expected.length === token.length &&
      crypto.timingSafeEqual(Buffer.from(token), Buffer.from(expected))
    ) {
      return true;
    }
  }
  return false;
}

// Inside handleVerifyLecturePasskeyAuth:
const challengeCreatedAtMillis =
  typeof challengeData.createdAt?.toMillis === 'function'
    ? challengeData.createdAt.toMillis()
    : Date.now();

const isTokenCurrentlyValid = isValidLectureQrToken(
  bingoData.sessionSecret,
  tokenToVerify,
  Date.now(),
  rotationIntervalMs
);
const isTokenValidAtChallengeCreation = isValidLectureQrToken(
  bingoData.sessionSecret,
  tokenToVerify,
  challengeCreatedAtMillis,
  rotationIntervalMs
);

if (!isTokenCurrentlyValid && !isTokenValidAtChallengeCreation) {
  throw new HttpsError(
    'invalid-argument',
    'The scanned QR token has expired. Please scan the current code on the screen.'
  );
}

This guarantees that an in-room student who scanned a live QR code never fails if the projector rotates while their biometric prompt is open, whereas a remote student receiving a forwarded WhatsApp screenshot after the grace window cannot even initiate Phase 1.

5. Keylogger-Proof Shared Lab PC Login & 3-Tier Emergency Bypass

How do teachers and students sign into untrusted shared lab workstations without touching the physical keyboard?

Diagram 3: Zero-Keystroke Lab PC Login & 3-Tier Emergency Recovery

Zero-Keystroke Custom Token Exchange (`handleVerifyDesktopLoginPasskey`)

When a teacher or student walks up to a lab PC and clicks 📱 Scan QR Code, the workstation calls initiateDesktopLoginSession, creating an ephemeral document at loginSessions/{sessionId} with a 256-bit sessionSecret and listening to it via Firestore onSnapshot.

When the user scans the PC monitor with their paired phone and completes Face ID, handleVerifyDesktopLoginPasskey:

Verifies the rotating desktop QR HMAC token (desktop_login_qr_${timeInterval}).
Looks up the hardware credentialID in studentPasskeys and verifies the cryptographic signature and replay counter (verification.authenticationInfo.newCounter).
Dynamically resolves the user's role (teacher vs. student) and mints a short-lived Firebase Custom Token via getAuth().createCustomToken(studentUid, { role: detectedRole }).
Writes { status: 'authorized', customToken } to loginSessions/{sessionId}. The desktop browser immediately calls signInWithCustomToken(auth, customToken)—signing the instructor into the Gemini 4 Argon (gemini-4-argon) command center with zero keystrokes typed on the lab keyboard.

Solving Real-World Classroom Hardware Realities: The 3-Tier Emergency Bypass

In a cohort of 150 students, every single week at least two students will arrive with:

A dead phone battery (0%),
A cracked camera lens that cannot focus on a QR code, or
A non-GMS Android handset (e.g., certain Huawei / HarmonyOS or custom ROM devices) that lacks Google Credential Manager support for WebAuthn passkeys.

If security blocks teaching, instructors will turn security off. We engineered three frictionless, fully audited recovery tiers in functions/ai_flows/passkeyFlows.js:

Tier A — 1-Click Podium Bypass (handleApproveTeacherPasskeyBypass): The student clicks "Request Teacher Bypass" on their lab PC (handleRequestTeacherPasskeyBypass), entering their desk number. A live toast appears on the instructor's podium dashboard; one click grants a 180-minute lesson-scoped bypass in classes/{classId}/studentProperties/{uid}.passkeyBypass.
Tier B — Aisle Walk-Around 6-Digit PIN (handleVerifyTeacherPasskeyBypassPin): While walking the lab aisles, the instructor can type their daily 6-digit emergency PIN directly onto the student's screen to unlock attendance for that session.
Tier C — Non-GMS Device Exemption & Phone Replacement (handleResetStudentPasskey): For students who buy a new phone or use a non-GMS device, the teacher can reset their hardware binding or toggle a roster exemption—with every action permanently recorded in passkeyAuditLogs.

6. Security & Performance Benchmarks

Authentication & Presence Flow	Median End-to-End Latency	Keystrokes on Shared Lab PC	WhatsApp Photo Relay Vulnerable?	Proxy Account Sharing Vulnerable?	Cloud Cost per 100 Verifications
Legacy Static QR / Password Form	~12.4 s	25–40 keys	❌ Yes (Instant group forward)	❌ Yes (Shared password)	~$0.0001
In-Class Routine Passkey Bingo (`verifyPasskeyAuth`)	1.6 s	0 keys	✅ No (Per-student challenge)	✅ No (`1-Phone = 1-Student` lock)	~$0.0002
200-Seat Lecture Hall Rotating QR (`verifyLecturePasskeyAuth`)	1.8 s	0 keys	✅ No (15s HMAC + Challenge Tolerance)	✅ No (`credentialID` + `deviceFingerprint`)	~$0.0003
Shared Lab PC QR Login (`verifyDesktopLoginPasskey`)	2.1 s	0 keys	✅ No (15s HMAC + Physical monitor scan)	✅ No (Hardware Secure Enclave)	~$0.0003

7. Key Takeaways

Never Trust Client-Side Role Claims: Use GCIP beforeUserCreated blocking functions with a deterministic hierarchy (Pre-Enrolled Roster $\rightarrow$ Username Regex $\rightarrow$ Subdomain Priority $\rightarrow$ Zero-Trust Student Default) so unauthorized users can never access high-quota Gemini 4 Argon (gemini-4-argon) teacher flows.
Enforce 1-Phone = 1-Student at the Hardware Layer: Combining WebAuthn platform authenticators (credentialID) with a persistent deviceFingerprint check in Firestore stops students from logging into a classmate's account on their own phone.
Stamp Token Validity at Challenge Creation: When pairing 15-second rotating HMAC QR codes with biometric prompts, validating the QR token at challenge creation time eliminates false rejections when the projector rotates mid-Face ID scan.
Build Audited Human Escape Hatches: A 3-tier teacher bypass (1-click podium approval, aisle PIN, and phone-replacement reset) keeps classes moving smoothly when a student's battery dies, without weakening cryptographic enforcement for the rest of the cohort.

Google AI Classroom Part 3: Frontier Multimodal Orchestration — Right-Sizing Gemini 4 & On-Device Gemma 4 E2B with Google Genkit

Cyrus Wong — Tue, 10 Nov 2026 06:12:34 +0000

Author: Cyrus Wong (黃俊彥) — Google Developer Expert - Cloud (AI), Senior Lecturer at HKIIT

When we run an AI-powered computer science or engineering classroom for 40 to 100+ concurrent students, supporting live teaching, inclusive learning, student performance evaluation, and proctored exams requires radically different latency, context-window, and FinOps profiles:

Every 16ms–3s on the Student's Laptop (Real-Time Proctored Exam Investigation): During a practical test or examination, a human invigilator cannot watch 40+ screens and webcams simultaneously, and calling cloud APIs every second for 40 seats would require 432,000+ cloud calls per exam. High-frequency posture, gaze, and speech-intent investigation must run locally in the browser via MediaPipe, LiteRT Whisper-Tiny, and LiteRT-LM Gemma 4 E2B (gemma-4-E2B-it-web.litertlm) at $0.00 API cost, immediately uploading synchronized screen + webcam + audio evidence when an irregularity is confirmed.
Every 4 Seconds at the Lecture Podium (Inclusive Teaching & Live Comprehension Checks): When a lecturer speaks Cantonese mixed with English Kubernetes/React jargon ("呢個 Pod 個 Readiness Probe fail 咗"), students need 9-language subtitles on their screens in under 800 milliseconds—making Gemini 3.5 Flash-Lite (gemini-3.5-flash-lite) ($0.25 In / $1.50 Out per 1M tokens) or on-device Gemma 4 E2B ideal.
Standard Student Performance Video Grading & Whole-Lecture CC Generation: Evaluating a 45-minute 1-FPS student lab recording against a practical task rubric or generating bilingual subtitles and YouTube study chapters from a 3-hour Opus lecture audio stream requires balanced multimodal throughput—where Gemini 3.8 Flash (gemini-3.8-flash) ($0.50 In / $3.00 Out per 1M tokens) shines.
Complex Rubric Synthesis, Multi-Step Code Evaluation & Capstone Supervision (Frontier Reasoning): When a teacher wants to watch an instructor demonstration video and synthesize a partial-credit grading rubric, evaluate complex student coding performance, audit 60 weekly capstone logs against live 720p mobile hardware demos, or meta-optimize a JSON-schema prompt, the workload demands Gemini 4 Argon (gemini-4-argon) ($2.00 In / $10.00 Out per 1M tokens) with its 1-million-token shared multimodal context window and deep reasoning capabilities.

With the arrival of Gemini 4 (gemini-4-argon) in Vertex AI alongside Gemma 4 E2B (gemma-4-E2B-it-web.litertlm) in the browser via LiteRT-LM WebGPU, we engineered our Google Genkit orchestration layer in functions/ai_flows/ so teachers can pick gemini-4-argon across every single classroom workflow—while automatically enforcing parameter sanitization, exponential backoff with jitter, cross-model failover, and real-time FinOps quota governance.

2. Architectural Deep-Dive: The 5-Model Edge-to-Cloud Right-Sizing Matrix

Rather than hardcoding a single model ID across our backend, we built a 5-Model Right-Sizing Architecture spanning Tier-1 Browser WebGPU Workers and Tier-4 Google Genkit Cloud Functions. Teachers can configure a class-wide default model in ClassManagement.jsx and override it on the fly inside any feature studio:

Complete Matrix of Teacher-Selectable Gemini 4 Surfaces

Every AI-powered workflow in the repository exposes Gemini 4 Argon (gemini-4-argon), Gemini 3.8 Flash (gemini-3.8-flash), and Gemini 3.5 Flash-Lite (gemini-3.5-flash-lite):

Classroom Workflow	Frontend Selector Component	Backend Cloud Function / Genkit Flow	Recommended Model Strategy
1. Class-Wide Default & Live Vision Monitor	`ClassManagement.jsx` & `ControlsPanel.jsx`	`analyzeImageFlow` / `analyzeAllImagesFlow`	`gemini-3.5-flash-lite` for instant classroom peeks; switch to `gemini-4-argon` for deep code inspection on a student's IDE screenshot.
2. Practical Task Rubric Synthesis	`TaskEditorModal.jsx` (`#task-ai-model-select`)	`extractTaskDemoSteps`	Use `gemini-4-argon` to watch your reference demo video and synthesize nuanced partial-credit marking criteria and visual proof requirements.
3. Practical Task Video Auto-Grading	`TaskGradingMatrixView.jsx` (`Grading AI Model`)	`evaluateTaskSubmission`	Defaults to `taskData.model`; override with `gemini-4-argon` in the Grading Matrix toolbar for complex multi-step debugging labs or exam appeals.
4. After-Class Cohort Video Map-Reduce	`VideoLibrary.jsx` & `VideoAnalysisJobs.jsx`	`analyzeSingleVideoFlow` & `generateLabTaskPrompt`	Use `gemini-3.8-flash` for Stage-1 per-student video logs and `gemini-4-argon` for Stage-2 cohort rubric discovery.
5. Whole-Lecture Subtitles (CC) & YouTube Chapters	`ClassManagement.jsx` & `LectureRecordingsView.jsx`	`processLectureSubtitles`	Single-pass pure Opus audio ingestion (`thinkingBudget: 0`, `maxOutputTokens: 65536`) with `gemini-3.8-flash` or `gemini-4-argon`.
6. Capstone Project Meeting Supervision	`ProjectMeetingsView.jsx`	`analyzeProjectMeeting`	Stage 1 diarizes audio via `gemini-3.5-transcribe-preview`; Stage 2 uses `gemini-4-argon` or `gemini-3.8-flash` to cross-examine weekly logs against 720p mobile demo videos and photos.
7. Bingo Question Bank AI Generator	`BingoQuestionBankModal.jsx` (`Bingo AI Model`)	`generateBingoQuestionBank`	Use `gemini-3.5-flash-lite` for fast recall MCQs or `gemini-4-argon` for scenario-based architecture diagnostic questions.
8. AI Prompt Studio Meta-Optimizer (`✨ Optimize`)	`PromptForm.jsx` & `PromptManagement.jsx`	`analysisFlows.js`	Select `gemini-4-argon` next to `Optimize` to rewrite rough instructor grading notes into schema-safe system prompts while locking `{{variable}}` tags.
9. Live Subtitles & Multilingual Captions	`TeacherSubtitleControlModal.jsx`	`translateTeacherSpeech` & `litertGemma.worker.js`	Toggle between $0.00 on-device Gemma 4 E2B (`client` mode) and Cloud Functions (`gemini-3.5-flash-lite` / `gemini-3.8-flash` / `gemini-4-argon`).

3. Implementation Walkthrough 1: Inside `generateWithResilience` — Gemini 4 Parameter Sanitization & Self-Healing Failover

One of the most important engineering lessons when adopting Gemini 4 (gemini-4-argon) in production is that frontier reasoning models enforce stricter API parameter contracts than earlier generation models:

Strict Parameter Contract on gemini-4-*: Passing legacy sampling parameters (temperature, topP, top_p, topK, top_k) inside config to gemini-4-argon throws an immediate 400 INVALID_ARGUMENT error because Gemini 4 manages internal reasoning sampling natively.
Transient Overload & Quota Spikes in Live Labs: When 40 students submit their practical exam videos simultaneously or a teacher triggers analyzeAllImagesFlow across 40 screens with up to maxToolRoundtrips = numScreenshots * 3, regional Vertex AI endpoints can return 429 RESOURCE_EXHAUSTED, 503 UNAVAILABLE, or ECONNRESET. The pipeline must never fail a student's submission—it must retry with exponential backoff + random jitter and seamlessly fail over to gemini-3.5-flash-lite if the primary model is exhausted.

Here is the exact sequence executed by generateWithResilience for every Genkit invocation across our backend:

Production Code Listing 1: `generateWithResilience` in `functions/ai_flows/analysisFlows.js`

Here is the production implementation in functions/ai_flows/analysisFlows.js showing how we strip unsupported sampling keys when modelName.startsWith('gemini-4') and handle network/quota retries before failing over:

/**
 * Executes AI generation with automatic exponential backoff, jitter,
 * connection reset recovery, and fallback to flash-lite if the primary model is overloaded.
 */
export async function generateWithResilience(
  generateConfig,
  preferredModel,
  fallbackModel = 'gemini-3.5-flash-lite'
) {
  const modelsToTry = [preferredModel];
  if (fallbackModel && preferredModel !== fallbackModel) {
    modelsToTry.push(fallbackModel);
  }

  let lastError = null;

  for (const modelName of modelsToTry) {
    const maxAttempts = 3;
    for (let attempt = 1; attempt <= maxAttempts; attempt++) {
      try {
        let effectiveGenerateConfig = generateConfig;
        if (
          typeof modelName === 'string' &&
          modelName.startsWith('gemini-4') &&
          generateConfig?.config &&
          typeof generateConfig.config === 'object'
        ) {
          const {
            temperature: _t,
            topP: _p,
            top_p: _tp,
            topK: _k,
            top_k: _tk,
            ...restConfig
          } = generateConfig.config;
          effectiveGenerateConfig = {
            ...generateConfig,
            config: restConfig,
          };
        }
        const response = await ai.generate({
          ...effectiveGenerateConfig,
          model: vertexAI.model(modelName),
        });
        return { response, modelUsed: modelName };
      } catch (err) {
        lastError = err;
        const msg = String(err?.message || err);
        const isRetryable =
          msg.includes('overloaded') || msg.includes('connection reset') ||
          msg.includes('ECONNRESET') || msg.includes('ETIMEDOUT') ||
          msg.includes('fetch failed') || msg.includes('socket') ||
          msg.includes('timeout') || msg.includes('503') ||
          msg.includes('429') || msg.includes('504') ||
          msg.includes('RESOURCE_EXHAUSTED') || msg.includes('UNAVAILABLE') ||
          msg.includes('INTERNAL');

        console.warn(`[AI Resilience] Attempt ${attempt}/${maxAttempts} for model ${modelName} encountered: ${msg}`);
        if (!isRetryable && attempt === 1) break;

        if (attempt < maxAttempts) {
          const backoffMs = Math.min(800 * Math.pow(2, attempt - 1) + Math.random() * 400, 4000);
          await new Promise((res) => setTimeout(res, backoffMs));
        }
      }
    }
  }

  throw lastError;
}

4. Implementation Walkthrough 2: Two-Stage Audio Diarization & Multimodal Rubric Synthesis

Two of the best examples of model right-sizing in our codebase are Two-Stage Audio Invigilation (analyzeAudioFlow) and Automated Demo Video Rubric Synthesis (extractTaskDemoSteps):

Why Two-Stage Audio (`gemini-3.5-transcribe-preview` + Reasoning Model) Beats Single-Pass Audio Tool Calling

In analyzeAudioFlow, we discovered that passing complex Genkit tool definitions (recordAudioIrregularity, recordAudioAudit, sendMessageToStudent) directly to an acoustic Speech-to-Text model degrades transcription fidelity on Cantonese/English code-switching. Instead, we split the pipeline into two specialized stages—and if the student's browser already transcribed the audio via LiteRT Whisper-Tiny, Stage 1 is skipped altogether (suppliedTranscript), saving 100% of audio input tokens!

Production Code Listing 2: Multimodal Reference Video Rubric Synthesis in `functions/ai_flows/extractTaskDemoSteps.js` & `TaskEditorModal.jsx`

When creating a hands-on practical assignment in TaskEditorModal.jsx, an instructor can attach a YouTube URL, upload an MP4 demo, or select a recorded lecture, and choose Gemini 4 Argon (gemini-4-argon), Gemini 3.8 Flash (gemini-3.8-flash), or Gemini 3.5 Flash-Lite (gemini-3.5-flash-lite):

{/* web-app/src/components/tasks/TaskEditorModal.jsx */}
<div>
  <label htmlFor="task-ai-model-select" className="block text-xs font-semibold text-gray-700 mb-1">
    🤖 Gemini Model for Rubric Synthesis &amp; Grading:
  </label>
  <select
    id="task-ai-model-select"
    value={aiModel}
    onChange={(e) => setAiModel(e.target.value)}
    className="w-full px-3 py-2 border border-purple-300 rounded-lg bg-white text-gray-900 text-sm font-medium"
  >
    <option value="gemini-4-argon">🧠 Gemini 4 Argon (Flagship Reasoning &amp; 1M Context — $2.00 / $10.00 per 1M)</option>
    <option value="gemini-3.8-flash">✨ Gemini 3.8 Flash (Recommended Multimodal — $0.75 / $3.75 per 1M)</option>
    <option value="gemini-3.5-flash-lite">⚡ Gemini 3.5 Flash-Lite (Fastest &amp; Low Cost — $0.30 / $2.50 per 1M)</option>
  </select>
</div>

On the backend, extractTaskDemoSteps.js resolves Cloud Storage (gs://) or YouTube URIs, invokes generateWithResilience with responseMimeType: 'application/json', and normalizes the extracted milestone steps, expected visual proof, and partial-credit marking criteria:

// functions/ai_flows/extractTaskDemoSteps.js
const preferredModel = model || 'gemini-3.8-flash';
const promptParts = [{ text: metaPrompt }];
if (resolvedVideoUri) {
  promptParts.push({ media: { url: resolvedVideoUri, contentType: 'video/mp4' } });
}
const generateConfig = {
  prompt: promptParts,
  config: {
    temperature: 0.2, // Automatically stripped by generateWithResilience if preferredModel === 'gemini-4-argon'
    responseMimeType: 'application/json',
  },
};

const res = await generateWithResilience(generateConfig, preferredModel);
const usage = res.response?.usage || res.response?.usageMetadata || {};
const cost = calculateCost(usage, res.modelUsed || preferredModel);

5. Implementation Walkthrough 3: Three-Tier Prompt Governance & Modality-Specific Meta-Prompting (`✨ Optimize`)

Hardcoding system prompts inside backend code prevents lecturers from adapting AI grading to their specific syllabus (e.g., Cybersecurity Penetration Testing vs. Frontend React Development). At the same time, writing JSON-schema-safe prompts with required runtime template placeholders ({{transcript}}, {{captureInterval}}, {{studentEmail}}) is error-prone for busy educators.

We solved this in PromptForm.jsx and promptRegistry.js with a Three-Tier Prompt Governance Architecture and an in-place Gemini 4 Prompt Optimizer:

Production Code Listing 3: Optimizer Model Picker & Required Placeholder Validation in `web-app/src/components/prompt/PromptForm.jsx`

In PromptForm.jsx, teachers can switch between gemini-4-argon, gemini-3.8-flash, and gemini-3.5-flash-lite right next to the Optimize button, while interactive placeholder chips (ph.required) warn the instructor if any mandatory runtime interpolation variable is missing before saving:

{/* web-app/src/components/prompt/PromptForm.jsx */}
{setOptimizerModel && (
    <select
        aria-label="Prompt Optimizer Model"
        value={optimizerModel}
        onChange={(e) => setOptimizerModel(e.target.value)}
        disabled={isOptimizing || isReadOnly}
        className="optimizer-model-select"
        title="Select Gemini model for prompt optimization"
    >
        <option value="gemini-4-argon">🧠 Gemini 4 Argon</option>
        <option value="gemini-3.8-flash">✨ Gemini 3.8 Flash</option>
        <option value="gemini-3.5-flash-lite">⚡ Gemini 3.5 Flash-Lite</option>
    </select>
)}
<button 
    type="button" 
    onClick={handleOptimize} 
    disabled={isOptimizing || isReadOnly}
    title="Optimize prompt with Gemini"
>
    {isOptimizing ? 'Optimizing...' : 'Optimize'}
</button>
<button type="button" onClick={handleUndo} disabled={!originalPromptText || isReadOnly} className="secondary-btn">
    Undo
</button>

6. FinOps Governance: Dynamic Cloud Billing SKU Sync & Composite Pipeline Cost Accounting

When different teachers across a university department experiment with gemini-4-argon, gemini-3.8-flash, gemini-3.5-flash-lite, and two-stage composite pipelines (gemini-3.5-transcribe-preview + gemini-3.5-flash-lite), institutional cloud administrators need micro-dollar cost attribution and hard per-class budget caps (aiQuota vs. aiUsedQuota).

Production Code Listing 4: Multi-SDK Token Normalization & Composite Pipeline Pricing in `functions/ai_flows/cost.js`

In functions/ai_flows/cost.js, we maintain a warm 12-hour in-memory pricing cache synced from system_config/pricing (populated daily by our Cloud Billing Catalog API job syncGeminiPricing), normalize token property variations across Genkit, Google GenAI SDK, and Vertex AI (promptTokenCount vs. inputTokens), and compute exact USD costs down to $0.000001:

// Default baseline pricing lookup table for Gemini models in USD per 1 million tokens (Input / Output)
export const MODEL_PRICING = {
  'gemini-4-argon': { input: 2.00, output: 10.00 },
  'gemini-3.5-flash-lite': { input: 0.30, output: 2.50 },
  'gemini-3.8-flash': { input: 0.75, output: 3.75 },
  'gemini-3.7-flash': { input: 0.75, output: 3.75 },
  'gemini-3.7-pro': { input: 3.00, output: 15.00 },
  'gemini-3.5-transcribe': { input: 0.50, output: 2.50 },
  'gemini-3.5-transcribe-preview': { input: 0.50, output: 2.50 },
  'gemini-3.5-transcribe-live': { input: 0.60, output: 3.00 },
  'gemini-3.1-flash-live-preview': { input: 0.60, output: 2.50 },
};

export function calculateCost(usageMetadata, model = DEFAULT_MODEL) {
  if (!usageMetadata || typeof usageMetadata !== 'object') return 0;
  const pricing = getModelPricing(model);
  const promptTokens = Math.max(
    0,
    Number(
      usageMetadata.promptTokenCount ??
      usageMetadata.inputTokens ??
      usageMetadata.promptTokens ??
      usageMetadata.inputTokenCount ?? 0
    )
  );
  const candidatesTokens = Math.max(
    0,
    Number(
      usageMetadata.candidatesTokenCount ??
      usageMetadata.outputTokens ??
      usageMetadata.completionTokens ??
      usageMetadata.candidatesTokens ??
      usageMetadata.outputTokenCount ?? 0
    )
  );
  const inputRate = pricing?.input ?? MODEL_PRICING[DEFAULT_MODEL].input;
  const outputRate = pricing?.output ?? MODEL_PRICING[DEFAULT_MODEL].output;
  const inputCost = (promptTokens / 1000000) * inputRate;
  const outputCost = (candidatesTokens / 1000000) * outputRate;
  return Number((inputCost + outputCost).toFixed(6)) || 0;
}

Before any Genkit flow invokes Vertex AI, checkQuota(classId, estimatedCost) verifies that the class's aiUsedQuota has not exceeded its allocated aiQuota budget. Once the call completes, logJob() writes an immutable audit record to aiJobs/{jobId} with the actual model used (actualModel returned by generateWithResilience), and an atomic Firestore trigger (onAiJobCreated in quotaTriggers.js) increments classes/{classId}.aiUsedQuota via FieldValue.increment(cost).

7. FinOps & Latency Benchmark Table Across the 5-Model Constellation

Here is how our 5-model architecture performs in production across real HKIIT classroom workloads in asia-east2:

Model Identifier	Execution Tier	Input / Output Rate (per 1M Tokens)	Median Latency	Typical Classroom Workload Cost	Best Pedagogical Use Case
`gemma-4-E2B-it-web.litertlm`	Tier 1: Browser WebGPU (`@litert-lm/core`)	$0.00 / $0.00	420 ms	$0.0000 (Unlimited)	Continuous exam speech-intent proctoring & $0.00 client-mode live lecture subtitles.
`gemini-3.5-flash-lite`	Tier 4: Vertex AI Genkit	$0.30 / $2.50	620 ms	~$0.00018 / call	Sub-second 7-language live subtitles, Bingo screen verification & Cloud face fallback.
`gemini-3.5-transcribe-preview`	Tier 4: Vertex AI Genkit	$0.50 / $2.50	1.1 s	~$0.00045 / min	Stage-1 multi-speaker Cantonese/Mandarin/English capstone consultation diarization.
`gemini-3.8-flash`	Tier 4: Vertex AI Genkit	$0.75 / $3.75	2.8 s	~$0.0104 / student video	Default practical task video grading & single-pass 3-hour Opus lecture CC generation.
`gemini-4-argon`	Tier 4: Vertex AI Genkit	$2.00 / $10.00	4.6 s	~$0.0370 / student video	Reference video rubric synthesis, complex multi-step code grading, cohort Map-Reduce discovery & Prompt Studio meta-optimization.

8. Key Takeaways for Google Cloud & Genkit Builders

Sanitize Legacy Sampling Configs for gemini-4-*: Wrap all Genkit ai.generate() calls in a central middleware (generateWithResilience) that automatically strips temperature, topP, and topK whenever modelName.startsWith('gemini-4')—allowing teachers to switch between Gemini 3.8 and Gemini 4 Argon without breaking shared flow code.
Always Record modelUsed After Failover: If a primary gemini-4-argon request fails over to gemini-3.5-flash-lite during a regional quota spike, pass the actual modelUsed returned by generateWithResilience into calculateCost() and logJob() so class FinOps quotas are billed at the true underlying model rate.
Decouple Acoustic Diarization from Tool-Calling Reasoning: Using gemini-3.5-transcribe-preview (without tools) for verbatim multi-speaker transcription and passing the resulting text to gemini-3.5-flash-lite, gemini-3.8-flash, or gemini-4-argon (with tools) eliminates tool-call hallucinations on bilingual audio.
Empower Educators with Guardrailed Prompt Engineering: Combining a 3-tier prompt resolution hierarchy with an in-app Gemini 4 Argon (✨ Optimize) meta-optimizer and required {{placeholder}} chip validation lets non-programmer faculty safely customize AI rubrics without breaking downstream JSON parsers.

Google AI Classroom Part 2: Zero-Egress Browser Edge AI Investigator — Running Gemma 4 E2B, MediaPipe 3D Face Mesh & LiteRT Whisper in Web Workers

Cyrus Wong — Tue, 10 Nov 2026 02:57:14 +0000

Author: Cyrus Wong (黃俊彥) — Google Developer Expert - Cloud (AI), Senior Lecturer at HKIIT

1. Problem Statement: Why Proctored Tests & Practical Examinations Need an On-Device Browser Edge AI Investigator

In proctored tests and practical examinations, student screen captures, webcam frames, and audio evidence are recorded and uploaded to Cloud Storage for official exam invigilation and forensic audit trails. However, relying solely on human invigilators or cloud-polled snapshots during a live 40-to-100+ student examination creates three critical operational blind spots:

The Human Invigilator Attention Bottleneck: A single lecturer or invigilator cannot simultaneously watch 40+ student screens, 40+ live webcam feeds, and listen to 40+ microphones in real time to catch fleeting exam misconduct—such as glancing at a concealed smartphone, peeking at a neighbor's monitor, whispering answers, or querying a voice assistant.
Fleeting Violations Between Snapshots vs. 432,000+ Cloud AI Calls: Periodic cloud screen and webcam snapshots provide the historical archive, but a 3.5-second glance at a cheat sheet or a whispered answer often occurs between polling intervals. Conversely, streaming 30-FPS video and continuous audio from 40 exam seats to Cloud LLMs for real-time AI inspection every second would require 432,000+ cloud API calls per 3-hour exam—saturating exam lab bandwidth, triggering API rate limits, and incurring runaway cloud inference costs.
High False-Positive Rates Without Local Geometric & Semantic Reasoning: During a practical coding or cloud exam, students naturally blink, glance down at their keyboard, read exam questions aloud to themselves ("Let me check Question 3"), or call the invigilator for technical assistance ("Mr. Wong, my Docker daemon crashed"). Naive motion or audio-level triggers flood invigilators with false alarms.

Using Google MediaPipe (@mediapipe/tasks-vision), Google LiteRT.js (@litertjs/core Whisper-Tiny), and LiteRT-LM WebGPU (@litert-lm/core Gemma 4 E2B gemma-4-E2B-it-web.litertlm), we turned every student's browser into an always-on Zero-Egress Browser Edge AI Investigator—continuously inspecting 3D head pose, iris metric distance, and spoken semantic intent at 60 FPS with $0.00 cloud AI inference egress, and immediately flagging confirmed exam irregularities with synchronized dual screen + webcam + audio evidence for the invigilator.

2. Architectural Deep-Dive: Off-Main-Thread Web Worker & Hardware Frame Sync Pipeline

Running a 478-point 3D facial landmark detector, an iris-depth estimator, a Chirp-Z log-Mel spectrogram extractor, a Whisper speech-to-text model, and a 2-billion-parameter Gemma 4 E2B LLM on the main React thread would freeze the browser UI immediately.

In useFaceMonitor.js, litertWhisper.worker.js, litertGemma.worker.js, and useClientLiteRTGemma.js, we isolate speech and LLM reasoning inside dedicated Web Workers while synchronizing vision landmark detection to the browser's hardware video decoder via requestVideoFrameCallback():

3. Implementation Walkthrough 1: 478-Point 3D Face Mesh, Iris Metric Depth & The 3-Second Debounce State Machine

Legacy proctoring software is notorious for false-positive alerts: if a student sneezes, blinks, or glances down at their mechanical keyboard for 500 milliseconds, they get flagged for "cheating." Conversely, if a student sits in a corner desk where their webcam is mounted at a $15^\circ$ angle, uncalibrated head-pose detectors flag them continuously.

We solved both problems in useFaceMonitor.js through three mathematical and state-machine innovations:

1-Click Adaptive Neutral Pose Calibration (calibrateBaseline): Students click 🎯 Set Center Pose to zero out their personal baseline head Yaw ($\text{Yaw}_0$) and Pitch ($\text{Pitch}_0$) offsets for off-center monitor mounts.
Monocular Depth-from-Iris Distance Estimation ($D_{\text{cm}}$): Using the anatomical constancy of the human horizontal iris diameter ($d_{\text{iris}} \approx 11.7\text{ mm}$) and MediaPipe's iris boundary landmarks (469, 471 for left iris; 474, 476 for right iris), we compute physical camera-to-face distance in centimeters without any hardware depth sensor:

3-Second Debounce Gate with 600ms Blink/Jitter Grace Window & Lifecycle Incident Tracking: An anomaly (looking_away, no_face, multiple_faces, eyes_closed, talking) must persist continuously for debounceSeconds (default 3.0 seconds) before opening a single active incident document in Firestore (irregularities/{id}), and automatically updates endedAt and durationSeconds when the student returns to normal posture!

Production Code Listing 1: 3D Iris Depth, Eye-Gaze Ratio & Anti-Jitter Debounce in `web-app/src/hooks/useFaceMonitor.js`

Here is the exact real-time biometric geometry and debounce gate running inside useFaceMonitor.js:

// MediaPipe Iris: Depth-from-Iris Distance Calculation (11.7mm constant human iris diameter)
let calculatedDistanceCm = 55;
if (landmarks[469] && landmarks[471] && landmarks[474] && landmarks[476]) {
  const leftIrisPx = Math.hypot(landmarks[469].x - landmarks[471].x, landmarks[469].y - landmarks[471].y) * (video.videoWidth || 640);
  const rightIrisPx = Math.hypot(landmarks[474].x - landmarks[476].x, landmarks[474].y - landmarks[476].y) * (video.videoWidth || 640);
  const avgIrisPx = (leftIrisPx + rightIrisPx) / 2;

  if (avgIrisPx > 2) {
    const focalLength = 0.85 * (video.videoWidth || 640);
    calculatedDistanceCm = Math.round((focalLength * 1.17) / avgIrisPx);
    calculatedDistanceCm = Math.max(15, Math.min(220, calculatedDistanceCm));
    setMetricDistance(calculatedDistanceCm);
  }
}

// Horizontal Yaw & Vertical Pitch with Adaptive Neutral Baseline Calibration Offset
const dLeft = Math.hypot(nose.x - leftEyeOuter.x, nose.y - leftEyeOuter.y);
const dRight = Math.hypot(nose.x - rightEyeOuter.x, nose.y - rightEyeOuter.y);
const rawYaw = Math.round(((dLeft - dRight) / (dLeft + dRight)) * 100);
const dNoseChin = chin.y - nose.y;
const dNoseForehead = nose.y - forehead.y;
const rawPitch = Math.round(((dNoseChin - dNoseForehead) / (dNoseChin + dNoseForehead)) * 100);

const calculatedYaw = rawYaw - (baselineOffsetRef.current?.yaw || 0);
const calculatedPitch = rawPitch - (baselineOffsetRef.current?.pitch || 0);

// Anti-Jitter Debounce Gate Logic with 600ms Grace Window
const now = Date.now();
if (currentAnomaly) {
  lastAnomalyTimeRef.current = now;
  if (!anomalyStartRef.current || anomalyStartRef.current.type !== currentAnomaly) {
    anomalyStartRef.current = { type: currentAnomaly, startTime: now, details: detailsMessage };
  } else {
    const elapsedSeconds = (now - anomalyStartRef.current.startTime) / 1000;
    if (elapsedSeconds >= debounceSeconds) {
      setActiveViolation(currentAnomaly);
      startIncident(currentAnomaly, detailsMessage);
    }
  }
} else {
  // Grace window of 600ms to tolerate natural blinks or momentary landmark noise
  const timeSinceLastAnomaly = now - (lastAnomalyTimeRef.current || 0);
  if (timeSinceLastAnomaly > 600) {
    anomalyStartRef.current = null;
    setActiveViolation(null);
    if (activeIncidentDocRef.current) {
      resolveIncident();
    }
  }
}

4. Implementation Walkthrough 2: On-Device Whisper-Tiny Log-Mel DSP & Special Language Token Decoding

In Hong Kong classrooms, students and lecturers naturally code-switch between Cantonese, Mandarin, and English technical terminology within a single sentence (e.g., "唔該阿Sir，呢個 Question 3 個 option B 係咪啱㗎？").

To transcribe student speech on-device with zero UI jank, litertWhisper.worker.js and whisperTokenizer.js run @litertjs/core inside a dedicated Web Worker:

Dynamic Peak Auto-Gain & 80-Bin Log-Mel Spectrogram via Chirp-Z FFT: Whisper expects a normalized [1, 80, 3000] Float32 tensor representing 30 seconds of 16 kHz audio. Because quiet whispering during an exam often has low amplitude ($0.003 < \text{peak} < 0.25$), extractAudioFeatures() applies up to $6\times$ automatic gain normalization before computing a 400-point Hann-windowed Chirp-Z Fast Fourier Transform (hopLength = 160) and projecting onto 80 Slaney-normalized Mel filterbanks.
Whisper Special Language Token Decoding & WebGPU-to-WASM Delegate Failover: Rather than relying on brittle regex heuristics, extractWhisperLanguageToken() reads Whisper's native special language token (50259..50357, e.g. <|en|>, <|zh|>, <|yue|>) directly from the decoded token stream right after <|startoftranscript|> (50258), while Gemma 4 E2B (litertGemma.worker.js) and Cloud Gemini (analyzeAudioFlow) perform semantic language and code-switching classification ("language": "cantonese|mandarin|english|mixed") alongside exam intent evaluation.

Production Code Listing 2: WebGPU/WASM Whisper Compilation & Special Language Token Decoder in `litertWhisper.worker.js` and `whisperTokenizer.js`

/**
 * Extracts the language code emitted by Whisper's special language token
 * (`<|en|>`, `<|zh|>`, `<|yue|>`, etc.) in the output token sequence.
 */
export function extractWhisperLanguageToken(tokenIds, vocabulary = [], fallbackLanguage = 'auto') {
  if (!tokenIds || typeof tokenIds === 'string') return fallbackLanguage;
  const prefixLength = Math.min(tokenIds.length || 0, 8);
  for (let i = 0; i < prefixLength; i++) {
    const tokenId = Math.round(tokenIds[i]);
    const vocabToken = vocabulary?.[tokenId];
    if (typeof vocabToken === 'string') {
      const match = vocabToken.match(/^<\|([a-z]{2,3})\|>$/i);
      if (match) {
        return match[1].toLowerCase();
      }
    }
    if (tokenId >= WHISPER_LANGUAGE_TOKEN_START && tokenId <= WHISPER_LANGUAGE_TOKEN_END) {
      return WHISPER_LANGUAGE_ID_MAP[tokenId] || fallbackLanguage;
    }
  }
  return fallbackLanguage;
}

export async function compileWhisperModel(loadAndCompile, modelBuffer, preferredDelegate) {
  const compile = (accelerator) => loadAndCompile(
    new Uint8Array(modelBuffer),
    { accelerator }
  );

  if (preferredDelegate !== 'webgpu') {
    return { model: await compile('wasm'), delegate: 'wasm' };
  }

  try {
    return { model: await compile('webgpu'), delegate: 'webgpu' };
  } catch (webGpuError) {
    console.warn(
      '[LiteRTWorker] WebGPU could not compile the dynamic Whisper graph; retrying with WASM:',
      webGpuError?.message || webGpuError
    );
    return { model: await compile('wasm'), delegate: 'wasm' };
  }
}

5. Implementation Walkthrough 3: On-Device Gemma 4 E2B (`gemma-4-E2B-it-web.litertlm`) for Semantic Proctoring, Dialect Classification & Live Translation

Recognizing speech tokens is only half the battle: if a student mutters "Let me read this question carefully" (benign self-talk) or raises their hand and says "Mr. Wong, my Docker daemon crashed" (legitimate inquiry), a keyword-based detector would falsely accuse them of cheating.

We needed a true Large Language Model running locally in the browser to evaluate semantic intent and spoken dialect/code-switching. Using @litert-lm/core (v0.15.0) and the quantized Gemma 4 E2B model (gemma-4-E2B-it-web.litertlm, ~1.91 GB in WebGPU), litertGemma.worker.js serves two mission-critical classroom roles at $0.00 API cost:

EVALUATE_TRANSCRIPT (Exam Acoustic Intent & Dialect Investigator): Classifies student speech transcripts into one of 5 strict categories (COLLUSION_EXAM, EXTERNAL_AI_ASSIST, UNAUTHORIZED_TALK, LEGITIMATE_INQUIRY, or BENIGN) along with spoken dialect/code-switching ("language": "cantonese|mandarin|english|mixed"), severity, confidence, quoted evidence, and rationale—and supports custom teacher prompts ({{transcript}}, {{classId}}, {{studentEmail}}) authored in the AI Prompt Studio!
TRANSLATE_TRANSCRIPT ($0.00 Client-Mode Live Subtitles): When a lecturer selects On-Device Gemma 4 E2B in the Live Subtitles modal, the same WebGPU worker translates Cantonese/English code-switched lecture speech into up to 7 target languages (en, zh-Hant, zh-Hans, ja, ko, es, fr) while preserving technical identifiers like useEffect, Docker, and Python.

Production Code Listing 3: Gemma 4 E2B WebGPU Initialization & Structured Output Parser in `web-app/src/workers/litertGemma.worker.js`

const GEMMA_OUTPUT_INSTRUCTIONS = `Use exactly one category:
- COLLUSION_EXAM: asking for, offering, or discussing exam answers, questions, or options
- EXTERNAL_AI_ASSIST: asking a voice assistant, search engine, phone, or AI tool for help
- UNAUTHORIZED_TALK: unrelated conversation with another person during a silent exam
- LEGITIMATE_INQUIRY: procedural or technical help requested from the teacher or proctor
- BENIGN: self-talk, reading aloud, ambient speech, coughing, or silence

Also classify the spoken language/code-switching of the transcript in "language" as one of: "cantonese", "mandarin", "english", or "mixed".

Respond with only one JSON object and no markdown:
{"isViolation":boolean,"category":"COLLUSION_EXAM|EXTERNAL_AI_ASSIST|UNAUTHORIZED_TALK|LEGITIMATE_INQUIRY|BENIGN","severity":"critical|high|medium|low|none","confidence":number,"language":"cantonese|mandarin|english|mixed","evidence":"quoted key phrase","rationale":"short explanation"}`;

async function initializeGemma(modelUrl) {
  if (gemmaEngine) return { fromCache: gemmaLoadedFromCache };
  if (initializationPromise) return initializationPromise;

  initializationPromise = (async () => {
    const ua = self.navigator?.userAgent || '';
    const isMobileOrIos =
      /iPhone|iPad|iPod|Android|CriOS|FxiOS/i.test(ua) ||
      (self.navigator?.platform === 'MacIntel' && Number(self.navigator?.maxTouchPoints) > 1);
    if (isMobileOrIos) {
      throw new Error(
        'On-device Gemma 4 E2B (~1.9 GB) exceeds mobile browser memory limits; using Cloud fallback.'
      );
    }

    if (typeof self.navigator?.deviceMemory === 'number' && self.navigator.deviceMemory < 4) {
      throw new Error(
        `Device RAM (${self.navigator.deviceMemory} GB) is below the 4 GB minimum for Gemma 4 E2B; using Cloud fallback.`
      );
    }

    if (!navigator.gpu) {
      throw new Error('WebGPU is unavailable on this device');
    }

    configureLiteRtLmWasmLocation();
    const { modelStream, fromCache, cacheWritePromise } = await streamModelWithProgress(modelUrl);
    self.postMessage({ type: 'STATUS', payload: { status: 'loading', progress: 90 } });
    gemmaEngine = await Engine.create({
      model: modelStream,
      mainExecutorSettings: { maxNumTokens: 2048 },
    }, GEMMA_PROCTOR_SYSTEM_PROMPT);
    const cached = await cacheWritePromise;
    gemmaLoadedFromCache = fromCache;
    return { fromCache, cached: fromCache || cached };
  })();

  try {
    return await initializationPromise;
  } finally {
    initializationPromise = null;
  }
}

6. Edge Cases & Production Failure Modes We Solved: Fresh "DeepFreeze" Lab PCs, Dynamic Tensor Guards, iOS Jetsam OOM & Live Hot-Swapping

Deploying a 1.91 GB WebGPU LLM (gemma-4-E2B-it-web.litertlm) and an on-device Whisper TFLite worker across university computer labs and BYOD student devices exposed four real-world failure modes that we engineered around in StudentView.jsx, gemmaLiteRTLoader.js, useClientLiteRTGemma.js, and litertWhisper.worker.js:

"DeepFreeze" Wiped Lab PCs & Automatic Uncached Pre-Warming: University lab PCs reboot into a clean image every morning, wiping CacheStorage (isGemmaCached === false). If auto-preloading gated on isGemmaCached, a wiped lab PC would stay stuck at gemmaStatus: 'idle' (gemmaEngine: 'uninitialized') forever unless the student manually clicked a preload button. Instead, StudentView.jsx automatically triggers preloadGemmaModel() whenever preloadClientAi is broadcast by the teacher, whenever active audio monitoring starts (isAudioCaptureActive), or on the very first spoken transcript (handleLiveVoiceTranscript), requesting navigator.storage.persist() and streaming the 1.91 GB .litertlm bundle directly into Engine.create({ model: modelStream }).
Non-Blocking Cloud-to-Edge Live Hot-Swap: If an exam begins while Gemma 4 E2B is still downloading on a wiped lab PC, the student is never blocked from starting their exam. Any speech detected while isGemmaReady is false is transparently routed to our Cloud Genkit flow (analyzeAudio using gemini-3.5-transcribe-preview + gemini-3.5-flash-lite), and automatically hot-swaps to 100% local Gemma 4 E2B WebGPU inference the millisecond the worker emits INIT_COMPLETE (ready: true)!
Dynamic-Output TFLite Tensor Allocation Guard (clientInferenceSupported): Certain exported Whisper TFLite graphs declare their output sequences tensor with kTfLiteDynamic allocation, which causes @litertjs/core WASM LiteRtCompiledModel.run() to throw Failed to register output tensor buffer: Failed to set custom allocation for output. Our worker catches this on the first segment and flips clientInferenceSupported = false to prevent repeated WASM C++ error spam while seamlessly continuing transcription via the bound MediaStreamTrack recognizer and Firebase AI Logic STT fallback.
iOS / iPadOS WebKit Jetsam OOM Guard & Storage Quota Pre-Flight: Allocating a 1.91 GB WebGPU buffer on an iPhone or iPad (which masquerades as MacIntel with maxTouchPoints > 1 on iPadOS) triggers Apple's kernel Jetsam killer, crashing the browser tab. Similarly, laptops with < 2.5 GB free disk space throw QuotaExceededError mid-download. Our isMobileOrIosBrowser(), deviceMemory < 4, and precheckGemmaViability() guards fail fast in < 5ms and route mobile/low-RAM devices cleanly to Cloud Fallback.

Production Code Listing 4: Cloud-to-Edge Hot-Swap & Dual Firestore Alerting in `StudentView.jsx` and `useClientLiteRTGemma.js`

Notice how StudentView.jsx automatically starts preloadGemmaModel() if gemmaStatus === 'idle' while routing transcripts to Cloud Genkit until isGemmaReady becomes true, and how useClientLiteRTGemma.js writes confirmed violations and LLM-classified speechLanguage to Firestore:

// In StudentView.jsx: Non-blocking Cloud-to-Edge Hot-Swap + Auto-Start on Idle
const handleLiveVoiceTranscript = useCallback(async (transcript) => {
  if (!shouldEvaluateVoiceWithGemma) return null;

  if (isGemmaReady) {
    const localResult = await evaluateSpeechWithGemma(transcript);
    if (localResult) {
      return localResult;
    }
  } else if (isGemmaWorkable && gemmaStatus === 'idle' && preloadGemmaModel) {
    Promise.resolve(preloadGemmaModel()).catch((err) =>
      console.debug('[StudentView] Auto-starting Gemma 4 E2B preload on transcript:', err)
    );
  }

  if (!shouldSendTranscriptToCloud(effectiveVoiceAiMode)) return null;

  const analyzeAudioCallable = httpsCallable(functions, 'analyzeAudio');
  const response = await analyzeAudioCallable({
    transcript,
    classId: activeClass,
    studentUid: user?.uid,
    studentEmail: user?.email,
  });
  return response?.data || null;
}, [activeClass, effectiveVoiceAiMode, evaluateSpeechWithGemma, gemmaStatus, isGemmaReady, isGemmaWorkable, preloadGemmaModel, shouldEvaluateVoiceWithGemma, user?.email, user?.uid]);

// In useClientLiteRTGemma.js: Dual Firestore Irregularity & Status Telemetry Update
if (isViolation && currentClassId && currentStudentUid) {
  const irregPayload = {
    classId: currentClassId,
    studentUid: currentStudentUid,
    email: currentStudentEmail || '',
    studentEmail: currentStudentEmail || '',
    title: `AI Speech Alert: ${category.replace(/_/g, ' ')}`,
    message: rationale || evidence || `Spoken text: "${transcript}"`,
    type: 'audio',
    category,
    severity,
    confidence,
    evidence: evidence || '',
    rationale: rationale || '',
    transcriptSnippet: transcript,
    transcript: transcript,
    source: 'on_device_gemma',
    timestamp: serverTimestamp(),
  };

  await addDoc(collection(db, 'irregularities'), irregPayload);
  await addDoc(collection(db, 'classes', currentClassId, 'irregularities'), irregPayload);
}

if (currentClassId && currentStudentUid) {
  const statusRef = doc(db, 'classes', currentClassId, 'status', currentStudentUid);
  await setDoc(statusRef, {
    gemmaAlert: isViolation ? category : null,
    gemmaSeverity: isViolation ? severity : null,
    gemmaConfidence: confidence,
    speechLanguage: language,
    lastGemmaTimestamp: Date.now(),
  }, { merge: true });
}

7. Browser Edge AI Performance & Memory Benchmarks

We benchmarked our 3-model browser edge stack across standard HKIIT lab hardware (Intel Core i5-12500 / 16 GB RAM integrated UHD 770 GPU, and Apple M2 MacBook Air 8 GB RAM) running Chrome 130+:

On-Device Edge Model	Runtime & Delegate	Model Size (Cached in `CacheStorage`)	Cold Load (From Cache)	Inference Latency	Main-Thread UI Frame Rate	Cloud API Cost
MediaPipe FaceLandmarker (478-pt + Iris)	`@mediapipe/tasks-vision` (WebGL GPU)	~3.8 MB	180 ms	11–16 ms / frame	60 FPS (`requestVideoFrameCallback`)	$0.00
LiteRT Whisper-Tiny (Multilingual)	`@litertjs/core` (WebGPU / WASM + Chirp-Z FFT)	~41 MB	420 ms	280–450 ms / segment	60 FPS (Isolated Worker)	$0.00
LiteRT-LM Gemma 4 E2B (`gemma-4-E2B-it-web.litertlm`)	`@litert-lm/core` v0.15.0 (WebGPU)	~1,915 MB (1.91 GB)	2.4 s (Streamed from Cache)	380–620 ms / JSON eval	60 FPS (Isolated Worker)	$0.00
Cloud Fallback (`analyzeFaceFallback` / `analyzeAudio`)	Vertex AI Genkit (`gemini-3.5-flash-lite` / `gemini-4-argon`)	0 MB	0 ms	650–1,100 ms	60 FPS (Async HTTPS Callable)	~$0.0002 / check

8. Key Takeaways for Web AI & Google Cloud Builders

Stream Large .litertlm Models Directly into Engine.create(): Instead of buffering a 1.91 GB ArrayBuffer twice in memory, tee the fetch / CacheStorage ReadableStream directly into @litert-lm/core Engine.create({ model: modelStream }) while reporting live download percentages (5%..90%) to the UI.
Always Dispose Ephemeral LLM Conversations: Inside a long-running Web Worker, calling await conversation.delete() in a finally block after every single-turn EVALUATE_TRANSCRIPT or TRANSLATE_TRANSCRIPT request prevents KV-cache memory leaks over a 3-hour exam.
Guard Against Mobile/iPadOS OOMs Before Fetching: Checking maxTouchPoints > 1 on MacIntel (iPadOS), deviceMemory < 4, and navigator.storage.estimate() before downloading multi-gigabyte weights prevents browser tab crashes and seamlessly routes mobile learners to Cloud Gemini fallback.
Combine Geometric Heuristics with Debounce Gates: Pairing MediaPipe's 11.7 mm iris depth estimation and neutral-pose calibration with a 3.0-second debounce gate and 600 ms blink grace window eliminates false-positive cheating flags while catching real irregularities with dual screen-and-webcam visual proof.

DEV Community: Google Developer Experts

Stop Calling getGenerativeModel() Everywhere! The Hidden Traps of Scaling Firebase AI to Multi-Model Angular Apps

Architecture & Mental Model

1. Defining AiModelOptions and Deterministic Cache Keys

1.1. The AiModelOptions Interface

1.2. Private State and generateKey() in AiModelFactory

2. Creating Models On Demand and Invalidating the Cache in AiModelFactory

2.1. Lazily Loading the SDK and Assembling Parameters (createModel, createModelParams, createGenerationConfig)

2.2. Reusing Models On Demand (getModel and getTtsModel)

2.3. Evicting Cached Models When Remote Config Rotates the Backend (getValidBackend)

3. Refactoring VisionService and TextToSpeechService (and the Shallow-Spread Trap)

3.1. Simplifying VisionService for Future Multi-Model Routing

3.2. Reusing #ttsModel in TextToSpeechService Without Wiping Out responseModalities

Takeaways

Resources

Google AI Classroom Part 10: Green AI FinOps, Self-Cleaning Storage Lifecycles & 1-Command Terraform IaC for Multi-School Deployment

1. The Three Pillars of Institutional Day-2 Production Readiness

2. Diagram 1: Real-Time AI FinOps Pipeline, Live Cloud Billing SKU Sync & Atomic Quota Circuit Breakers

Production Code 1: Live Google Cloud Billing Catalog SKU Sync (syncGeminiPricing) & Multi-Stage Cost Engine (cost.js)

3. Diagram 2: 9-Category Event-Driven Storage Quota & 4-Stage Cascading Class Purge (onClassDocDeleted)

Production Code 2: 9-Category Event-Driven Cloud Storage Quota Metering & Self-Healing Non-Negative Drift Guard (storageQuota.js)

Production Code 3: The 4-Stage Zero-Orphan Cascading Class Purge (onClassDocDeleted in cleanupTriggers.js)

4. Diagram 3: 1-Command Zero-Click Terraform Onboarding & Multi-Layer Automated Verification Pyramid

How ./setup-new-project.sh Eliminates Manual Cloud Console Clicks

5. Summary of the 10-Part Google AI Classroom Series

Explore the Open-Source Repository

Google AI Classroom Part 9: Gamified "Bingo" Active Presence Verification & Zero-Trust Proctored Exam Mode on Google Cloud

1. Problem Statement: Defeating "Fake Video Playback", Proxy Classmates & Static Screen Decoys

2. Diagram 1: Four Classroom Bingo Verification Modes & The Gemini 4 Question Bank Studio

Engineering Highlights of the Bingo Question Engine

Production Code 1: Genkit Structured Schema Generation & Fisher-Yates Positional De-Biasing (bingoFlows.js)

The 3 Prompts Powering Gamified Classroom Bingo (bingoFlows.js)

1. Question Bank Studio Prompt (Bingo Question Bank Generator.md / generateBingoQuestionBank)

2. Mode 2 Prompt — 1-to-Many Teacher Broadcast Screen + 5-Minute Live Subtitle Fusion + Anti-Duplication Rule (resolveBingoQuestion)

3. Mode 3 Prompt — Personalized Student Lab Screen Presence Verification (Bingo Student Screen Presence Question Generator.md)

3. Diagram 2: The Non-Punitive Two-Strike State Machine with Google Cloud Tasks & 3-Color Attendance Heatmap

Production Code 2: Event-Driven Two-Strike Cloud Tasks Scheduling & Elapsed AFK Deduction (bingoFlows.js)

4. Diagram 3: Zero-Trust Exam Mode Lockdown & Court-Ready .docx Forensic Dossiers

5. Deep Dive into the Zero-Trust Exam Code: Storage Metadata Locks, Signed URL Shielding & .docx Dossiers

Production Code 3: Write-Once Exam Metadata Locks in Cloud Storage (storage.rules) & Signed URL Broker (getStudentVideoPlaybackUrl.js)

Production Code 4: Court-Ready Microsoft Word (.docx) & RFC 4180 CSV Forensic Dossier Compiler (processReportJob.js)

The 3 Zero-Trust Exam Mode Invigilation Prompts Feeding the .docx Dossier

1. Screen Irregularity & Tool-Calling Invigilator (admin/prompts/images/AI invigilator.md)

2. Cloud Fallback Face & Gaze Invigilator (admin/prompts/images/Cloud Fallback Face & Gaze Invigilator.md)

3. Live Rolling Audio Collusion & Whisper Invigilator (admin/prompts/audios/AI Voice Invigilator (Live Rolling Audio).md)

What's Next in Part 10?

Google AI Classroom Part 8: AI Project Supervision — Streamlined Mobile 720p Demo, 2-Stage Diarization & Fake-Log Auditing with Gemini 4

1. The Capstone Supervision Bottleneck: ChatGPT Logbooks, Free-Riders & Multi-Device Consultation Friction

2. Diagram 1: Self-Healing Google Sheets Sync & Pre-Meeting Fake-Log AI Auditor

Why Hamming Distance = 1 Student ID Healing Saved Hours of Manual Fixes

Production Code 1: Pre-Meeting Fake-Log Detection & Live Code-Walkthrough Challenge Generator (handlePrepareProjectLogQuestions)

3. Diagram 2: Streamlined Single-Device Teacher Mobile Supervision Studio (Voice + 0.1s Photo Snap + Auto-Sliced 720p Demo Video)

Atomic Stale Analysis Invalidation When Moving Clips Across Groups or Weeks

4. Diagram 3: Two-Stage Multimodal Diarization & Gemini 4 Argon Cross-Examination (analyzeProjectMeeting)

5. Deep Dive into the Production Code: Long-Clip Truncation Recovery & Deterministic Roster Reconciliation

Production Code 2: Stage 1 Acoustic Diarization with Automatic Low-Volume & Long-Clip Truncation Recovery (projectMeetingFlows.js)

The 2-Stage Multimodal Prompts Powering analyzeProjectMeeting

1. Stage 1 Prompt — Multilingual Acoustic Speaker Diarization & Mobile Demo Video Transcription (gemini-3.5-transcribe-preview)

2. Stage 2 Prompt — Multimodal Roster Attribution & Weekly Log vs. Live Demo Cross-Examination (gemini-4-argon / gemini-3.8-flash)

Production Code 3: Stage 2 Deterministic Roster Reconciliation (reconcileMeetingRosterResults)

6. 1-Click Class-Wide HTML Email Dispatch & Longitudinal Student Portfolios

What's Next in Part 9?

Google AI Classroom Part 7: Serverless Map-Reduce-Map Video Intelligence & Rubric-Driven Practical Task Auto-Grading with Gemini 4 Argon

1. The 80-Hour Lab Grading Bottleneck & The HTTP 429 Quota Wall

2. Diagram 1: From Periodic Screenshots to Forensic 1-FPS H.264 MP4s (80% Storage & 90% Token Reduction)

Engineering Highlights of the Cloud Run FFmpeg Pipeline

3. Diagram 2: Two-Stage Map-Reduce-Map Cohort Video Discovery with Cloud Tasks & Gemini 4 Argon

Production Code 1: The Reduce Stage — Synthesizing a Cohort Grading Rubric from 40 Student Videos (generateLabTaskPrompt.js)

The 3 Prompts Powering the Map-Reduce-Map Video Pipeline

1. Phase 1 (Exploratory Map Prompt) — Unsupervised Activity Discovery (Summarize Student's Work.md)

2. Phase 2 (Reduce Meta-Prompt) — Synthesizing a Cohort Rubric from 40 Observations (Two-Stage Map-Reduce Lab Rubric Synthesizer.md)

3. Phase 3 (Second-Map Evaluation Prompt) — Example Output Synthesized by Gemini 4 Argon

4. Diagram 3: 4-Tab Practical Tasks, Retroactive Post-Class Slicing & Deterministic Rubric Reconciliation

5. Deep Dive into the Production Code: Rubric Extraction, Zero-Race Stream-Copy Slicing & Deterministic Score Reconciliation

Production Code 2: Multimodal Reference Video Rubric Synthesis (extractTaskDemoSteps.js)

Production Code 3: Eliminating the Post-Lesson Screenshot Deletion Race Condition with Ultrafast Stream-Copy Slicing (slicePendingTaskSubmissions)

Production Code 4: Deterministic Rubric Reconciliation & Mathematical Guardrails (evaluateTaskSubmission.js)

6. Architectural Comparison: Choosing the Right Gemini Model for Video Intelligence & Rubric Grading

What's Next in Part 8?

Google AI Classroom Part 6: Low-Bandwidth Classroom Screen Broadcasting & Single-Pass Whole-Lecture Subtitles with Gemini 4 Argon

1. Defining `AiModelOptions` and Deterministic Cache Keys

1.1. The `AiModelOptions` Interface

1.2. Private State and `generateKey()` in `AiModelFactory`

2. Creating Models On Demand and Invalidating the Cache in `AiModelFactory`

2.1. Lazily Loading the SDK and Assembling Parameters (`createModel`, `createModelParams`, `createGenerationConfig`)

2.2. Reusing Models On Demand (`getModel` and `getTtsModel`)

2.3. Evicting Cached Models When Remote Config Rotates the Backend (`getValidBackend`)

3. Refactoring `VisionService` and `TextToSpeechService` (and the Shallow-Spread Trap)

3.1. Simplifying `VisionService` for Future Multi-Model Routing

3.2. Reusing `#ttsModel` in `TextToSpeechService` Without Wiping Out `responseModalities`

Production Code 1: Live Google Cloud Billing Catalog SKU Sync (`syncGeminiPricing`) & Multi-Stage Cost Engine (`cost.js`)

3. Diagram 2: 9-Category Event-Driven Storage Quota & 4-Stage Cascading Class Purge (`onClassDocDeleted`)

Production Code 2: 9-Category Event-Driven Cloud Storage Quota Metering & Self-Healing Non-Negative Drift Guard (`storageQuota.js`)

Production Code 3: The 4-Stage Zero-Orphan Cascading Class Purge (`onClassDocDeleted` in `cleanupTriggers.js`)

How `./setup-new-project.sh` Eliminates Manual Cloud Console Clicks

Production Code 1: Genkit Structured Schema Generation & Fisher-Yates Positional De-Biasing (`bingoFlows.js`)

The 3 Prompts Powering Gamified Classroom Bingo (`bingoFlows.js`)

1. Question Bank Studio Prompt (`Bingo Question Bank Generator.md` / `generateBingoQuestionBank`)

2. Mode 2 Prompt — 1-to-Many Teacher Broadcast Screen + 5-Minute Live Subtitle Fusion + Anti-Duplication Rule (`resolveBingoQuestion`)

3. Mode 3 Prompt — Personalized Student Lab Screen Presence Verification (`Bingo Student Screen Presence Question Generator.md`)

Production Code 2: Event-Driven Two-Strike Cloud Tasks Scheduling & Elapsed AFK Deduction (`bingoFlows.js`)

4. Diagram 3: Zero-Trust Exam Mode Lockdown & Court-Ready `.docx` Forensic Dossiers

5. Deep Dive into the Zero-Trust Exam Code: Storage Metadata Locks, Signed URL Shielding & `.docx` Dossiers

Production Code 3: Write-Once Exam Metadata Locks in Cloud Storage (`storage.rules`) & Signed URL Broker (`getStudentVideoPlaybackUrl.js`)

Production Code 4: Court-Ready Microsoft Word (`.docx`) & RFC 4180 CSV Forensic Dossier Compiler (`processReportJob.js`)

The 3 Zero-Trust Exam Mode Invigilation Prompts Feeding the `.docx` Dossier

1. Screen Irregularity & Tool-Calling Invigilator (`admin/prompts/images/AI invigilator.md`)

2. Cloud Fallback Face & Gaze Invigilator (`admin/prompts/images/Cloud Fallback Face & Gaze Invigilator.md`)

3. Live Rolling Audio Collusion & Whisper Invigilator (`admin/prompts/audios/AI Voice Invigilator (Live Rolling Audio).md`)

Production Code 1: Pre-Meeting Fake-Log Detection & Live Code-Walkthrough Challenge Generator (`handlePrepareProjectLogQuestions`)

3. Diagram 2: Streamlined Single-Device Teacher Mobile Supervision Studio (Voice + `0.1s` Photo Snap + Auto-Sliced `720p` Demo Video)

4. Diagram 3: Two-Stage Multimodal Diarization & Gemini 4 Argon Cross-Examination (`analyzeProjectMeeting`)

Production Code 2: Stage 1 Acoustic Diarization with Automatic Low-Volume & Long-Clip Truncation Recovery (`projectMeetingFlows.js`)

The 2-Stage Multimodal Prompts Powering `analyzeProjectMeeting`

1. Stage 1 Prompt — Multilingual Acoustic Speaker Diarization & Mobile Demo Video Transcription (`gemini-3.5-transcribe-preview`)

2. Stage 2 Prompt — Multimodal Roster Attribution & Weekly Log vs. Live Demo Cross-Examination (`gemini-4-argon` / `gemini-3.8-flash`)

Production Code 3: Stage 2 Deterministic Roster Reconciliation (`reconcileMeetingRosterResults`)

Production Code 1: The Reduce Stage — Synthesizing a Cohort Grading Rubric from 40 Student Videos (`generateLabTaskPrompt.js`)

1. Phase 1 (Exploratory Map Prompt) — Unsupervised Activity Discovery (`Summarize Student's Work.md`)

2. Phase 2 (Reduce Meta-Prompt) — Synthesizing a Cohort Rubric from 40 Observations (`Two-Stage Map-Reduce Lab Rubric Synthesizer.md`)

Production Code 2: Multimodal Reference Video Rubric Synthesis (`extractTaskDemoSteps.js`)

Production Code 3: Eliminating the Post-Lesson Screenshot Deletion Race Condition with Ultrafast Stream-Copy Slicing (`slicePendingTaskSubmissions`)

Production Code 4: Deterministic Rubric Reconciliation & Mathematical Guardrails (`evaluateTaskSubmission.js`)

Implementation Walkthrough: 32×18 RGB Pixel Diffing, Hardware `ImageCapture`, & 950 KB Payload Guard

4. Single-Pass Audio-Only Whole-Lecture Subtitles with Gemini 4 Argon (`gemini-4-argon`)

1. Fixing Long-Audio Timescale Drift (`normalizeLectureAudio` & `calibrateSubtitleTimeline`)

2. The Chromium `<track>` Zero-Duration Cue Drop Guard (`buildWebVTT` & `buildSRT`)

3. Fixing Unseekable `MediaRecorder` WebM Videos (`ensureIndexedLectureVideo`)

1. Killing the #1 Cause of Bridge Strikes: Deterministic `ft-in` to `m` Parsing

3.1 Server Engine: `translateTeacherSpeech` with Rolling History & `thinkingBudget: 0`

3.2 Client Engine: $0.00 On-Device `Gemma 4 E2B` in Web Worker with Streaming CacheStorage

Sample 1: Full Assembled Server-Mode Prompt (`translateTeacherSpeech`) with 4-Turn Rolling History & 9-Language JSON Output

Sample 2: Cantonese-English Code-Switching Lecture Translator (`admin/prompts/translations/Cantonese-English Code-Switching Lecture Translator.md`)

Sample 3: Healthcare & Medical Sciences Clinical Lecture Translator (`admin/prompts/translations/Healthcare & Medical Sciences Clinical Lecture Translator.md`)

Sample 4: Business & Financial Accounting Lecture Translator (`admin/prompts/translations/Business & Financial Accounting Lecture Translator.md`)

Sample 5: Engineering & Applied Sciences Lecture Translator (`admin/prompts/translations/Engineering & Applied Sciences Lecture Translator.md`)

Sample 6: Client-Mode On-Device `Gemma 4 E2B` Prompt & Multimodal Audio Fallback STT Prompt

Edge Case 1: The Mobile iOS WebKit 1.9 GB `Jetsam OOM` Crash Guard

Edge Case 2: Whisper Silence Hallucinations (`"Thank you for watching"`, `"♪"`, Foreign Script Artifacts)

Edge Case 4: iOS Safari WebKit `AudioContext.resume()` Deadlock & Zero-Gain Pull Node

Edge Case 5: The Wireless Mic Battery Trap (`isMicSilentWarning` Watchdog)