Flutter Edge AI #

On-device AI for Flutter on Android, iOS, Web, macOS, Windows and Linux. Run any model a registered engine supports (models), or the model the operating system already ships, with text, vision, audio, function calling, embeddings, RAG and speech — no server, no cloud. The core is small: you add only the runtimes and stores your app uses.

Formerly flutter_gemma. Through 1.11.3 this package shipped as flutter_gemma. Version 2.0 also moves RAG out of core into flutter_edge_ai_rag, and 2.3.0 changes the embedding API. See the migration guide.

📖 Full documentation: flutteredge.ai

Features #

• Pluggable engines: LiteRT-LM (.litertlm), MediaPipe (.task), ONNX Runtime, and the OS's own models (Gemini Nano, Apple Foundation Models, Windows AI Foundry, Chrome Prompt API).
• Multimodal: image and audio input on the models that support it.
• Function calling and thinking mode on the models that support them.
• CPU, GPU and NPU backends — Qualcomm NPU on Android and Linux arm64 (opt-in), Intel NPU on Windows.
• Embeddings and on-device RAG: text, image and audio embeddings (EmbeddingGemma 2 and 1, Gecko, ONNX models — models) with nine task types, stored in Qdrant Edge or SQLite + sqlite-vec, payload filters included.
• Speech: on-device STT, TTS and a push-to-talk voice loop.
• Agent skills: SKILL.md tools the model invokes through function calling.
• Model management: installs from the network, assets, bundled files or local paths; Hugging Face installs in one call; retries, typed download errors, LoRA weights (MediaPipe .task models).
• Genkit integration for on-device and hybrid cloud flows.

Packages #

Core registers no engine. Add flutter_edge_ai plus what your app needs:

Supported platforms #

¹ Web .litertlm takes text input only — an image or audio message throws UnsupportedError; function calling and thinking work.

² CPU only: a GPU .task does not load on iOS; use a .litertlm for the iOS GPU.

LiteRT-LM ships arm64 on Android, iOS and macOS, x86_64 on Windows and x86_64 / arm64 on Linux; on Android, MediaPipe .task also runs on x86_64 / armeabi-v7a, and on Linux ONNX is x86_64 only. Details, GPU backends and per-feature limits: installation.

Installation #

dependencies:
  flutter_edge_ai: ^2.4.0
  flutter_edge_ai_litertlm: ^1.14.0   # or any other engine from the table

Upgrade every flutter_edge_ai package together. Minimum versions with core 2.3.0: flutter_edge_ai_litertlm 1.13.0 · flutter_edge_ai_embeddings 2.3.0 · flutter_edge_ai_onnx 0.5.8 · flutter_edge_ai_mediapipe 1.2.2 · flutter_edge_ai_builtin_ai 0.3.5 · flutter_edge_ai_speech 0.5.6 · flutter_edge_ai_agent 0.2.9 · flutter_edge_ai_rag 1.1.0 · flutter_edge_ai_sqlite 2.1.0 · flutter_edge_ai_qdrant 2.1.0 · genkit_flutter_edge_ai 0.9.1. An older engine or store fails to compile against 2.3.0; see the migration guide. With core 2.4.0, getActiveModel(multimodalOrder:) and activeVisionBackend / activeAudioBackend need flutter_edge_ai_litertlm 1.14.0, and Genkit tool call ids need genkit_flutter_edge_ai 0.10.0; older versions compile against 2.4.0 but ignore them.

Then complete the platform setup below.

Platform setup #

iOS #

Set the deployment target to 15.0 (16.0 if you use flutter_edge_ai_mediapipe) and link pods statically in ios/Podfile:

platform :ios, '15.0'
use_frameworks! :linkage => :static

For large models add com.apple.developer.kernel.extended-virtual-addressing, com.apple.developer.kernel.increased-memory-limit and com.apple.developer.kernel.increased-debugging-memory-limit to Runner.entitlements. The iOS Simulator runs on CPU only.

Android #

Release builds need <uses-permission android:name="android.permission.INTERNET"/> in android/app/src/main/AndroidManifest.xml to download models (Flutter's template adds it only to the debug and profile manifests). Anything backed by LiteRT — .litertlm, LiteRT embeddings, speech — needs minSdk 30 and is arm64-v8a only. flutter_edge_ai_builtin_ai needs minSdk 26 (and macOS 12.0). The GPU and NPU uses-native-library entries still merge in from the plugin manifest automatically.

Web #

Copy cache_api.js and opfs_helper.js from this package's web/ folder into your app's web/ and load them in index.html, then add the script for each engine you use. For MediaPipe:

<script src="cache_api.js"></script>
<script src="opfs_helper.js"></script>
<script type="module">
import { FilesetResolver, LlmInference } from 'https://cdn.jsdelivr.net/npm/@mediapipe/tasks-genai@0.10.29';
window.FilesetResolver = FilesetResolver;
window.LlmInference = LlmInference;
</script>

The LiteRT-LM, ONNX, web embeddings and SQLite snippets are in the web setup guide. MediaPipe and LiteRT-LM run on WebGPU; ONNX also runs on CPU (WASM).

macOS #

macOS needs a build step that stages the LiteRT-LM companion libraries into the app (why), and it runs from a CocoaPods Podfile. Flutter 3.44+ uses Swift Package Manager by default, so a new app has no macos/Podfile: turn SPM off for the app in pubspec.yaml:

flutter:
  config:
    enable-swift-package-manager: false

Run flutter pub get (it writes macos/Podfile), then replace that Podfile's post_install block with the one below; the next build runs pod install itself. A green build proves nothing: without the block the first model load fails with Library not loaded: @rpath/libGemmaModelConstraintProvider.dylib. If your app already has a macos/Podfile (another plugin needs CocoaPods), keep SPM on and just paste the block. Prefer the pubspec setting to flutter config --no-enable-swift-package-manager, which changes only your machine — not your teammates' or CI.

post_install do |installer|
  installer.pods_project.targets.each do |target|
    flutter_additional_macos_build_settings(target)
  end

  # flutter_gemma: stage the upstream Apple companion dylibs into the built
  # .app. `hook/build.dart` deliberately skips them from Native Assets on macOS
  # (#247 — Google ships them without `-Wl,-headerpad_max_install_names`, so the
  # JIT bundling path cannot rewrite their install_name), which leaves this
  # build phase to stage them.
  #
  # The phase only LOCATES and RUNS a script; the staging logic itself lives in
  # flutter_gemma_litertlm and is delivered next to the dylibs it stages. That
  # is deliberate: this block is frozen into your Xcode project, and a copy of
  # the logic frozen there cannot be fixed by upgrading the package.
  installer.aggregate_targets.each do |aggregate_target|
    aggregate_target.user_targets.each do |user_target|
      phase_name = '[flutter_gemma] Setup LiteRT-LM macOS'

      # Only the app target embeds the Frameworks/ this phase patches.
      # RunnerTests inherits Runner's framework search paths and has no
      # Contents/Frameworks of its own — having the phase there creates a
      # cross-target dependency on Runner's framework output that Xcode reports
      # as "Cycle inside Flutter Assemble" (#300). Remove any stale copy from
      # non-app targets and skip them.
      unless user_target.name == 'Runner'
        user_target.build_phases
          .select { |p| p.respond_to?(:name) && p.name == phase_name }
          .each { |p| user_target.build_phases.delete(p) }
        next
      end

      existing = user_target.shell_script_build_phases.find { |p| p.name == phase_name }
      phase = existing || user_target.new_shell_script_build_phase(phase_name)
      # The embedded LiteRtLm binary is an INPUT so the phase re-runs whenever
      # Flutter's always-out-of-date `embed` phase re-copies the raw, unpatched
      # binary over the patched one. Without it Xcode caches the phase after the
      # first build and the second incremental build ships an unpatched
      # LiteRtLm that fails dlopen at runtime (#368).
      phase.input_paths = [
        '$(BUILT_PRODUCTS_DIR)/$(PRODUCT_NAME).app/Contents/Frameworks/LiteRtLm.framework/Versions/A/LiteRtLm',
      ]
      # A declared output lets Xcode order the phase in its dependency graph
      # instead of treating it as "runs every build with no outputs" — the other
      # half of the cycle warning (#300). The script touches this file.
      phase.output_paths = ['$(DERIVED_FILE_DIR)/flutter_gemma_litertlm_macos.stamp']
      phase.shell_script = <<~SHELL
        set -e
        STAGER="${HOME}/Library/Caches/flutter_gemma/native/macos_arm64/stage_macos_companions.sh"
        if [ ! -f "${STAGER}" ]; then
          echo "[flutter_gemma] ERROR: ${STAGER} not found." >&2
          echo "  flutter_gemma_litertlm 1.6.2+ installs it there from its build hook." >&2
          echo "  Upgrade the package, then: flutter clean && flutter pub get" >&2
          exit 1
        fi
        sh "${STAGER}" "${BUILT_PRODUCTS_DIR}/${PRODUCT_NAME}.app/Contents/Frameworks"
        mkdir -p "$(dirname "${SCRIPT_OUTPUT_FILE_0}")"
        touch "${SCRIPT_OUTPUT_FILE_0}"
      SHELL
    end
  end
end

Add to macos/Runner/DebugProfile.entitlements and Release.entitlements:

<key>com.apple.security.cs.disable-library-validation</key>
<true/>
<key>com.apple.security.network.client</key>
<true/>

Windows and Linux #

Nothing to add: the native libraries are downloaded and bundled at build time. GPU on Linux needs a vendor Vulkan driver (NVIDIA, AMD or Intel) — Mesa's llvmpipe is not enough for Gemma 4.

Quick start #

Initialization #

Register the engines you added, once, in main():

import 'package:flutter/widgets.dart';
import 'package:flutter_edge_ai/flutter_edge_ai.dart';
import 'package:flutter_edge_ai_litertlm/flutter_edge_ai_litertlm.dart';

Future<void> main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await FlutterEdgeAi.initialize(
    inferenceEngines: const [LiteRtLmEngine()],
    // Gated Hugging Face models need a token; never hard-code it.
    huggingFaceToken: const String.fromEnvironment('HUGGINGFACE_TOKEN').isEmpty
        ? null
        : const String.fromEnvironment('HUGGINGFACE_TOKEN'),
  );
  runApp(const MyApp());
}

Install a model #

Once per device; installed models survive restarts, fromFile ones included. Installing a name again from another source replaces it; core never deletes a file you installed with fromFile.

await FlutterEdgeAi.installModel(
  modelType: ModelType.gemmaIt,
  fileType: ModelFileType.litertlm, // selects the engine — declare it
).fromNetwork(
  'https://huggingface.co/litert-community/Gemma3-1B-IT/resolve/main/Gemma3-1B-IT_multi-prefill-seq_q4_ekv4096.litertlm',
).withProgress((progress) => print('Downloading: $progress%')).install();

litert-community/Gemma3-1B-IT is gated: accept its license on Hugging Face, then pass your token with --dart-define=HUGGINGFACE_TOKEN=….

Chat #

final model = await FlutterEdgeAi.getActiveModel(maxTokens: 2048);
final chat = await model.createChat();

await chat.addQueryChunk(Message.text(text: 'Explain quantum computing', isUser: true));

// Stream the reply token by token:
final reply = StringBuffer();
await for (final response in chat.generateChatResponseAsync()) {
  if (response is TextResponse) reply.write(response.token);
}

await model.close();

getActiveModel returns the same model while the active model and its arguments stay the same. Another model, or other arguments, builds a new one and closes the previous handle. A model is closed from the moment its close() starts (model.isClosed); getActiveModel waits for that close to finish, then builds a new model. With no model installed it throws NoActiveModelError, a StateError.

Sampling left unset comes from the model: the sampler a native .litertlm file ships, else the defaults its family publishes (SamplingParams.forModelType), else topK: 40. Pass topK: 1 for greedy decoding. What each engine reads, and why .litertlm keeps the first session's values: Sampling.

Decide (experimental) #

Pick one of N options — an intent, an action, a screen, a tool — for an input, or abstain when none fits. Every embedding model can do it:

// ignore_for_file: experimental_member_use
final embedder = await FlutterEdgeAi.getActiveEmbedder();
final decider = await embedder.createDecider(options: [
  DeciderOption('set_timer', examples: [
    EmbeddingInput.text('set a timer for ten minutes'),
    EmbeddingInput.text('wake me in half an hour'),
  ]),
  DeciderOption('call_contact', examples: [
    EmbeddingInput.text('call mum'),
    EmbeddingInput.text('ring Alex on his mobile'),
  ]),
]);
final answer = await decider.choose(
  DeciderInput(EmbeddingInput.text('remind me in 20 minutes')),
);
print(answer.choice ?? 'none of these (${answer.abstainReason})');

Options and inputs are embedded with TaskType.sentenceSimilarity; an option is the centroid of its examples (and description). The decider abstains when the input's best cosine is below a threshold cross-fitted on the options' own examples. Store a fitted decider with toArtifact() and restore it with loadDecider — pinned to its embedding space, so a CPU-fitted decider is never loaded onto an NPU bundle. Measured with EmbeddingGemma 2 on CLINC150 and HWU64, the numbers and their limits: Decisions. InferenceModel.createDecider(question:, options:, calibration:) is the language-model kind, with answers of the same shape; no engine scores options in this release, so it throws DeciderUnsupportedException.

Three things that fail quietly #

• maxTokens is the context window, not the reply length: prompt, history and answer together. .litertlm needs at least 1024. Cap the reply with createChat(maxOutputTokens: …) (.litertlm; MediaPipe ignores it).
• Message.isUser defaults to false. A user message without isUser: true gets an empty answer.
• fileType picks the engine, not the file name. installModel defaults to ModelFileType.task, so a .litertlm must say ModelFileType.litertlm.

Supported models #

Any model a registered engine runs. Which models run on which engine and platform, the ModelType to install each with, function calling, thinking, vision and audio support, and download links: models.

Embedding models: EmbeddingGemma 1 and Gecko (.tflite plus a tokenizer, every platform) and EmbeddingGemma 2 (one .litertlm file: text, images on the 440m and 740m bundles, audio on the 740m, and several parts in one vector with EmbeddingInput(text:, images:, audio:), natively and on the Web), all through flutter_edge_ai_litertlm; MiniLM-class .onnx models through flutter_edge_ai_onnx. Download links, sizes, formats per platform, embedding and speech models: models.

Going further #

• Getting started — sessions, system instructions, removing models
• Models — model sources, Hugging Face installs
• Function calling · Multimodal · Thinking mode
• Embeddings & RAG · Decisions · Speech · Agent skills
• Built-in AI · ONNX Runtime · Genkit · Memory diagnostics
• Desktop · Troubleshooting
• Fine-tune a small model and convert it to .litertlm with litetune (alpha).

Teach your AI assistant this package #

flutter_edge_ai ships agent skills that teach Claude Code, Codex, Cursor and other coding assistants this API:

dart run skills@ get --all

Links #

• Migration guide · Changelog · Desktop support
• Example app · Issues · Contributing