AGENT SKILL

flutter-edge-ai-onnx

Use when running ONNX models with flutter_edge_ai_onnx (ModelFileType.onnx) — ORT-GenAI text generation (e.g. Phi-3.5-mini) or ONNX embeddings — on macOS arm64, Linux x64, Windows x64 and ARM64, Android arm64, iOS arm64, or in the browser through Transformers.js. Also use when an ONNX install is routed to the wrong engine, genai_config.json is missing, or getActiveModel throws "No inference engine can handle this model" on another platform. For .litertlm models use flutter-edge-ai-inference; for .task, flutter-edge-ai-mediapipe.

License 许可
MIT
最近更新
2026-11-14
Stars
★ 662
Forks
180

01 前言 Front Matter

name
flutter-edge-ai-onnx
description
Use when running ONNX models with flutter_edge_ai_onnx (ModelFileType.onnx) — ORT-GenAI text generation (e.g. Phi-3.5-mini) or ONNX embeddings — on macOS arm64, Linux x64, Windows x64 and ARM64, Android arm64, iOS arm64, or in the browser through Transformers.js. Also use when an ONNX install is routed to the wrong engine, genai_config.json is missing, or getActiveModel throws "No inference engine can handle this model" on another platform. For .litertlm models use flutter-edge-ai-inference; for .task, flutter-edge-ai-mediapipe.

03 SKILL.md 正文

The ONNX engine

Rules

  1. Depend on flutter_edge_ai and flutter_edge_ai_onnx, and import both. The engine package does not re-export core.
  2. Declare fileType: ModelFileType.onnx. Without it the install defaults to task and never reaches OnnxEngine.
  3. An ORT-GenAI model is a directory — genai_config.json, the .onnx graph, its weights and a tokenizer. Install it with fromHuggingFace(repo), which downloads the whole folder, or point fromFile at a local genai_config.json. A single-file download or a Flutter asset cannot produce it.
  4. Native generation runs on macOS arm64, Linux x64, Windows x64, Windows ARM64 (verified on a Windows 11 ARM64 runner), Android arm64 and iOS arm64. On any other native host no engine accepts the model and getActiveModel throws No inference engine can handle this model. Web is a separate arm with its own rules (below).
  5. Android needs minSdk 24 — the build hook fails the build below it. Phi-3.5-mini peaks near 3.7 GB of RAM, so target devices with 6 GB of RAM or more.
  6. Text only: no images, no audio, no LoRA.
  7. Sampling follows the model's own config — genai_config.json natively, generation_config.json on web — which is greedy unless it sets do_sample. A temperature, topK or topP you pass overrides it and switches sampling on; temperature: 0 or topK: 1 keeps it greedy, natively, a temperature or topP without topK over a config with top_k: 1 also gets a top-k (the family's, else 40); on web the seed is not applied yet and topP is ignored (Transformers.js samples with top-k only). On native activeBackend is null — ORT-GenAI picks the execution provider from genai_config.json, which the engine does not read; on web it reports the backend that loaded.
  8. Response formats are enabled only on native macOS arm64 and only when the model directory has tokenizer.json. Linux and Windows archives contain llguidance but stay refused until their host runs pass; Android, iOS and Web have no enabled constrained-decoding path.

Setup

flutter pub add flutter_edge_ai flutter_edge_ai_onnx flutter_edge_ai_embeddings
import 'package:flutter_edge_ai/flutter_edge_ai.dart';
import 'package:flutter_edge_ai_embeddings/flutter_edge_ai_embeddings.dart';
import 'package:flutter_edge_ai_onnx/flutter_edge_ai_onnx.dart';

await FlutterEdgeAi.initialize(
  inferenceEngines: [OnnxEngine()],
  embeddingBackends: [OnnxEmbeddingBackend()],
  embeddingTokenizers: [GemmaEmbeddingTokenizers()],
);

await FlutterEdgeAi.installModel(
  modelType: ModelType.phi,
  fileType: ModelFileType.onnx,
).fromHuggingFace('microsoft/Phi-3.5-mini-instruct-onnx').install();

final InferenceModel model = await FlutterEdgeAi.getActiveModel(maxTokens: 4096);

A repo with several execution-provider folders resolves to its CPU/mobile folder automatically — the bundled runtime is CPU-only.

A bundle shipped with the app:

await FlutterEdgeAi.installModel(
  modelType: ModelType.phi,
  fileType: ModelFileType.onnx,
).fromFile('$path/genai_config.json').install();

Sessions, chats and streaming work as in the flutter-edge-ai-inference skill. openSession / openChat are available natively since flutter_edge_ai_onnx 0.8.0 and on Web since 0.9.0. On web, each session keeps its own history on the shared Transformers.js pipeline, holds no KV cache and is uncapped unless maxConcurrentSessions is set. Natively, each opened session has its own generator, whose KV cache is maxTokens × 2 × layers × KV heads × head size × dtype size once it has had a turn (0.75 MiB per token for Phi-3.5-mini int4); exceeding maxConcurrentSessions throws StateError (unset: 2 on Android and iOS, unlimited on desktop). FlutterEdgeAi.openModel remains unsupported (UnsupportedError on a supported host; StateError where no ONNX engine claims the spec), so use getActiveModel.

Structured output

Call model.refusalFor(format) before openSession(responseFormat: format) or openChat(responseFormat: format). A supported ONNX model advertises the whole core dialect and StructuredDelivery.incremental; generated chunks arrive before an IncompleteStructuredOutputException or NonConformingStructuredOutputException. A prompt that does not fit fails with ContextLimitException and is not kept. Never put the format on createSession / createChat, and never combine it with tools or thinking.

Web

On web OnnxEngine runs the model through Transformers.js. Install it by Hugging Face repo URL; the browser downloads and caches the files on first use:

await FlutterEdgeAi.installModel(
  modelType: ModelType.qwen,
  fileType: ModelFileType.onnx,
).fromNetwork('https://huggingface.co/onnx-community/Qwen2.5-0.5B-Instruct').install();

The repo must be in Transformers.js layout, as the onnx-community ones are. ORT-GenAI repos such as microsoft/Phi-3.5-mini-instruct-onnx do not run in the browser. fromFile and fromAsset install on web without complaint and then throw UnsupportedError from the first openSession — on web the model identity has to be a repo, not a file. fromBundled('<id>') serves one from the app's own origin. PreferredBackend.cpu forces WASM; anything else tries WebGPU first.

Add to web/index.html <head>, before Flutter boots — the first script for generation, the second for embeddings:

<script type="module">
window.transformersReady = (async () => {
  const m = await import('https://cdn.jsdelivr.net/npm/@huggingface/transformers@4.3.1');
  window.transformers = m;
  return m;
})();
</script>
<script type="module">
window.ortReady = (async () => {
  const m = await import('https://cdn.jsdelivr.net/npm/onnxruntime-web@1.30.0/dist/ort.bundle.min.mjs');
  m.env.wasm.wasmPaths = 'https://cdn.jsdelivr.net/npm/onnxruntime-web@1.30.0/dist/';
  window.ort = m;
  return m;
})();
</script>

Embeddings

OnnxEmbeddingBackend handles single-file .onnx or .ort embedding models, installed with FlutterEdgeAi.installEmbedder() like any other — see the flutter-edge-ai-rag skill for the indexing flow.

On native it takes WordPiece (BERT-style, e.g. all-MiniLM-L6-v2) and SentencePiece (e.g. EmbeddingGemma-300M-ONNX) models and runs on CPU: activeBackend is PreferredBackend.cpu and preferredBackend is not applied. On web it takes WordPiece models only — all-MiniLM-L6-v2 works there, EmbeddingGemma-300M-ONNX is native-only.

ONNX embeddings are text-only: getActiveEmbedder(supportImage: true) or supportAudio: true throws EmbeddingLoadException. A graph with a fixed sequence length reports it as maxInputTokens, and a longer input — special tokens and task prefix included — throws EmbeddingInputTooLongException instead of being truncated. A dynamic-shape graph declares no limit: maxInputTokens is null.

04 同仓库更多 Skills