flutter-edge-ai-onnx
Use when running ONNX models with flutter_edge_ai_onnx (ModelFileType.onnx) — ORT-GenAI text generation (e.g. Phi-3.5-mini) or ONNX embeddings — on macOS arm64, Linux x64, Windows x64 and ARM64, Android arm64, iOS arm64, or in the browser through Transformers.js. Also use when an ONNX install is routed to the wrong engine, genai_config.json is missing, or getActiveModel throws "No inference engine can handle this model" on another platform. For .litertlm models use flutter-edge-ai-inference; for .task, flutter-edge-ai-mediapipe.
- License 许可
- MIT
- 最近更新
- 2026-11-14
- Stars
- ★ 662
- Forks
- 180
01 前言 Front Matter
- name
- flutter-edge-ai-onnx
- description
- Use when running ONNX models with flutter_edge_ai_onnx (ModelFileType.onnx) — ORT-GenAI text generation (e.g. Phi-3.5-mini) or ONNX embeddings — on macOS arm64, Linux x64, Windows x64 and ARM64, Android arm64, iOS arm64, or in the browser through Transformers.js. Also use when an ONNX install is routed to the wrong engine, genai_config.json is missing, or getActiveModel throws "No inference engine can handle this model" on another platform. For .litertlm models use flutter-edge-ai-inference; for .task, flutter-edge-ai-mediapipe.
03 SKILL.md 正文
The ONNX engine
Rules
- Depend on
flutter_edge_aiandflutter_edge_ai_onnx, and import both. The engine package does not re-export core. - Declare
fileType: ModelFileType.onnx. Without it the install defaults totaskand never reachesOnnxEngine. - An ORT-GenAI model is a directory —
genai_config.json, the.onnxgraph, its weights and a tokenizer. Install it withfromHuggingFace(repo), which downloads the whole folder, or pointfromFileat a localgenai_config.json. A single-file download or a Flutter asset cannot produce it. - Native generation runs on macOS arm64, Linux x64, Windows x64, Windows ARM64 (verified on a Windows 11 ARM64 runner), Android arm64 and iOS arm64. On any other native host no engine accepts the model and
getActiveModelthrowsNo inference engine can handle this model. Web is a separate arm with its own rules (below). - Android needs
minSdk 24— the build hook fails the build below it. Phi-3.5-mini peaks near 3.7 GB of RAM, so target devices with 6 GB of RAM or more. - Text only: no images, no audio, no LoRA.
- Sampling follows the model's own config —
genai_config.jsonnatively,generation_config.jsonon web — which is greedy unless it setsdo_sample. Atemperature,topKortopPyou pass overrides it and switches sampling on;temperature: 0ortopK: 1keeps it greedy, natively, atemperatureortopPwithouttopKover a config withtop_k: 1also gets a top-k (the family's, else 40); on web the seed is not applied yet andtopPis ignored (Transformers.js samples with top-k only). On nativeactiveBackendisnull— ORT-GenAI picks the execution provider fromgenai_config.json, which the engine does not read; on web it reports the backend that loaded. - Response formats are enabled only on native macOS arm64 and only when the model directory has
tokenizer.json. Linux and Windows archives contain llguidance but stay refused until their host runs pass; Android, iOS and Web have no enabled constrained-decoding path.
Setup
flutter pub add flutter_edge_ai flutter_edge_ai_onnx flutter_edge_ai_embeddings
import 'package:flutter_edge_ai/flutter_edge_ai.dart';
import 'package:flutter_edge_ai_embeddings/flutter_edge_ai_embeddings.dart';
import 'package:flutter_edge_ai_onnx/flutter_edge_ai_onnx.dart';
await FlutterEdgeAi.initialize(
inferenceEngines: [OnnxEngine()],
embeddingBackends: [OnnxEmbeddingBackend()],
embeddingTokenizers: [GemmaEmbeddingTokenizers()],
);
await FlutterEdgeAi.installModel(
modelType: ModelType.phi,
fileType: ModelFileType.onnx,
).fromHuggingFace('microsoft/Phi-3.5-mini-instruct-onnx').install();
final InferenceModel model = await FlutterEdgeAi.getActiveModel(maxTokens: 4096);
A repo with several execution-provider folders resolves to its CPU/mobile folder automatically — the bundled runtime is CPU-only.
A bundle shipped with the app:
await FlutterEdgeAi.installModel(
modelType: ModelType.phi,
fileType: ModelFileType.onnx,
).fromFile('$path/genai_config.json').install();
Sessions, chats and streaming work as in the flutter-edge-ai-inference skill. openSession / openChat are available natively since flutter_edge_ai_onnx 0.8.0 and on Web since 0.9.0. On web, each session keeps its own history on the shared Transformers.js pipeline, holds no KV cache and is uncapped unless maxConcurrentSessions is set. Natively, each opened session has its own generator, whose KV cache is maxTokens × 2 × layers × KV heads × head size × dtype size once it has had a turn (0.75 MiB per token for Phi-3.5-mini int4); exceeding maxConcurrentSessions throws StateError (unset: 2 on Android and iOS, unlimited on desktop). FlutterEdgeAi.openModel remains unsupported (UnsupportedError on a supported host; StateError where no ONNX engine claims the spec), so use getActiveModel.
Structured output
Call model.refusalFor(format) before
openSession(responseFormat: format) or openChat(responseFormat: format). A
supported ONNX model advertises the whole core dialect and
StructuredDelivery.incremental; generated chunks arrive before an
IncompleteStructuredOutputException or
NonConformingStructuredOutputException. A prompt that does not fit fails with
ContextLimitException and is not kept. Never put the format on createSession
/ createChat, and never combine it with tools or thinking.
Web
On web OnnxEngine runs the model through Transformers.js. Install it by Hugging Face repo URL; the browser downloads and caches the files on first use:
await FlutterEdgeAi.installModel(
modelType: ModelType.qwen,
fileType: ModelFileType.onnx,
).fromNetwork('https://huggingface.co/onnx-community/Qwen2.5-0.5B-Instruct').install();
The repo must be in Transformers.js layout, as the onnx-community ones are. ORT-GenAI repos such as microsoft/Phi-3.5-mini-instruct-onnx do not run in the browser. fromFile and fromAsset install on web without complaint and then throw UnsupportedError from the first openSession — on web the model identity has to be a repo, not a file. fromBundled('<id>') serves one from the app's own origin. PreferredBackend.cpu forces WASM; anything else tries WebGPU first.
Add to web/index.html <head>, before Flutter boots — the first script for generation, the second for embeddings:
<script type="module">
window.transformersReady = (async () => {
const m = await import('https://cdn.jsdelivr.net/npm/@huggingface/transformers@4.3.1');
window.transformers = m;
return m;
})();
</script>
<script type="module">
window.ortReady = (async () => {
const m = await import('https://cdn.jsdelivr.net/npm/onnxruntime-web@1.30.0/dist/ort.bundle.min.mjs');
m.env.wasm.wasmPaths = 'https://cdn.jsdelivr.net/npm/onnxruntime-web@1.30.0/dist/';
window.ort = m;
return m;
})();
</script>
Embeddings
OnnxEmbeddingBackend handles single-file .onnx or .ort embedding models, installed with FlutterEdgeAi.installEmbedder() like any other — see the flutter-edge-ai-rag skill for the indexing flow.
On native it takes WordPiece (BERT-style, e.g. all-MiniLM-L6-v2) and SentencePiece (e.g. EmbeddingGemma-300M-ONNX) models and runs on CPU: activeBackend is PreferredBackend.cpu and preferredBackend is not applied. On web it takes WordPiece models only — all-MiniLM-L6-v2 works there, EmbeddingGemma-300M-ONNX is native-only.
ONNX embeddings are text-only: getActiveEmbedder(supportImage: true) or supportAudio: true throws EmbeddingLoadException. A graph with a fixed sequence length reports it as maxInputTokens, and a longer input — special tokens and task prefix included — throws EmbeddingInputTooLongException instead of being truncated. A dynamic-shape graph declares no limit: maxInputTokens is null.
04 同仓库更多 Skills
send-email
Send an email.
query-wikipedia
Query summary from Wikipedia for a given topic.
qr-code
Generates a QR code for the given url.
kitchen-adventure
Act as a dungeon master for a text-based adventure set in a world where everyone is a sentient kitchen appliance. Trigger when user says "start kitchen adventure".
interactive-map
Show an interactive map view for the given location.
get-current-time
Get the user's current local date, time, and day of the week.