AGENT SKILL

nodetool-model-provider-config

Configure NodeTool model providers and API keys (list_secrets, request_secret), pick a model with find_model, or set up local Ollama, llama.cpp and Hugging Face models.

使用场景图像与多媒体
License 许可
AGPL-3.0
最近更新
2026-11-15
Stars
★ 635
Forks
91

01 前言 Front Matter

name
nodetool-model-provider-config
featured
true
description
Configure NodeTool model providers and API keys (list_secrets, request_secret), pick a model with find_model, or set up local Ollama, llama.cpp and Hugging Face models.

03 SKILL.md 正文

You help users configure AI model providers and select the right models for their tasks.

In the product

The agent has no shell. Configure keys and pick models with these tools:

  1. list_secrets shows which credentials are configured. It never returns a value.
  2. request_secret {key, reason, help_url} opens a dialog where the user types a missing key. The value is saved from the dialog and never passes through the agent. Never ask the user to paste a key into chat. When the run has no interactive client, name the missing key and point the user to Settings > Secrets.
  3. find_model {capability} returns a real {provider, model_id} and a ref to assign verbatim to a node's model property. Call it before adding any generic AI node. Never write a model id from memory, including the examples below.
  4. list_provider_models {provider} lists one provider's language models.

Inside a code action the same calls are nodetool.secrets / nodetool.settings (load api-settings) and nodetool.models (load api-models).

The nodetool provider is NodeTool's managed service. Its models run on platform keys and draw from the user's credit balance, so it needs no key of the user's own. Every other provider bills the user's own account.

Provider Overview

Provider Type Key Env Var Models
OpenAI Cloud OPENAI_API_KEY GPT, GPT-Image, TTS, Whisper
Anthropic Cloud ANTHROPIC_API_KEY Claude Opus, Sonnet, Haiku
Gemini (Google) Cloud GEMINI_API_KEY Gemini, Veo, Nano Banana
xAI Cloud XAI_API_KEY Grok
Ollama Local OLLAMA_API_URL Qwen, Llama 3, Mistral, any GGUF. Separate install, must be running
HuggingFace Cloud HF_TOKEN Hosted Inference Providers
HuggingFace local (huggingface-local) Local — Diffusers/Transformers models via the HuggingFace Python pack
NodeTool (nodetool) Cloud, managed — (credits) Curated models
FAL (fal_ai) Cloud FAL_API_KEY Fast image/video generation
KIE (kie) Cloud KIE_API_KEY Image, video and music models from many vendors
ElevenLabs (elevenlabs) Cloud ELEVENLABS_API_KEY TTS, voice, sound effects
Replicate Cloud REPLICATE_API_TOKEN Community models
Comfy Cloud Cloud COMFY_API_KEY No model list; runs a ComfyUI workflow on Comfy's GPUs via lib.comfy.RunWorkflowOnCloud
vLLM Local VLLM_BASE_URL (required) Self-hosted, OpenAI-compatible
LM Studio Local LMSTUDIO_API_URL Models loaded in LM Studio's local server
llama.cpp server (llama_cpp) Local LLAMA_CPP_URL (required) A llama-server you run
llama.cpp local (node_llama_cpp) Local — GGUF files in-process; runtime from Package Manager → Engines & tools
MLX Local — MLX Python pack, Apple Silicon only

Other registered chat providers (any of these is valid for the CLI's -p/--provider): groq, mistral, deepseek, moonshot, sarvam (SARVAM_API_KEY, also Bulbul TTS and Saaras ASR), minimax, zai, cerebras, alibaba (DASHSCOPE_API_KEY, Qwen via Alibaba Cloud Model Studio), together, openrouter, requesty, opper, cheaperinference, api_route, runcrate, standardcompute, uberllm, codex, claude_agent_sdk, lmstudio, azure_openai (AZURE_OPENAI_API_KEY, AZURE_OPENAI_ENDPOINT, models from AZURE_OPENAI_DEPLOYMENTS), bedrock (BEDROCK_ACCESS_KEY_ID, BEDROCK_SECRET_ACCESS_KEY, BEDROCK_REGION).

From a shell (self-host)

These run in the user's terminal, not in the product. nodetool models providers shows configured providers, nodetool models recommended the curated list, and nodetool models by-provider <provider> one provider's ids.

# Via CLI (encrypted storage)
nodetool secrets store OPENAI_API_KEY
nodetool secrets store ANTHROPIC_API_KEY
nodetool secrets store GEMINI_API_KEY
nodetool secrets store HF_TOKEN
nodetool secrets store FAL_API_KEY
nodetool secrets store REPLICATE_API_TOKEN
nodetool secrets store COMFY_API_KEY

# Via environment variables
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
export GEMINI_API_KEY=AI...
export HF_TOKEN=hf_...
export FAL_API_KEY=...
export REPLICATE_API_TOKEN=r8_...
export COMFY_API_KEY=comfyui-...
export OLLAMA_API_URL=http://localhost:11434

Model Selection by Task

The tables name model families as starting points. find_model ranks what is actually available on this install and returns the exact id.

whisper.cpp speech recognition

Both providers are local-only and unavailable on the cloud profile. whisper_cpp runs speech recognition in the backend through the optional @fugood/whisper.node@1.1.3 runtime package. Install whisper.cpp from the Package Manager, then download a GGML model from Models. It serves nodetool.text.AutomaticSpeechRecognition and whisper_cpp.LiveTranscription. Use the absolute model path returned by discovery unchanged as the ASR model id.

Models use the Hugging Face hub cache (see HuggingFace models below for how its location resolves). WHISPER_CPP_MODELS_DIR adds another directory. WHISPER_CPP_GPU_BACKEND accepts auto, metal, cuda, vulkan, or cpu. auto uses Metal on macOS. On Windows and Linux it tries the CUDA build, then the Vulkan build, then the default CPU build. Restart the backend after changing the backend setting. Live transcription accepts base64 PCM16 mono chunks with an optional content_metadata.sample_rate and uses Silero VAD when installed, otherwise fixed windows.

whisper_cpp_server calls a user-run whisper-server. Set WHISPER_CPP_SERVER_URL to its base URL. The model id is default, representing the model loaded by that server. It requires no native runtime package.

Language / Chat

Need Family Provider Notes
Best quality GPT flagship, Claude Opus or Sonnet OpenAI, Anthropic Highest capability
Good balance GPT mini, Gemini Flash OpenAI, Gemini Fast + cheap
Local/private Llama, Qwen Ollama No data leaves machine
Lightweight local Small Llama, Mistral 7B Ollama Low memory
Code Claude Sonnet, GPT flagship Anthropic, OpenAI Best for coding

Image Generation

Need Model Provider Notes
Best quality FLUX.2 Dev HuggingFace, FAL State-of-art
Fast FLUX Schnell HuggingFace Quick iterations
Versatile SDXL HuggingFace Many LoRAs available
API-based GPT Image 2, Nano Banana OpenAI, Gemini/KIE No local GPU needed

Video Generation

Need Model Provider
Best quality Sora 2 Pro OpenAI (KIE)
Fast Wan 2.6 KIE
Image-to-video Kling 2.6 KIE
Talking avatar Kling AI Avatar KIE

Speech & Audio

Need Model Provider
TTS (quality) ElevenLabs ElevenLabs
TTS (fast/free) Kokoro HuggingFace pack, MLX pack, Transformers.js
ASR (accuracy) Whisper Large V3 HuggingFace pack, whisper.cpp
ASR (fast) Whisper Turbo HuggingFace pack, MLX pack, whisper.cpp

Embeddings

Need Model Provider
General text text-embedding-3-small OpenAI
Best quality text-embedding-3-large OpenAI
Local/free sentence-transformers HuggingFace

Local Model Setup

Ollama (Easiest)

Ollama is a separate program. NodeTool does not ship or start it, so it must be running before its models appear. On macOS and Windows, download it from ollama.com. On Linux:

# Install
curl -fsSL https://ollama.com/install.sh | sh

# Pull models
ollama pull llama3
ollama pull mistral
ollama pull qwen2

# Verify
ollama list

# NodeTool finds Ollama at 127.0.0.1:11434
# Override: OLLAMA_API_URL in Settings → Integrations → Local Model Servers,
# or export OLLAMA_API_URL=http://host:11434

OLLAMA_CONTEXT_LENGTH sets the context window sent with each request. Unset, NodeTool uses the model's own value, capped at 32768.

Python packs (HuggingFace local, MLX, Wan2GP)

Local Diffusers/Transformers models, MLX models and Wan2GP video run in the Python worker, and each needs its pack.

  • Desktop app: Tools → Package Manager → Python nodes. Installing a pack sets up Python first if it is missing. The HuggingFace pack pulls a PyTorch 2.14 build matched to the GPU. MLX is offered only on Apple Silicon Macs. On a Mac, both packs need Apple Silicon and macOS 14 or newer.
  • Without the desktop app: a Python 3.11+ env with pip install nodetool-core nodetool-huggingface (plus nodetool-mlx on Apple Silicon), then start the server with NODETOOL_PYTHON pointing at that env's python. curl -fsSL https://raw.githubusercontent.com/nodetool-ai/nodetool/main/install.sh | bash -s -- -y --pack huggingface builds that env on Linux and macOS. --pack takes huggingface, mlx, or wan2gp. On macOS, always set NODETOOL_PYTHON to the env it prints.
  • NODETOOL_TORCH_DEVICE (cuda, cuda:1, mps, cpu) picks the device.
  • Wan2GP nodes call a Wan2GP server the user runs. WAN2GP_MCP_URL (default http://127.0.0.1:7866/mcp) or the node's server_url points at it.

HuggingFace models

Download local models in the Model Manager before running the workflow. A node that downloads on first use can fail on a gated repo or stall a run.

For gated models:

  1. Accept the license on the model page at huggingface.co
  2. Store HF_TOKEN in Settings (or nodetool secrets store HF_TOKEN). Model Manager downloads use it
  3. Download the model in the Model Manager

Models go to the Hugging Face hub cache: HF_HUB_CACHE, else HUGGINGFACE_HUB_CACHE, else $HF_HOME/hub, else $XDG_CACHE_HOME/huggingface/hub, else ~/.cache/huggingface/hub. The server and the Python worker (through huggingface_hub) apply this one order. The desktop app keeps ~/.cache/huggingface when XDG_CACHE_HOME is set (as in Flatpak) but that cache already holds models.

llama.cpp

Two providers. node_llama_cpp loads GGUF files in-process after the runtime is installed from Package Manager → Engines & tools. llama_cpp calls a llama-server the user runs, at LLAMA_CPP_URL. Best for CPU inference and quantized models.

MODEL_FOLDERS (Settings → Folders) adds folders, such as a ComfyUI models folder, that node_llama_cpp and stable_diffusion_cpp search with their subfolders. llama.cpp lists the GGUF language models in them. stable-diffusion.cpp lists .safetensors checkpoints that bundle their VAE and text encoders. Separate folders with ;.

whisper.cpp and Transformers.js

whisper.cpp is speech recognition only (see above). Transformers.js runs small ONNX models in-process after its runtime is installed from Package Manager → Software. Its cache is TRANSFORMERS_JS_CACHE_DIR.

MLX (Apple Silicon only)

Needs the MLX pack and macOS 14 or newer. Lower memory usage than standard PyTorch.

Local Inference Performance

Framework Throughput Memory Hardware
llama.cpp Medium Excellent CPU, GPU
MLX Good Excellent Apple Silicon
Transformers Medium Good Any

Provider-Agnostic Nodes

These nodes work with any provider — just select the model:

Node Purpose
nodetool.agents.Agent Any LLM for chat/reasoning
nodetool.image.TextToImage Any image generation model
nodetool.image.ImageToImage Any image transformation model
nodetool.video.TextToVideo Any video generation model
nodetool.video.ImageToVideo Any image-to-video model
nodetool.audio.TextToSpeech Any TTS model
nodetool.text.AutomaticSpeechRecognition Any ASR model

Provider Capabilities

Capability OpenAI Anthropic Google Ollama HF
Chat/Text yes yes yes yes yes
Vision yes yes yes some yes
Image Gen yes (GPT-Image) no no no yes
Video Gen no no yes (Veo) no some
TTS yes no no no yes
ASR yes (Whisper) no no no yes
Embeddings yes no yes yes yes
Tool Calling yes yes yes some no

Common Pitfalls

  • Wrong key env var name: Each provider has a specific name (see table above)
  • Ollama not running: NodeTool does not start it. Start the Ollama app or ollama serve
  • Gated HF models: Must accept terms on huggingface.co first
  • GPU memory: Large models need 8-24GB VRAM; use quantized versions
  • PyTorch on the CPU despite a GPU: the desktop app installs CUDA 12.8 for Turing (RTX 20xx) and newer (driver 570+), CUDA 12.6 for Maxwell to Volta, and CPU for Kepler, pre-RDNA 2 AMD and AMD on Windows. It also falls back to CPU, with a warning, when no GPU build resolves. A manual pip install torch on Linux gets CUDA 13.0, which needs driver 580+ and Turing or newer, as does the Docker worker image
  • No local provider in the model menu: the Python pack is missing, or NODETOOL_PYTHON_ON_DEMAND=true and no workflow has started the worker yet
  • Rate limits: Cloud providers have rate limits; implement retries or use local
  • Model ID mismatch: Take ids from find_model or list_provider_models, never from memory. From a shell, nodetool models by-provider <provider> lists them
  • Key pasted into chat: Use request_secret. A key in a chat message is stored in the thread

Repository only: custom provider

This is TypeScript for contributors working in a NodeTool checkout. The in-product agent cannot add a provider. Providers extend BaseProvider from @nodetool-ai/runtime (not @nodetool-ai/core). Both generateMessage and generateMessages take a single args object.

import {
  BaseProvider,
  type ProviderId,
  type Message,
  type ProviderStreamItem,
  type ProviderTool,
  type LanguageModel,
} from "@nodetool-ai/runtime";

export class MyProvider extends BaseProvider {
  private apiKey: string;

  constructor(kwargs: Record<string, unknown> = {}) {
    super("my_provider" as ProviderId);
    this.apiKey = String(kwargs["MY_API_KEY"] ?? process.env.MY_API_KEY ?? "");
  }

  static override requiredSecrets(): string[] {
    return ["MY_API_KEY"];
  }

  // Non-streaming: return a single assistant Message.
  async generateMessage(args: {
    messages: Message[];
    model: string;
    tools?: ProviderTool[];
  }): Promise<Message> {
    // Call your API with args.messages / args.model …
    return { role: "assistant", content: "response text" };
  }

  // Streaming: yield ProviderStreamItem chunks.
  async *generateMessages(args: {
    messages: Message[];
    model: string;
    tools?: ProviderTool[];
  }): AsyncGenerator<ProviderStreamItem> {
    yield { type: "chunk", content: "response text", done: false };
  }

  override async getAvailableLanguageModels(): Promise<LanguageModel[]> {
    return [{ id: "my-model", name: "My Model", provider: "my_provider" }];
  }
}

Register it with registerProvider("my_provider", MyProvider) from @nodetool-ai/runtime.

04 同仓库更多 Skills

nodetool-ai
nodetool-ai/nodetool

humanizer

Rewrite AI-sounding text so it reads like the writer without changing what it says. Use when editing or reviewing prose for AI tells: not-X-but-Y contrasts, one-line closers, staged openers, forced triads, dashes everywhere, inflated claims, sales language, stock AI words, bold labels, or filler.

★ 635
nodetool-ai
nodetool-ai/nodetool

nodetool

Use the NodeTool MCP server to build, validate, run, and debug visual AI workflows, and to work with assets, collections, and jobs. Use when the user mentions NodeTool or asks for a workflow graph, a node, or a media generation run on their NodeTool server.

开发工程师Skill与智能体开发
★ 635
nodetool-ai
nodetool-ai/nodetool

wan-2-6-prompting

Prompt Alibaba's Wan 2.6 video — text-to-video with a global style line plus timing-bracketed shots, and image-to-video that describes only what changes over time. Covers the shot-mode switch per provider, the prompt budget, negative prompts, motion intensity wording, audio input and prompt expansion. Use whenever the model id names a Wan 2.6 video route (alibaba/wan-2.6/text-to-video, alibaba/wan-2.6/image-to-video, wan-video/wan-2.6-t2v, wan-video/wan-2.6-i2v, wan-video/wan2.6-i2v-flash) on generate_video, animate_image or a TextToVideo node. Not for Wan 2.5 or 2.7, or the Wan 2.6 image routes.

图像与多媒体
★ 635
nodetool-ai
nodetool-ai/nodetool

video-workflow

Build a reusable NodeTool video workflow for batches or repeated inputs. For a single finished piece, use a storyboard skill.

图像与多媒体Skill与智能体开发
★ 635
nodetool-ai
nodetool-ai/nodetool

video-clone

Analyze a reference video and recreate its shot structure with a new product or cast in NodeTool.

图像与多媒体
★ 635
nodetool-ai
nodetool-ai/nodetool

video-audio-continuity

Keep sound continuous across a multi-scene piece cut from generated video — why one clip per scene hard-cuts the audio at every boundary, when to write all the scenes into a single generation instead, the sound-design vocabulary that survives a provider's copyright filter, and how to check the length, the aspect ratio and the mix of what came back. Use when a piece has more than one scene and the video model writes its own audio (Seedance 2, Veo 3, MiniMax H3, Kling 2.6 and later), or when a cut's sound drops out at the shot changes.

图像与多媒体
★ 635