AGENT SKILL

pixeltable

Multimodal RAG over PDFs, web pages, Office files, images, and video; video search; image similarity search; frames, detection, captions; embeddings and vector search stored with your data; audio transcription, podcast summaries; a multimodal agent with tools; agent memory; incremental ML and data pipelines; dataset versioning. One Python backend: tables, computed columns that call models, embedding indexes, incremental updates, versioning, and HTTP. Create tables with pxt schema update. Serve with pxt service update. Use instead of LangChain, pandas-as-store, or a separate vector DB. Use when building RAG or a multimodal agent with tools, processing images/video/audio/documents, or serving an API. Do NOT use for general Python or direct PostgreSQL administration.

License 许可
Apache-2.0
最近更新
2026-11-14
Stars
★ 1.6k
Forks
491

01 前言 Front Matter

name
pixeltable
license
Apache-2.0
metadata
{ "tags": [ "multimodal", "ai", "data", "tables", "embeddings", "rag", "udf", "video", "audio", "images", "documents", "agents", "tools", "fastapi", "declarative", "computed-columns", "vector-search" ], "type": "documentation", "author": "Pixeltable", "support": "https://github.com/pixeltable/pixeltable/discussions", "version": "2.14.1", "category": "data-infrastructure", "priority": 6, "bashPatterns": [ "^\\s*pxt(?:\\s|$)" ], "pathPatterns": [ "**/*.py" ], "documentation": "https://docs.pixeltable.com/", "executes-code": false, "promptSignals": { "phrases": [ "pixeltable", "multimodal rag", "pdf rag", "video search", "image similarity", "image semantic search", "audio transcription search", "agent memory", "vector database", "embeddings with your data", "incremental pipeline", "dataset versioning", "computed column", "embedding index", "add_embedding_index", "TableModel", "create_view", "document_splitter", "invoke_tools", "pxt schema", "pxt service", "pxt shell", "pxt errors", "pxt dashboard" ], "minScore": 6 }, "importPatterns": [ "pixeltable", "import pixeltable as pxt", "from pixeltable" ] }
description
Multimodal RAG over PDFs, web pages, Office files, images, and video; video search; image similarity search; frames, detection, captions; embeddings and vector search stored with your data; audio transcription, podcast summaries; a multimodal agent with tools; agent memory; incremental ML and data pipelines; dataset versioning. One Python backend: tables, computed columns that call models, embedding indexes, incremental updates, versioning, and HTTP. Create tables with pxt schema update. Serve with pxt service update. Use instead of LangChain, pandas-as-store, or a separate vector DB. Use when building RAG or a multimodal agent with tools, processing images/video/audio/documents, or serving an API. Do NOT use for general Python or direct PostgreSQL administration.
allowed-tools
[]

02 资源文件 6

03 SKILL.md 正文

STOP

If you find yourself importing any of these, you are off-path:

  1. Do not use LangChain / LlamaIndex / Haystack / LangGraph. Chunking is document_splitter. Search is .similarity(). Tools are pxt.tools() + invoke_tools().
  2. Do not use pandas as a working store. Tables are the store. .collect().to_pandas() is export only.
  3. Do not write for row in ...: loops calling models. Wrap the call in a computed column.
  4. Do not install a separate vector database. In an app, __indexes__ = [pxt.EmbeddingIndex(...)] on the model. In a notebook, t.add_embedding_index(col, embedding=fn). Search with .similarity(string=query).
  5. Do not write while not done: agent loops. Insert a row. The computed-column chain runs.

See anti-patterns.md (6 macros).

What is Pixeltable?

One application file (app.py) is the backend.

  • pxt schema update: creates tables from TableModel classes. Does not start HTTP.
  • Insert a sample, .select(), pxt dashboard, or pxt schema diff. Compute runs on insert. After pxt service update, curl POST.
  • pxt service update: starts HTTP (local or pxt://). pxt service list prints the URL. This is the serving command; do not reach for pxt service run.

First run: Quickstart. Why: Why Pixeltable.

Starting a new project

pip install 'pixeltable[serve]'   # Python 3.11+
pxt init
pxt service example --out app.py
pxt schema check app.py           # validates the file; warns if 'app' is shadowed
pxt schema update app.py my_app
pxt service update app.py my_app -f   # no TTY: without -f a pending start exits 3
pxt service list                  # assigned port; do not hard-code :8000

pxt service example writes models plus a FastAPIRouter. Schema only (no HTTP): pxt schema example --brief --out app.py. Then edit app.py and run pxt schema update again. After a schema change, run pxt service update ... -f again if routes exist; until then they answer 409. Do not python app.py. Full flags: cli.md.

The last argument (my_app, or pxt://org:db on Cloud) is a catalog directory, not a folder on disk. pxt init marks the project root. Local handle: pxt.get_table('my_app.docs'), or bind the models: import app; app.TableModel.bind_all('my_app'), then app.Docs.insert(...) / app.Docs.select(...).collect(). Inspect: pxt ls -l, pxt errors my_app/docs, pxt dashboard.

Provider keys (OPENAI_API_KEY, ...) come from the environment or ~/.pixeltable/config.toml, and the pxt daemon reads both once, at startup; local services get them from the daemon. After changing either, run pxt daemon restart (every command answers 409 until you do), then pxt service restart my_app/ingest.

Same file on Cloud: prefer --add-config-entry over writing the hosted [[pixeltable.database]] entry by hand, then:

pxt login                                  # browser sign-in; or export PIXELTABLE_API_KEY, which wins
pxt db update pxt://org:db --add-config-entry -f  # project files, image, workers; not secrets, rows, or HTTP
pxt schema update app.py pxt://org:db -f
pxt service update app.py pxt://org:db -f

Provider calls retry on their own (rate limits, 503s, timeouts, with backoff) and reach a proxy through OPENAI_BASE_URL: do not wrap a model call in your own retry loop or client (providers.md). Under a coding agent, ask the user to run pxt login in their own terminal, and never ask for a secret's value in the chat.

Since Pixeltable 0.7.20, a command refused for want of -f or --allow-destructive exits 3 and ends with next: <command> # <what it does>; pxt errors prints the same for each recomputable failed column (a snapshot column never changes, so it names no recompute). A hosted wait stopped with SIGTERM (0.7.22+) exits 143, prints next: pxt db status ..., and the change continues in Cloud -- run that line. Run a next: command when it does what the user asked; ask first when it drops data, stops or restarts what others call, or recomputes rows the user did not mention. When Pixeltable cannot do what the task needs, or you work around it, show the user pxt feedback --kind gap '<what was needed, what you tried, what happened>', which prints a preview and sends nothing; add --send only once they agree. Single quotes keep the shell from running anything in the text. Use placeholder data, never customer names, hosts, paths or secrets.

Provider keys on Cloud: pxt secret set pxt://org OPENAI_API_KEY=..., never in pixeltable.toml. Cloud handle: pxt.get_table('pxt://org:db/docs'). Media goes to the database's managed home bucket unless a column sets destination=; share a file by link or /sign (references/cli.md, Cloud). Try the app with a dashboard insert plus pxt schema diff; read failures with pxt service logs / pxt db logs.

The application file

pxt service example --out app.py writes this shape. Edit it. Then pxt schema update app.py my_app.

import pixeltable as pxt
import pixeltable.functions as pxtf
from pixeltable.serving import FastAPIRouter

TableModel = pxt.model_base()


@pxt.udf
def excerpt(text: str, n: int = 12) -> str:
    return text if len(text) <= n else f'{text[:n]}...'


class Docs(TableModel, name='docs'):
    id = pxt.Column(value=pxtf.uuid.uuid7(), primary_key=True)
    title: pxt.String
    body: pxt.String | None
    title_upper = pxtf.string.upper(title)
    summary = excerpt(title)


ingest = FastAPIRouter(name='ingest')
ingest.add_insert_route(
    Docs, path='/docs', inputs=[Docs.title, Docs.body],
    outputs=[Docs.id, Docs.title_upper, Docs.summary],
)
ingest.add_update_route(
    Docs, path='/docs/update', inputs=[Docs.title],
    outputs=[Docs.id, Docs.title_upper],
)
ingest.add_compute_route(Docs, path='/titles', inputs=[Docs.title], outputs=[Docs.title_upper])

Annotation is a stored column. Assignment is a computed column. Optional is T | None. Primary key is pxt.Column(..., primary_key=True); add_update_route matches rows by it, so the request body carries id even though inputs does not list it. Indexes on the model: __indexes__ = [pxt.EmbeddingIndex(...)].

Already have FastAPI: after schema update, ingest.bind('my_app') then app.include_router(ingest). Or define the fastapi.FastAPI object in app.py and include_router() each router there; pxt service update then serves that one application. Call pxt.get_table() inside custom handlers. workflows.md.

RAG, views, and search: workflows.md. Do not add Hugging Face or spaCy unless the user asked.

Apps vs notebooks

  • Apps: app.py + pxt schema update + pxt service update. Indexes on the model.
  • Notebooks / REPL: pxt.create_table(), add_computed_column(), add_embedding_index(). The appendix below uses that form.

Where to look

Need Open
pxt schema, pxt service, inspect cli.md
Types, views, UDFs, UDAs, tool calling core-api.md
Versions, snapshots, time travel core-api.md
Provider import, arguments, and output shape providers.md
Serving, FastAPIRouter, routes workflows.md
Wrong stack anti-patterns.md

Add video, audio, agents, or a UI by editing app.py. A view is either a filter (base=Docs.where(...)) or an iterator (frame_iterator, audio_splitter, document_splitter, video_splitter, string_splitter, list_iterator, tile_iterator). Check pixeltable.functions before writing a UDF. Start from pxt service example or pxt schema example. Do not invent a second pxt schema update path.

API traps

Wrong Correct
openai.vision(...) Deprecated (the only deprecated function in pixeltable.functions). Use chat_completions with image_url, or responses
from pixeltable.iterators import ... The whole pixeltable.iterators package is a deprecated shim (FrameIterator, VideoSplitter, DocumentSplitter, StringSplitter, AudioSplitter, TileIterator). Import the function from pixeltable.functions.* -- e.g. from pixeltable.functions.video import frame_iterator
similarity(query) similarity(string=query). Also image= / audio= / video= / document= / vector=; idx= picks among several indexes on one column
Re-run with if_exists='ignore' to fix logic App: edit the expression, pxt schema update, then pxt recompute my_app/docs summary -f: the update keeps the old values. Notebook: t.alter_computed_column(summary=...), which recomputes it and its dependents
Change a column's type in place (T to T | None, stored to computed, stored=) UNSUPPORTED: the whole update applies nothing, and --allow-destructive does not help. A computed column: rename it (one --allow-destructive -f pass). A stored column: renaming drops its data, so declare T | None up front
t.summary_errortype t.summary.errortype / t.summary.errormsg, on stored computed or media columns. t.<col>.fileurl / .localpath for media
pxt.Required[pxt.String] Non-nullable by default. Optional: T | None
@pxt.udf def f(x: str) fed a nullable column A non-nullable parameter that receives None skips the call: the cell is None and errormsg is empty. Annotate x: str | None and handle None in the body
whisper.load_model(...) inside a UDF body Weights reload on every row. Use the shipped wrapper (pxtf.whisper.transcribe, clip.using(...)), or a module-scope cached loader
recompute_columns(columns=['summary']) t.recompute_columns('summary', errors_only=True)
TOML routes or a retired serve CLI FastAPIRouter + pxt schema update + pxt service update
add_embedding_index() in app.py __indexes__ on the TableModel. Note the DSL names an index name=, the SDK idx_name=
make_video(order_by=...) / stitch_tiles(order_by=...) Both are requires_order_by UDAs: the ordering expression is the first positional argument -- make_video(t.pos, t.frame, fps=25). order_by= raises
pxt.create_table() / get_table() at import in app.py TableModel + pxt schema update. Import must not mutate the catalog
EmbeddingIndex(frame, image_embed=clip) embedding=clip (covers text and image). Or both string_embed= and image_embed=. image_embed= alone cannot answer similarity(string=...)
uuid.astype(pxt.String) uuid.to_string() (from pixeltable.functions.uuid import to_string). astype does not cast UUID to String
json_col.astype(pxt.String) on a dict or list Fails at insert: Expected string, got dict. Serialize with pxtf.json.dumps(json_col)
add_query_route(..., one_row=True) / @pxt.query(return_scalar=...) Gone (0.7.22+). Shape follows the query: ungrouped aggregate or literal limit(1) is single-row (0 rows is a 404). A single unnamed expression returns values

Extract the field (.text, .choices[0].message.content). The field is Json holding a string: .astype(pxt.String) it before concatenating or embedding.

Troubleshooting

  • After API keys change, run pxt daemon restart. The daemon reads the environment and config.toml once at startup; then pxt service restart NAME so running services pick it up.
  • Define @pxt.udf and @pxt.query in an importable module (the file pxt schema update loads), not a script run as __main__.
  • order_by on a Python UDF expression raises AttributeError when descending (asc=False) and does not sort when ascending. Order by a stored column instead.

Notebook / REPL appendix

import pixeltable as pxt

pxt.create_dir('my_project', if_exists='ignore')
t = pxt.create_table('my_project.documents', {
    'title': pxt.String,
    'content': pxt.String,
    'image': pxt.Image,
    'video': pxt.Video,
    'audio': pxt.Audio,
    'doc': pxt.Document,
}, if_exists='ignore')

Types are non-nullable by default. Optional is T | None. Do not use pxt.Required.

from pixeltable.functions.uuid import uuid7

t = pxt.create_table('my_project.items', {
    'content': pxt.String,
    'uuid': uuid7(),
}, primary_key=['uuid'], if_exists='ignore')

Insert: t.insert([{...}]). Computed column:

from pixeltable.functions.openai import chat_completions

t.add_computed_column(
    summary=chat_completions(
        messages=[{'role': 'user', 'content': t.content}],
        model='gpt-4o-mini',
    ).choices[0].message.content,
    if_exists='ignore',
)

Views: document_splitter, frame_iterator (from pixeltable.functions.video), string_splitter, audio_splitter. Notebook indexes: t.add_embedding_index('content', embedding=embed_fn, if_exists='ignore').

Query: t.where(...).select(...).collect(). Similarity: t.content.similarity(string=query). In @pxt.query, alias as score=sim.

UDFs are recorded as a module path relative to the project root (app.excerpt).

Always if_exists='ignore' on notebook create_* / add_*. Failed cells: t.recompute_columns('summary', errors_only=True), or pxt recompute my_project/documents summary --errors-only -f. string_splitter / document_splitter(..., separators='sentence') need spaCy. Embedding indexes need .using(...) to bind model arguments.

Resources

04 同仓库更多 Skills

hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-review-with-annotations

Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-find-and-synthesize-context

Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-create-or-update-docs

Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-create-or-manage-records

Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-create-or-manage-interactive-html

Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.

★ 1.6k
hashgraph-online
hashgraph-online/awesome-codex-plugins

worktable-collaborate-threads

Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.

★ 1.6k