pixeltable
Multimodal RAG over PDFs, web pages, Office files, images, and video; video search; image similarity search; frames, detection, captions; embeddings and vector search stored with your data; audio transcription, podcast summaries; a multimodal agent with tools; agent memory; incremental ML and data pipelines; dataset versioning. One Python backend: tables, computed columns that call models, embedding indexes, incremental updates, versioning, and HTTP. Create tables with pxt schema update. Serve with pxt service update. Use instead of LangChain, pandas-as-store, or a separate vector DB. Use when building RAG or a multimodal agent with tools, processing images/video/audio/documents, or serving an API. Do NOT use for general Python or direct PostgreSQL administration.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-14
- Stars
- ★ 1.6k
- Forks
- 491
01 前言 Front Matter
- name
- pixeltable
- license
- Apache-2.0
- metadata
- { "tags": [ "multimodal", "ai", "data", "tables", "embeddings", "rag", "udf", "video", "audio", "images", "documents", "agents", "tools", "fastapi", "declarative", "computed-columns", "vector-search" ], "type": "documentation", "author": "Pixeltable", "support": "https://github.com/pixeltable/pixeltable/discussions", "version": "2.14.1", "category": "data-infrastructure", "priority": 6, "bashPatterns": [ "^\\s*pxt(?:\\s|$)" ], "pathPatterns": [ "**/*.py" ], "documentation": "https://docs.pixeltable.com/", "executes-code": false, "promptSignals": { "phrases": [ "pixeltable", "multimodal rag", "pdf rag", "video search", "image similarity", "image semantic search", "audio transcription search", "agent memory", "vector database", "embeddings with your data", "incremental pipeline", "dataset versioning", "computed column", "embedding index", "add_embedding_index", "TableModel", "create_view", "document_splitter", "invoke_tools", "pxt schema", "pxt service", "pxt shell", "pxt errors", "pxt dashboard" ], "minScore": 6 }, "importPatterns": [ "pixeltable", "import pixeltable as pxt", "from pixeltable" ] }
- description
- Multimodal RAG over PDFs, web pages, Office files, images, and video; video search; image similarity search; frames, detection, captions; embeddings and vector search stored with your data; audio transcription, podcast summaries; a multimodal agent with tools; agent memory; incremental ML and data pipelines; dataset versioning. One Python backend: tables, computed columns that call models, embedding indexes, incremental updates, versioning, and HTTP. Create tables with pxt schema update. Serve with pxt service update. Use instead of LangChain, pandas-as-store, or a separate vector DB. Use when building RAG or a multimodal agent with tools, processing images/video/audio/documents, or serving an API. Do NOT use for general Python or direct PostgreSQL administration.
- allowed-tools
- []
02 资源文件 6
- 参考文档agents/openai.yaml261 B
- 参考文档references/anti-patterns.md3.1 KB
- 参考文档references/cli.md26.2 KB
- 参考文档references/core-api.md20.1 KB
- 参考文档references/providers.md6.7 KB
- 参考文档references/workflows.md6.3 KB
03 SKILL.md 正文
STOP
If you find yourself importing any of these, you are off-path:
- Do not use LangChain / LlamaIndex / Haystack / LangGraph. Chunking is
document_splitter. Search is.similarity(). Tools arepxt.tools()+invoke_tools(). - Do not use pandas as a working store. Tables are the store.
.collect().to_pandas()is export only. - Do not write
for row in ...:loops calling models. Wrap the call in a computed column. - Do not install a separate vector database. In an app,
__indexes__ = [pxt.EmbeddingIndex(...)]on the model. In a notebook,t.add_embedding_index(col, embedding=fn). Search with.similarity(string=query). - Do not write
while not done:agent loops. Insert a row. The computed-column chain runs.
See anti-patterns.md (6 macros).
What is Pixeltable?
One application file (app.py) is the backend.
pxt schema update: creates tables fromTableModelclasses. Does not start HTTP.- Insert a sample,
.select(),pxt dashboard, orpxt schema diff. Compute runs on insert. Afterpxt service update, curl POST. pxt service update: starts HTTP (local orpxt://).pxt service listprints the URL. This is the serving command; do not reach forpxt service run.
First run: Quickstart. Why: Why Pixeltable.
Starting a new project
pip install 'pixeltable[serve]' # Python 3.11+
pxt init
pxt service example --out app.py
pxt schema check app.py # validates the file; warns if 'app' is shadowed
pxt schema update app.py my_app
pxt service update app.py my_app -f # no TTY: without -f a pending start exits 3
pxt service list # assigned port; do not hard-code :8000
pxt service example writes models plus a FastAPIRouter. Schema only (no HTTP): pxt schema example --brief --out app.py. Then edit app.py and run pxt schema update again. After a schema change, run pxt service update ... -f again if routes exist; until then they answer 409. Do not python app.py. Full flags: cli.md.
The last argument (my_app, or pxt://org:db on Cloud) is a catalog directory, not a folder on disk. pxt init marks the project root. Local handle: pxt.get_table('my_app.docs'), or bind the models: import app; app.TableModel.bind_all('my_app'), then app.Docs.insert(...) / app.Docs.select(...).collect(). Inspect: pxt ls -l, pxt errors my_app/docs, pxt dashboard.
Provider keys (OPENAI_API_KEY, ...) come from the environment or ~/.pixeltable/config.toml, and the pxt daemon reads both once, at startup; local services get them from the daemon. After changing either, run pxt daemon restart (every command answers 409 until you do), then pxt service restart my_app/ingest.
Same file on Cloud: prefer --add-config-entry over writing the hosted [[pixeltable.database]] entry by hand, then:
pxt login # browser sign-in; or export PIXELTABLE_API_KEY, which wins
pxt db update pxt://org:db --add-config-entry -f # project files, image, workers; not secrets, rows, or HTTP
pxt schema update app.py pxt://org:db -f
pxt service update app.py pxt://org:db -f
Provider calls retry on their own (rate limits, 503s, timeouts, with backoff) and reach a proxy through OPENAI_BASE_URL: do not wrap a model call in your own retry loop or client (providers.md). Under a coding agent, ask the user to run pxt login in their own terminal, and never ask for a secret's value in the chat.
Since Pixeltable 0.7.20, a command refused for want of -f or --allow-destructive exits 3 and ends with next: <command> # <what it does>; pxt errors prints the same for each recomputable failed column (a snapshot column never changes, so it names no recompute). A hosted wait stopped with SIGTERM (0.7.22+) exits 143, prints next: pxt db status ..., and the change continues in Cloud -- run that line. Run a next: command when it does what the user asked; ask first when it drops data, stops or restarts what others call, or recomputes rows the user did not mention. When Pixeltable cannot do what the task needs, or you work around it, show the user pxt feedback --kind gap '<what was needed, what you tried, what happened>', which prints a preview and sends nothing; add --send only once they agree. Single quotes keep the shell from running anything in the text. Use placeholder data, never customer names, hosts, paths or secrets.
Provider keys on Cloud: pxt secret set pxt://org OPENAI_API_KEY=..., never in pixeltable.toml. Cloud handle: pxt.get_table('pxt://org:db/docs'). Media goes to the database's managed home bucket unless a column sets destination=; share a file by link or /sign (references/cli.md, Cloud). Try the app with a dashboard insert plus pxt schema diff; read failures with pxt service logs / pxt db logs.
The application file
pxt service example --out app.py writes this shape. Edit it. Then pxt schema update app.py my_app.
import pixeltable as pxt
import pixeltable.functions as pxtf
from pixeltable.serving import FastAPIRouter
TableModel = pxt.model_base()
@pxt.udf
def excerpt(text: str, n: int = 12) -> str:
return text if len(text) <= n else f'{text[:n]}...'
class Docs(TableModel, name='docs'):
id = pxt.Column(value=pxtf.uuid.uuid7(), primary_key=True)
title: pxt.String
body: pxt.String | None
title_upper = pxtf.string.upper(title)
summary = excerpt(title)
ingest = FastAPIRouter(name='ingest')
ingest.add_insert_route(
Docs, path='/docs', inputs=[Docs.title, Docs.body],
outputs=[Docs.id, Docs.title_upper, Docs.summary],
)
ingest.add_update_route(
Docs, path='/docs/update', inputs=[Docs.title],
outputs=[Docs.id, Docs.title_upper],
)
ingest.add_compute_route(Docs, path='/titles', inputs=[Docs.title], outputs=[Docs.title_upper])
Annotation is a stored column. Assignment is a computed column. Optional is T | None. Primary key is pxt.Column(..., primary_key=True); add_update_route matches rows by it, so the request body carries id even though inputs does not list it. Indexes on the model: __indexes__ = [pxt.EmbeddingIndex(...)].
Already have FastAPI: after schema update, ingest.bind('my_app') then app.include_router(ingest). Or define the fastapi.FastAPI object in app.py and include_router() each router there; pxt service update then serves that one application. Call pxt.get_table() inside custom handlers. workflows.md.
RAG, views, and search: workflows.md. Do not add Hugging Face or spaCy unless the user asked.
Apps vs notebooks
- Apps:
app.py+pxt schema update+pxt service update. Indexes on the model. - Notebooks / REPL:
pxt.create_table(),add_computed_column(),add_embedding_index(). The appendix below uses that form.
Where to look
| Need | Open |
|---|---|
pxt schema, pxt service, inspect |
cli.md |
| Types, views, UDFs, UDAs, tool calling | core-api.md |
| Versions, snapshots, time travel | core-api.md |
| Provider import, arguments, and output shape | providers.md |
| Serving, FastAPIRouter, routes | workflows.md |
| Wrong stack | anti-patterns.md |
Add video, audio, agents, or a UI by editing app.py. A view is either a filter (base=Docs.where(...)) or an iterator (frame_iterator, audio_splitter, document_splitter, video_splitter, string_splitter, list_iterator, tile_iterator). Check pixeltable.functions before writing a UDF. Start from pxt service example or pxt schema example. Do not invent a second pxt schema update path.
API traps
| Wrong | Correct |
|---|---|
openai.vision(...) |
Deprecated (the only deprecated function in pixeltable.functions). Use chat_completions with image_url, or responses |
from pixeltable.iterators import ... |
The whole pixeltable.iterators package is a deprecated shim (FrameIterator, VideoSplitter, DocumentSplitter, StringSplitter, AudioSplitter, TileIterator). Import the function from pixeltable.functions.* -- e.g. from pixeltable.functions.video import frame_iterator |
similarity(query) |
similarity(string=query). Also image= / audio= / video= / document= / vector=; idx= picks among several indexes on one column |
Re-run with if_exists='ignore' to fix logic |
App: edit the expression, pxt schema update, then pxt recompute my_app/docs summary -f: the update keeps the old values. Notebook: t.alter_computed_column(summary=...), which recomputes it and its dependents |
Change a column's type in place (T to T | None, stored to computed, stored=) |
UNSUPPORTED: the whole update applies nothing, and --allow-destructive does not help. A computed column: rename it (one --allow-destructive -f pass). A stored column: renaming drops its data, so declare T | None up front |
t.summary_errortype |
t.summary.errortype / t.summary.errormsg, on stored computed or media columns. t.<col>.fileurl / .localpath for media |
pxt.Required[pxt.String] |
Non-nullable by default. Optional: T | None |
@pxt.udf def f(x: str) fed a nullable column |
A non-nullable parameter that receives None skips the call: the cell is None and errormsg is empty. Annotate x: str | None and handle None in the body |
whisper.load_model(...) inside a UDF body |
Weights reload on every row. Use the shipped wrapper (pxtf.whisper.transcribe, clip.using(...)), or a module-scope cached loader |
recompute_columns(columns=['summary']) |
t.recompute_columns('summary', errors_only=True) |
| TOML routes or a retired serve CLI | FastAPIRouter + pxt schema update + pxt service update |
add_embedding_index() in app.py |
__indexes__ on the TableModel. Note the DSL names an index name=, the SDK idx_name= |
make_video(order_by=...) / stitch_tiles(order_by=...) |
Both are requires_order_by UDAs: the ordering expression is the first positional argument -- make_video(t.pos, t.frame, fps=25). order_by= raises |
pxt.create_table() / get_table() at import in app.py |
TableModel + pxt schema update. Import must not mutate the catalog |
EmbeddingIndex(frame, image_embed=clip) |
embedding=clip (covers text and image). Or both string_embed= and image_embed=. image_embed= alone cannot answer similarity(string=...) |
uuid.astype(pxt.String) |
uuid.to_string() (from pixeltable.functions.uuid import to_string). astype does not cast UUID to String |
json_col.astype(pxt.String) on a dict or list |
Fails at insert: Expected string, got dict. Serialize with pxtf.json.dumps(json_col) |
add_query_route(..., one_row=True) / @pxt.query(return_scalar=...) |
Gone (0.7.22+). Shape follows the query: ungrouped aggregate or literal limit(1) is single-row (0 rows is a 404). A single unnamed expression returns values |
Extract the field (.text, .choices[0].message.content). The field is Json holding a string: .astype(pxt.String) it before concatenating or embedding.
Troubleshooting
- After API keys change, run
pxt daemon restart. The daemon reads the environment andconfig.tomlonce at startup; thenpxt service restart NAMEso running services pick it up. - Define
@pxt.udfand@pxt.queryin an importable module (the filepxt schema updateloads), not a script run as__main__. order_byon a Python UDF expression raisesAttributeErrorwhen descending (asc=False) and does not sort when ascending. Order by a stored column instead.
Notebook / REPL appendix
import pixeltable as pxt
pxt.create_dir('my_project', if_exists='ignore')
t = pxt.create_table('my_project.documents', {
'title': pxt.String,
'content': pxt.String,
'image': pxt.Image,
'video': pxt.Video,
'audio': pxt.Audio,
'doc': pxt.Document,
}, if_exists='ignore')
Types are non-nullable by default. Optional is T | None. Do not use pxt.Required.
from pixeltable.functions.uuid import uuid7
t = pxt.create_table('my_project.items', {
'content': pxt.String,
'uuid': uuid7(),
}, primary_key=['uuid'], if_exists='ignore')
Insert: t.insert([{...}]). Computed column:
from pixeltable.functions.openai import chat_completions
t.add_computed_column(
summary=chat_completions(
messages=[{'role': 'user', 'content': t.content}],
model='gpt-4o-mini',
).choices[0].message.content,
if_exists='ignore',
)
Views: document_splitter, frame_iterator (from pixeltable.functions.video), string_splitter, audio_splitter. Notebook indexes: t.add_embedding_index('content', embedding=embed_fn, if_exists='ignore').
Query: t.where(...).select(...).collect(). Similarity: t.content.similarity(string=query). In @pxt.query, alias as score=sim.
UDFs are recorded as a module path relative to the project root (app.excerpt).
Always if_exists='ignore' on notebook create_* / add_*. Failed cells: t.recompute_columns('summary', errors_only=True), or pxt recompute my_project/documents summary --errors-only -f. string_splitter / document_splitter(..., separators='sentence') need spaCy. Embedding indexes need .using(...) to bind model arguments.
Resources
- Quickstart
- CLI
- Developer MCP
- Cloud MCP (optional; reads and operates Cloud, see cli.md)
- Docs
04 同仓库更多 Skills
worktable-review-with-annotations
Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.
worktable-find-and-synthesize-context
Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.
worktable-create-or-update-docs
Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.
worktable-create-or-manage-records
Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.
worktable-create-or-manage-interactive-html
Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.
worktable-collaborate-threads
Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.