cometchat-moderation
Trust & Safety for CometChat — auto-moderate text/image/video/file with dashboard rules + keyword lists, block or flag-for-review, let users report messages, and keep an audit trail. Cross-family: rules/lists/queues are dashboard+REST; report/block UI is built into each family's UI Kit. Triggers: 'add moderation', 'profanity filter', 'image moderation', 'block bad words', 'report a message', 'flag inappropriate content', 'content safety', 'spam/scam detection', 'CSAM/minor safety', 'malware scan attachments', 'moderation queue'.
- License 许可
- MIT
- 最近更新
- 2026-11-14
- Stars
- ★ 1.6k
- Forks
- 491
01 前言 Front Matter
- name
- cometchat-moderation
- license
- MIT
- metadata
- { "tags": "cometchat moderation trust-safety profanity image-moderation spam csam malware report flag audit", "author": "CometChat", "version": "1.0.0" }
- description
- Trust & Safety for CometChat — auto-moderate text/image/video/file with dashboard rules + keyword lists, block or flag-for-review, let users report messages, and keep an audit trail. Cross-family: rules/lists/queues are dashboard+REST; report/block UI is built into each family's UI Kit. Triggers: 'add moderation', 'profanity filter', 'image moderation', 'block bad words', 'report a message', 'flag inappropriate content', 'content safety', 'spam/scam detection', 'CSAM/minor safety', 'malware scan attachments', 'moderation queue'.
- compatibility
- CometChat Moderation (dashboard rules/lists + Flagged/Blocked/Reviewed queues) · Moderation REST API v3 · per-family UI Kit report/block + Chat SDK flagging.
03 SKILL.md 正文
Ground truth: capabilities and rule names are FETCHED from the live docs —
{DOCS_BASE}/moderation/overview,/moderation/{rules-management,lists-management,flagged-messages,blocked-messages,reviewed-messages},{DOCS_BASE}/rest-api/moderation-apis/overview, and the client hooks{DOCS_BASE}/ui-kit/<family>/core-features#moderation(built-in report) +{DOCS_BASE}/sdk/<platform>/ai-moderation(SDK flagging).DOCS_BASE = https://www.cometchat.com/docs; append.md. Most of this is dashboard configuration; verify the exact rule/list options and any REST shapes against the docs before asserting them.
Use this skill when
Making a chat safe/compliant: profanity/toxicity filtering, image/video safety, spam/scam detection, malware scanning of attachments, letting users report content, a moderator review queue, or an audit trail for T&S. Trust & Safety is a procurement gate for social, gaming, marketplace, dating, education and healthcare apps.
How it works
Every message runs through the rule engine as it's sent; a rule either lets it through, blocks it (hidden immediately), or flags it for a moderator. Users can also report a message manually, which flags it. Blocked/flagged items land in dashboard queues; a moderator's decision (approve/block/mark reviewed) moves it to the audit trail.
send → rules → pass → delivered
→ block → Blocked Messages
→ flag → Flagged Messages → moderator → Reviewed Messages (audit)
1. Configure rules + lists (dashboard / REST)
- Rules Management (
/moderation/rules-management): turn on the built-in rules you need and set each to block or flag. Available categories (fetch the current list — it grows):- Text: word-pattern match (profanity), contact/email removal, spam & scam, platform circumvention, toxicity, hate & harassment, self-harm, explicit content, privacy/sensitive-info, impersonation, violent/terroristic, non-consensual sexual content.
- Image: unsafe/prohibited, minor-safety/CSAM, graphic violence, explicit/sexual, hate symbols, fraud (fake IDs), malware scan.
- Video / Audio / File: malware & virus scan of attachments.
- Lists Management (
/moderation/lists-management): your own keyword / regex / sentence lists, linked to rules — this is the "banned words" surface. Lists are reusable across rules. - Programmatic setup: the Moderation REST API (
/rest-api/moderation-apis/overview) manages rules + keyword lists from your server (e.g. per-tenant config, CI-seeded rules). Fetch its shapes before writing calls.
2. Wire the client (per family — built into the UI Kit)
- Report a message: the UI Kits ship a built-in Report Message action on the message — enable it via that family's moderation feature (
{DOCS_BASE}/ui-kit/<family>/core-features#moderation; wire throughcometchat-<family>-features). No hand-rolled report UI. - Blocked content: rule-blocked messages simply don't render for recipients; nothing extra to build.
- SDK path (custom UI / headless): flag messages directly with the Chat SDK (
{DOCS_BASE}/sdk/<platform>/ai-moderation) — the per-feature fallback when a family has no drop-in.
3. Moderator review + audit
- Flagged Messages / Blocked Messages dashboards: moderators review, approve false positives, or confirm blocks — and use them to refine rules.
- Reviewed Messages is the audit trail of every moderation decision (who did what, when) for compliance records — pair with
cometchat-compliancefor retention/eDiscovery of these records.
Legacy extensions — migrate, don't double-run
If the app still uses the legacy extensions (Profanity Filter, Data Masking, Image Moderation), disable them before enabling the new moderation rules. Running both processes every message twice → performance issues and message delays. Treat this as a required migration step, not optional.
Enablement note
Moderation is a dashboard/plan capability — enable it on the app (and on the production app, not just dev). The CometChat MCP can turn extensions/AI on (cometchat_set_feature), list and toggle rules (cometchat_list_moderation_rules, cometchat_set_moderation_rule), create keyword lists (cometchat_create_moderation_list) and show flagged/blocked messages (cometchat_list_flagged_messages); detailed rule conditions are configured in the dashboard or via the Moderation REST API.
Common pitfalls
- Legacy extensions left on alongside new rules — double-processing, delays. Disable the old ones first.
- Rules configured on dev only — content ships unmoderated in production; enable on the production app.
- Hand-rolling a report button — the UI Kit already ships Report Message; wire the built-in.
- Block vs flag confused — block hides immediately (good for clear violations); flag routes to a human (good for borderline). Set each rule deliberately.
- No one watching the queues — flagged content needs a moderator workflow; wire notifications/ownership.
- Assuming a rule exists — fetch the current rule catalog; don't promise a category the docs don't list.
Verify it works
A message with a banned word/keyword is blocked or flagged per the rule · an unsafe image is caught · an attachment is malware-scanned · a user's Report action flags a message into the queue · a moderator decision appears in Reviewed Messages · legacy extensions are off · rules are enabled on the production app.
04 同仓库更多 Skills
worktable-review-with-annotations
Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.
worktable-find-and-synthesize-context
Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.
worktable-create-or-update-docs
Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.
worktable-create-or-manage-records
Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.
worktable-create-or-manage-interactive-html
Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.
worktable-collaborate-threads
Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.