[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6R5G3XPZ2GKGEWB2MZHSPHH":3,"skill-same:01M6R5G3XPZ2GKGEWB2MZHSPHH":90},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":30,"publishedAt":47,"linkOnly":48,"markdownContent":49,"markdownBody":50,"frontMatter":51,"refName":52,"commitSha":53,"skillDirectory":54,"forksCount":13,"repositoryHtmlUrl":55,"repositoryHomepageUrl":6,"ownerHtmlUrl":56,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":57,"install":58,"resources":71},"01M6R5G3XPZ2GKGEWB2MZHSPHH","model-debugging",null,"Debug and diagnose model errors in Pollinations services. Analyze logs, find error patterns, identify affected users.","APPLEPIE6969\u002FPollinations","APPLEPIE6969","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F242827480?v=4","01M6R5D0EAJN2DFJB3X6QTDANJ","01M6R5D0K2TSEDG15DK4M26BHH",0,"MIT","https:\u002F\u002Fgithub.com\u002FAPPLEPIE6969\u002FPollinations\u002Fblob\u002Fwan-aspect-ratio-fix\u002F.claude\u002Fskills\u002Fmodel-debugging\u002FSKILL.md",[17,22,25],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000010","后端与接口","SCENARIO","使用场景",{"categoryNo":23,"categoryName":24,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000016","图像与多媒体",{"categoryNo":26,"categoryName":27,"taxonomyType":28,"taxonomyTypeDesc":29},"01JBSEEDCAT000000000000033","数据分析师","PROFESSION","适用职业",[31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46],"debugging","model","Model Debugging","debug","diagnose","logs","services","analyze","find","pollinations","patterns","users","error","errors","affected","error patterns","2026-11-10 10:12:30",false,"---\nname: model-debugging\ndescription: Debug and diagnose model errors in Pollinations services. Analyze logs, find error patterns, identify affected users.\n---\n\n# Model Debugging Skill\n\nUse this skill when:\n- Investigating model failures, high error rates, or service issues\n- Finding users affected by errors (402 billing, 403 permissions, 500 backend)\n- Analyzing Tinybird\u002FCloudflare logs for patterns\n- Diagnosing specific request failures\n\n# Understanding Model Monitor Error Rates\n\n**Why does the Model Monitor show high error rates when models work fine manually?**\n\nThe Model Monitor at https:\u002F\u002Fmonitor.pollinations.ai shows **all real-world traffic**, including:\n\n- **401 errors**: Anonymous users without API keys (most common)\n- **402 errors**: Users with insufficient pollen balance or exhausted API key budget\n- **403 errors**: Users denied access to specific models (API key restrictions)\n- **400 errors**: Invalid request parameters (e.g., `openai-audio` without `modalities` param)\n- **429 errors**: Rate-limited requests\n- **500\u002F504 errors**: Actual backend failures (investigate these)\n\nWhen you test manually with a valid secret key (`sk_`), you bypass auth\u002Fquota issues, so models appear to work fine.\n\n**Key insight**: High 401\u002F402\u002F403\u002F400 rates are **expected** from real-world usage. Focus investigation on 500\u002F504 errors.\n\n---\n\n# Data Flow Architecture\n\n```\nUser Request → enter.pollinations.ai (Cloudflare Worker)\n                    ↓\n              Logs to Cloudflare Workers Observability\n                    ↓\n              Events stored in D1 database\n                    ↓\n              Batched to Tinybird (async, 100-500 events)\n                    ↓\n              Model Monitor queries Tinybird (model_health.pipe)\n```\n\n**Structured Logging**: enter.pollinations.ai uses LogTape with:\n- `requestId`: Unique per request (passed to downstream via `x-request-id` header)\n- `status`, `body`: Full error response from downstream services\n- Context: `method`, `routePath`, `userAgent`, `ipAddress`\n\n---\n\n# Quick Diagnostics\n\n## 1. Check Model Monitor\nView current model health at: https:\u002F\u002Fmonitor.pollinations.ai\n\n## 2. Query Recent Errors from D1 Database\n```bash\n# Via enter.pollinations.ai worker (requires wrangler)\ncd enter.pollinations.ai\nnpx wrangler d1 execute pollinations-db --remote --command \"SELECT model_requested, response_status, error_message, COUNT(*) as count FROM event WHERE response_status >= 400 AND created_at > datetime('now', '-1 hour') GROUP BY model_requested, response_status, error_message ORDER BY count DESC LIMIT 20\"\n```\n\n## 3. Capture Live Logs\n\n### enter.pollinations.ai (Cloudflare Worker)\n```bash\ncd enter.pollinations.ai\nwrangler tail --format json | tee logs.jsonl\n# Or with formatting:\nwrangler tail --format json | npx tsx scripts\u002Fformat-logs.ts\n```\n\n### gen.pollinations.ai (image + text gateway)\nImage and text generation now run inside the gen Cloudflare Worker (the legacy EC2 `image-pollinations` and `text-pollinations` services are decommissioned). Use `wrangler tail` from `gen.pollinations.ai\u002F`:\n```bash\ncd gen.pollinations.ai\nwrangler tail --format json | tee gen-logs.jsonl\n```\n\n### Legacy anonymous image (OVH)\nAnonymous traffic to `image.pollinations.ai` still terminates on the OVH host:\n```bash\n# Real-time logs\nssh -i ~\u002F.ssh\u002Fid_rsa_ovh ubuntu@57.130.31.42 \"sudo journalctl -u image-pollinations -f\"\n\n# Last 3 minutes\nssh -i ~\u002F.ssh\u002Fid_rsa_ovh ubuntu@57.130.31.42 \"sudo journalctl -u image-pollinations --since '3 minutes ago' --no-pager\" > legacy-image-logs.txt\n```\n\n---\n\n# Common Error Patterns\n\n## Azure Content Safety DNS Failure\n**Error**: `getaddrinfo ENOTFOUND gptimagemain1-resource.cognitiveservices.azure.com`\n**Cause**: Azure Content Safety resource deleted or misconfigured\n**Impact**: Fail-open (content proceeds without safety check)\n**Fix**: Create new Azure Content Safety resource and update `.env`:\n```\nAZURE_CONTENT_SAFETY_ENDPOINT=https:\u002F\u002F\u003Cnew-resource>.cognitiveservices.azure.com\u002F\nAZURE_CONTENT_SAFETY_API_KEY=\u003Cnew-key>\n```\n\n## Azure Kontext Content Filter\n**Error**: `Content rejected due to sexual\u002Fhate\u002Fviolence content detection`\n**Cause**: Azure's content moderation blocking prompts\u002Fimages\n**Impact**: 400 error returned to user\n**Fix**: User error - prompt violates content policy\n\n## Vertex AI Invalid Image\n**Error**: `Provided image is not valid`\n**Cause**: User passing unsupported image URL (e.g., Google Drive links)\n**Impact**: 400 error returned to user\n**Fix**: User error - need direct image URL\n\n## Translation Service Down\n**Error**: `No active translate servers available`\n**Cause**: Translation service unavailable\n**Impact**: Prompts not translated (non-fatal)\n**Fix**: Check translation service status\n\n## OpenAI Audio Invalid Voice\n**Error**: `Invalid value for audio.voice`\n**Cause**: User requesting unsupported voice name\n**Impact**: 400 error returned to user\n**Fix**: User error - use supported voices: alloy, echo, fable, onyx, nova, shimmer, coral, verse, ballad, ash, sage, etc.\n\n## Oversized Text Seed Surfaced as 500\n**Error**: `'seed' must be Integer`, `invalid request error`, or a generic upstream 500\n**Cause**: A client sent a seed above signed INT32 max (`2147483647`) to a strict provider\n**Impact**: The provider may misclassify invalid client input as 500, inflating model health errors\n**Fix**: Reject oversized seeds as 400 at gateway validation; group incidents by user, API key, and request shape before treating them as a model outage\n\n## Veo No Video Data\n**Error**: `No video data in response`\n**Cause**: Vertex AI returned empty video response\n**Impact**: 500 error\n**Fix**: Check Vertex AI quota\u002Fstatus, may be transient\n\n---\n\n# Environment Variables to Check\n\nImage and text env vars now live in the gen Worker secrets (`gen.pollinations.ai\u002Fsecrets\u002F{dev,staging,prod}.vars.json`, SOPS-encrypted). Decrypt to inspect:\n```bash\nsops -d gen.pollinations.ai\u002Fsecrets\u002Fprod.vars.json | jq 'keys[] | select(test(\"AZURE|GOOGLE|CLOUDFLARE|OPENAI\"))'\n```\n\nKey variables:\n- `AZURE_CONTENT_SAFETY_ENDPOINT` - Azure Content Safety API endpoint\n- `AZURE_CONTENT_SAFETY_API_KEY` - Azure Content Safety API key\n- `GOOGLE_PROJECT_ID` - Google Cloud project for Vertex AI\n- `AZURE_MYCELI_PROD_SWEDEN_API_KEY` - Shared Azure API key (Kontext, GPT Image, GPT Image 1.5)\n\n---\n\n# Updating Secrets\n\nSecrets are stored encrypted with SOPS:\n- `gen.pollinations.ai\u002Fsecrets\u002F{dev,staging,prod}.vars.json`\n- `enter.pollinations.ai\u002Fsecrets\u002F{dev,staging,prod}.vars.json`\n\nTo update:\n```bash\n# Decrypt, edit, re-encrypt\nsops gen.pollinations.ai\u002Fsecrets\u002Fprod.vars.json\n\n# Deploy to the gen Worker (secrets ship with the deploy)\ncd gen.pollinations.ai && npm run deploy\n```\n\n---\n\n# Log Analysis Commands\n\n```bash\n# Count errors by type (against captured wrangler-tail JSON)\njq -r '.logs[]?.message[]? \u002F\u002F .message? \u002F\u002F empty' gen-logs.jsonl | grep -oE \"(Azure Flux Kontext|Vertex AI|No active translate|getaddrinfo ENOTFOUND)\" | sort | uniq -c | sort -rn\n\n# Find content filter rejections\njq -r '.logs[]?.message[]? \u002F\u002F .message? \u002F\u002F empty' gen-logs.jsonl | grep -i \"Content rejected\" | sort | uniq -c\n```\n\n---\n\n# Model-Specific Debugging\n\n| Model | Backend | Common Issues |\n|-------|---------|---------------|\n| `flux` | Azure\u002FReplicate | Rate limits, content filter |\n| `kontext` | Azure Flux Kontext | Content filter (strict) |\n| `nanobanana` | Vertex AI Gemini | Invalid image URLs, content filter |\n| `seedream-pro` | ByteDance ARK | NSFW filter, API key issues |\n| `veo` | Vertex AI | Quota, empty responses |\n| `openai-audio` | Azure OpenAI | Invalid voice names |\n| `deepseek` | DeepSeek API | Rate limits, API key |\n\n---\n\n# Cloudflare Workers Observability API\n\nThe enter.pollinations.ai worker has structured logging enabled. You can query logs programmatically via the Cloudflare Workers Observability API.\n\n## Prerequisites\n\n### 1. Get Account ID\n```bash\n# From wrangler.toml\ngrep account_id enter.pollinations.ai\u002Fwrangler.toml\n\n# Or from existing .env\ngrep CLOUDFLARE_ACCOUNT_ID image.pollinations.ai\u002F.env\n```\n\n### 2. Create API Token with Workers Observability Permission\n\n**Via Cloudflare Dashboard:**\n1. Go to https:\u002F\u002Fdash.cloudflare.com\u002Fprofile\u002Fapi-tokens\n2. Click **Create Token**\n3. Click **Create Custom Token**\n4. Configure:\n   - **Token name**: `Workers Observability Read`\n   - **Permissions**: \n     - Account → Workers Scripts → Read\n     - Account → Workers Observability → Edit (required for query API)\n   - **Account Resources**: Include → Your Account\n5. Click **Continue to summary** → **Create Token**\n6. Copy the token immediately (shown only once)\n\n### 3. Store Token Securely\n\nThe token is stored in SOPS-encrypted secrets:\n- **Location**: `enter.pollinations.ai\u002Fsecrets\u002Fenv.json`\n- **Key**: `CLOUDFLARE_OBSERVABILITY_TOKEN`\n\nTo add\u002Fupdate:\n```bash\n# Step 1: Decrypt to temp file\ncd \u002Fpath\u002Fto\u002Fpollinations\nsops -d enter.pollinations.ai\u002Fsecrets\u002Fenv.json > \u002Ftmp\u002Fenv.json\n\n# Step 2: Add the token (use jq)\njq '. + {\"CLOUDFLARE_OBSERVABILITY_TOKEN\": \"your_token\"}' \u002Ftmp\u002Fenv.json > \u002Ftmp\u002Fenv_updated.json\n\n# Step 3: Re-encrypt (must rename to match .sops.yaml pattern)\ncp \u002Ftmp\u002Fenv_updated.json \u002Ftmp\u002Fenv.json\nsops -e \u002Ftmp\u002Fenv.json > enter.pollinations.ai\u002Fsecrets\u002Fenv.json\n\n# Step 4: Cleanup\nrm \u002Ftmp\u002Fenv.json \u002Ftmp\u002Fenv_updated.json\n\n# Verify\nsops -d enter.pollinations.ai\u002Fsecrets\u002Fenv.json | jq 'keys'\n```\n\n**Note**: The `.sops.yaml` config requires filenames matching `env.json$` pattern.\n\n## API Endpoint\n\n```\nPOST https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F{account_id}\u002Fworkers\u002Fobservability\u002Ftelemetry\u002Fquery\n```\n\n## Query Examples\n\n### Setup: Get Credentials from SOPS\n\n```bash\n# Extract credentials from encrypted secrets\nACCOUNT_ID=$(sops -d enter.pollinations.ai\u002Fsecrets\u002Fenv.json | jq -r '.CLOUDFLARE_ACCOUNT_ID')\nAPI_TOKEN=$(sops -d enter.pollinations.ai\u002Fsecrets\u002Fenv.json | jq -r '.CLOUDFLARE_OBSERVABILITY_TOKEN')\n```\n\n### List Available Log Keys (Working)\n\nThis endpoint works and shows what fields are available:\n\n```bash\ncurl -s \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F$ACCOUNT_ID\u002Fworkers\u002Fobservability\u002Ftelemetry\u002Fkeys\" \\\n  -H \"Authorization: Bearer $API_TOKEN\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\"timeframe\": {\"from\": '$(( $(date +%s) - 86400 ))'000, \"to\": '$(date +%s)'000}, \"datasets\": [\"workers\"]}' | jq '.result[:10]'\n```\n\n### Query Recent Errors (Last 15 Minutes)\n\n**Note**: The `\u002Fquery` endpoint requires a saved `queryId`. For ad-hoc queries, use the Cloudflare Dashboard Query Builder or `wrangler tail`.\n\n```bash\n# This format requires a saved query ID\n\n# Query errors with status >= 400\ncurl -s \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F$ACCOUNT_ID\u002Fworkers\u002Fobservability\u002Ftelemetry\u002Fquery\" \\\n  -H \"Authorization: Bearer $API_TOKEN\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"timeframe\": {\n      \"from\": '$(( $(date +%s) - 900 ))'000,\n      \"to\": '$(date +%s)'000\n    },\n    \"parameters\": {\n      \"datasets\": [\"workers\"],\n      \"filters\": [\n        {\"key\": \"$workers.scriptName\", \"operation\": \"eq\", \"type\": \"string\", \"value\": \"enter-pollinations-ai\"},\n        {\"key\": \"$metadata.statusCode\", \"operation\": \"gte\", \"type\": \"number\", \"value\": 400}\n      ],\n      \"calculations\": [{\"operator\": \"count\"}],\n      \"groupBys\": [\n        {\"type\": \"string\", \"value\": \"$metadata.statusCode\"},\n        {\"type\": \"string\", \"value\": \"$metadata.error\"}\n      ],\n      \"limit\": 50\n    }\n  }' | jq '.result.events.events[:20]'\n```\n\n### Query Errors by Model\n\n```bash\ncurl -s \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F$ACCOUNT_ID\u002Fworkers\u002Fobservability\u002Ftelemetry\u002Fquery\" \\\n  -H \"Authorization: Bearer $API_TOKEN\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"timeframe\": {\n      \"from\": '$(( $(date +%s) - 3600 ))'000,\n      \"to\": '$(date +%s)'000\n    },\n    \"parameters\": {\n      \"datasets\": [\"workers\"],\n      \"filters\": [\n        {\"key\": \"$workers.scriptName\", \"operation\": \"eq\", \"type\": \"string\", \"value\": \"enter-pollinations-ai\"},\n        {\"key\": \"$metadata.statusCode\", \"operation\": \"gte\", \"type\": \"number\", \"value\": 400}\n      ],\n      \"calculations\": [{\"operator\": \"count\"}],\n      \"groupBys\": [\n        {\"type\": \"string\", \"value\": \"model\"},\n        {\"type\": \"string\", \"value\": \"$metadata.statusCode\"}\n      ],\n      \"limit\": 100\n    }\n  }' | jq '.result.calculations[0].aggregates'\n```\n\n### Get Raw Error Events with Full Details\n\n```bash\ncurl -s \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F$ACCOUNT_ID\u002Fworkers\u002Fobservability\u002Ftelemetry\u002Fquery\" \\\n  -H \"Authorization: Bearer $API_TOKEN\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"timeframe\": {\n      \"from\": '$(( $(date +%s) - 900 ))'000,\n      \"to\": '$(date +%s)'000\n    },\n    \"parameters\": {\n      \"datasets\": [\"workers\"],\n      \"filters\": [\n        {\"key\": \"$workers.scriptName\", \"operation\": \"eq\", \"type\": \"string\", \"value\": \"enter-pollinations-ai\"},\n        {\"key\": \"$metadata.statusCode\", \"operation\": \"gte\", \"type\": \"number\", \"value\": 500}\n      ],\n      \"limit\": 20\n    }\n  }' | jq '.result.events.events[] | {\n    timestamp: .timestamp,\n    statusCode: .\"$metadata\".statusCode,\n    error: .\"$metadata\".error,\n    message: .\"$metadata\".message,\n    requestId: .\"$workers\".requestId,\n    url: .\"$metadata\".url\n  }'\n```\n\n### List Available Log Keys\n\n```bash\ncurl -s \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F$ACCOUNT_ID\u002Fworkers\u002Fobservability\u002Ftelemetry\u002Fkeys\" \\\n  -H \"Authorization: Bearer $API_TOKEN\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"timeframe\": {\n      \"from\": '$(( $(date +%s) - 3600 ))'000,\n      \"to\": '$(date +%s)'000\n    },\n    \"datasets\": [\"workers\"],\n    \"filters\": [\n      {\"key\": \"$workers.scriptName\", \"operation\": \"eq\", \"type\": \"string\", \"value\": \"enter-pollinations-ai\"}\n    ]\n  }' | jq '.result.keys'\n```\n\n## Structured Logging in enter.pollinations.ai\n\nThe worker uses LogTape for structured logging with these key fields:\n\n- **requestId**: Unique ID per request (first 8 chars shown in logs)\n- **method**: HTTP method (GET, POST)\n- **routePath**: Request URL\n- **status**: Response status code\n- **duration**: Request duration in ms\n\nDownstream errors are logged with:\n```typescript\nlog.warn(\"Chat completions error {status}: {body}\", {\n    status: response.status,\n    body: responseText,\n});\n```\n\n## Tinybird Analytics (Alternative)\n\nFor aggregated model health stats, query Tinybird directly.\n\n> **⚠️ Use the prod read token from SOPS — do NOT use `.tinyb`.** The `.tinyb` in `enter.pollinations.ai\u002Fobservability\u002F` points to the **staging** workspace (`pollinations_enter_staging`), which has ~no real traffic, so prod queries come back empty. Get the prod token instead:\n> ```bash\n> TB=$(sops -d enter.pollinations.ai\u002Fsecrets\u002Fprod.vars.json | jq -r '.TINYBIRD_READ_TOKEN')\n> ```\n> This single token works for **both** pipes (`\u002Fv0\u002Fpipes\u002F...`) and raw SQL (`\u002Fv0\u002Fsql`) against the prod workspace (`pollinations_enter`). The public read token in `apps\u002Fmodel-monitor\u002Fsrc\u002Fhooks\u002FuseModelMonitor.js` also works for pipes but is rotated periodically — pull it live, never hardcode (the one previously pinned in this skill went stale).\n\n```bash\nH=\"https:\u002F\u002Fapi.europe-west2.gcp.tinybird.co\"\n\n# Get model health stats — pass minutes (default pipe window is short; use 240 for last 4h)\ncurl -s \"$H\u002Fv0\u002Fpipes\u002Fmodel_health.json?token=$TB&minutes=240\" | jq '.data'\n\n# Detailed server-side error breakdown (full messages, upstream status\u002Fbody, user attribution)\ncurl -s \"$H\u002Fv0\u002Fpipes\u002Frecent_server_errors.json?token=$TB&minutes=240&limit=500\" -o \u002Ftmp\u002Ferrs.json\n```\n\n**`model_health` columns** (note: NOT `error_count`\u002F`error_rate`): `model`, `event_type`, `provider`, `model_used`, `total_requests`, `status_2xx`, `errors_4xx`, `errors_5xx`, `last_error_at`, `latency_p50_ms`, `latency_p95_ms`, `avg_latency_ms`, `last_request_at`. Sort by `errors_5xx` to find backend issues.\n\n**`recent_server_errors`** is the go-to pipe for root-causing (defined in `enter.pollinations.ai\u002Fobservability\u002Fendpoints\u002Frecent_server_errors.pipe`, params `minutes` default 1440, `limit` default 200). It returns `timestamp, status, upstream_status, upstream_host, upstream_body, message, error_code, error_class, model_requested, route_path, request_inputs, user_id, user_tier, api_key_id`. There is **no** `model_errors` pipe.\n\n> **JSON quirk**: `recent_server_errors` rows contain raw newlines in `stack`\u002F`message`, which break `jq`. Parse with Python instead: `python3 -c \"import json; d=json.load(open('\u002Ftmp\u002Ferrs.json'),strict=False); ...\"`.\n\n> **Reading 5xx**: `upstream_status` reveals the true cause. `502 (up 429)` = provider throttle (e.g. Bedrock \"Too many tokens\" — account-level TPM quota, often a peak-traffic spike across many users, not one abuser). `502 (up 403)` from `api.openai.com` with `unsupported_country_region_territory` = Cloudflare egress PoP in an OpenAI-blocked country. `500 (up 500)` from Vertex\u002FxAI = provider-side transient (\"high load\"\u002F\"Internal error\") — no action.\n\n---\n\n# Debugging Workflow\n\n1. **Check Model Monitor** - https:\u002F\u002Fmonitor.pollinations.ai\n   - Identify which models have high error rates\n   - Note the error code breakdown (401, 402, 403, 400, 500, etc.)\n\n2. **Query Cloudflare Logs** - Use the API queries above\n   - Get raw error events with full details\n   - Look for patterns in error messages\n   - Group by `user_id`, `api_key_id`, route, and sanitized `request_inputs` before calling the pattern a model-wide outage\n   - A concentrated burst from one caller can be invalid input even when the upstream reports 500\n\n3. **Correlate with Request ID** - If you have a specific request ID:\n   ```bash\n   # Filter by request ID\n   curl -s \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F$ACCOUNT_ID\u002Fworkers\u002Fobservability\u002Ftelemetry\u002Fquery\" \\\n     -H \"Authorization: Bearer $API_TOKEN\" \\\n     -H \"Content-Type: application\u002Fjson\" \\\n     -d '{\n       \"timeframe\": {\"from\": '$(( $(date +%s) - 86400 ))'000, \"to\": '$(date +%s)'000},\n       \"parameters\": {\n         \"datasets\": [\"workers\"],\n         \"filters\": [\n           {\"key\": \"$workers.requestId\", \"operation\": \"eq\", \"type\": \"string\", \"value\": \"REQUEST_ID_HERE\"}\n         ],\n         \"limit\": 100\n       }\n     }' | jq '.result.events.events'\n   ```\n\n4. **Check Gateway Logs** - Tail the gen Worker (image + text both run here):\n   ```bash\n   cd gen.pollinations.ai && wrangler tail --format json | tee gen-logs.jsonl\n   ```\n\n5. **Test Model Directly** - Verify if model is actually broken:\n   ```bash\n   TOKEN=$(grep ENTER_API_TOKEN_REMOTE enter.pollinations.ai\u002F.testingtokens | cut -d= -f2)\n   \n   # Test text model\n   curl -s 'https:\u002F\u002Fgen.pollinations.ai\u002Fv1\u002Fchat\u002Fcompletions' \\\n     -H \"Authorization: Bearer $TOKEN\" \\\n     -H 'Content-Type: application\u002Fjson' \\\n     -d '{\"model\": \"MODEL_NAME\", \"messages\": [{\"role\": \"user\", \"content\": \"Test\"}]}' \\\n     -w \"\\nHTTP: %{http_code}\\n\"\n   \n   # Test image model\n   curl -s 'https:\u002F\u002Fgen.pollinations.ai\u002Fimage\u002Ftest?model=MODEL_NAME&width=256&height=256' \\\n     -H \"Authorization: Bearer $TOKEN\" \\\n     -w \"\\nHTTP: %{http_code}\\n\" -o \u002Fdev\u002Fnull\n   ```\n\n---\n\n# Current Status & Limitations\n\n## Cloudflare Observability API\n\n**What works:**\n- `\u002Ftelemetry\u002Fkeys` - List available log fields ✅\n- `\u002Ftelemetry\u002Fvalues` - Get unique values for a field ✅\n- Token stored in SOPS: `enter.pollinations.ai\u002Fsecrets\u002Fenv.json` ✅\n\n**Limitations:**\n- `\u002Ftelemetry\u002Fquery` requires a saved `queryId` from the dashboard\n- For ad-hoc queries, use **Cloudflare Dashboard** → Workers & Pages → pollinations-enter → Observability → Investigate\n- Or use `wrangler tail` for real-time logs\n\n## Alternative: Tinybird (Recommended for Aggregates)\n\nTinybird provides pre-aggregated model health stats and raw event data.\n\n### Token Locations\n\n- **Prod read token (use this)**: `enter.pollinations.ai\u002Fsecrets\u002Fprod.vars.json` → `TINYBIRD_READ_TOKEN` (via SOPS). Works for both pipes and raw `\u002Fv0\u002Fsql` against prod (`pollinations_enter`).\n- **Public read token** (pipes only, rotates): `apps\u002Fmodel-monitor\u002Fsrc\u002Fhooks\u002FuseModelMonitor.js`.\n- **`.tinyb`** = **staging** workspace (`pollinations_enter_staging`) — empty of prod traffic. Only use for staging-specific debugging.\n\n### Basic Queries\n\n```bash\n# Prod read token from SOPS — works for pipes AND raw SQL\nTB=$(sops -d enter.pollinations.ai\u002Fsecrets\u002Fprod.vars.json | jq -r '.TINYBIRD_READ_TOKEN')\n\n# Get model health (last 4h)\ncurl -s \"https:\u002F\u002Fapi.europe-west2.gcp.tinybird.co\u002Fv0\u002Fpipes\u002Fmodel_health.json?token=$TB&minutes=240\" | jq '.data'\n```\n\n### Raw SQL Queries\n\nThe prod `TINYBIRD_READ_TOKEN` above can query the raw `generation_event_v2` datasource directly via `\u002Fv0\u002Fsql` (verified). Reuse `$TB`:\n\n```bash\n# Find users with frequent 403 errors (last 24 hours)\ncurl -s \"https:\u002F\u002Fapi.europe-west2.gcp.tinybird.co\u002Fv0\u002Fsql?token=$TB\" \\\n  --data-urlencode \"q=SELECT user_id, user_github_username, user_tier, count() as error_403_count\nFROM generation_event_v2\nWHERE response_status = 403\n  AND start_time > now() - interval 24 hour\n  AND user_id != ''\n  AND user_id != 'undefined'\nGROUP BY user_id, user_github_username, user_tier\nORDER BY error_403_count DESC\nLIMIT 20\"\n\n# Find users with 500 errors (actual backend issues)\ncurl -s \"https:\u002F\u002Fapi.europe-west2.gcp.tinybird.co\u002Fv0\u002Fsql?token=$TB\" \\\n  --data-urlencode \"q=SELECT user_github_username, model_requested, error_message, count() as error_count \nFROM generation_event_v2\nWHERE response_status >= 500 \n  AND start_time > now() - interval 24 hour \nGROUP BY user_github_username, model_requested, error_message \nORDER BY error_count DESC \nLIMIT 20\"\n\n# Check specific user's recent errors\ncurl -s \"https:\u002F\u002Fapi.europe-west2.gcp.tinybird.co\u002Fv0\u002Fsql?token=$TB\" \\\n  --data-urlencode \"q=SELECT start_time, response_status, model_requested, error_message \nFROM generation_event_v2\nWHERE user_github_username = 'USERNAME_HERE' \n  AND start_time > now() - interval 24 hour \nORDER BY start_time DESC \nLIMIT 50\"\n```\n\n### Datasource Schema\n\nThe `generation_event_v2` datasource is defined in `enter.pollinations.ai\u002Fobservability\u002Fdatasources\u002Fgeneration_event_v2.datasource` and includes:\n- `user_id`, `user_github_username`, `user_tier`\n- `response_status`, `error_message`, `error_response_code`\n- `model_requested`, `model_used`\n- `total_price`, `total_cost`\n- `start_time`, `end_time`, `response_time`\n\n---\n\n# Scripts\n\nHelper scripts for common debugging tasks. Run from repo root.\n\n## Find Users with 403 Errors (Quota Issues)\n\n```bash\n# Find users with >10 403 errors in last 24 hours\n.claude\u002Fskills\u002Fmodel-debugging\u002Fscripts\u002Ffind-403-users.sh 24 10\n```\n\n## Find 500 Errors (Backend Issues)\n\n```bash\n# Find 500+ errors grouped by user\u002Fmodel\u002Fmessage\n.claude\u002Fskills\u002Fmodel-debugging\u002Fscripts\u002Ffind-500-errors.sh 24\n```\n\n## Check Specific User's Errors\n\n```bash\n# See a user's recent errors\n.claude\u002Fskills\u002Fmodel-debugging\u002Fscripts\u002Fcheck-user-errors.sh superbrainai 24\n```\n\n---\n\n# Notes\n\n- **401 errors**: User authentication issues (no API key) - **expected from anonymous traffic**\n- **402 errors**: Pollen\u002Fbilling issues (user ran out of credits or key budget) - **expected**\n- **403 errors**: Permission issues (model not allowed for API key) - **expected**\n- **400 errors**: Usually user input errors (bad prompts, invalid params) - **expected**\n- **500 errors**: Backend\u002Finfrastructure issues - **investigate these**\n- **504 errors**: Timeouts (model too slow or hung) - **investigate these**\n\n---\n\n# Tested Models (All Working as of 2025-12-22)\n\n| Model | Type | Endpoint | Status |\n|-------|------|----------|--------|\n| `openai` | text | POST \u002Fv1\u002Fchat\u002Fcompletions | ✅ |\n| `openai-fast` | text | POST \u002Fv1\u002Fchat\u002Fcompletions | ✅ |\n| `openai-large` | text | POST \u002Fv1\u002Fchat\u002Fcompletions | ✅ |\n| `openai-audio` | text | GET \u002Ftext\u002F{prompt}?model=openai-audio&voice=alloy | ✅ (MP3) |\n| `claude` | text | POST \u002Fv1\u002Fchat\u002Fcompletions | ✅ |\n| `gemini-fast` | text | POST \u002Fv1\u002Fchat\u002Fcompletions | ✅ |\n| `flux` | image | GET \u002Fimage\u002F{prompt} | ✅ |\n| `nanobanana-pro` | image | GET \u002Fimage\u002F{prompt} | ✅ |\n| `seedream-pro` | image | GET \u002Fimage\u002F{prompt} | ✅ |\n| `seedance-pro` | video | GET \u002Fimage\u002F{prompt} | ✅ (MP4) |\n","# Model Debugging Skill\n\nUse this skill when:\n- Investigating model failures, high error rates, or service issues\n- Finding users affected by errors (402 billing, 403 permissions, 500 backend)\n- Analyzing Tinybird\u002FCloudflare logs for patterns\n- Diagnosing specific request failures\n\n# Understanding Model Monitor Error Rates\n\n**Why does the Model Monitor show high error rates when models work fine manually?**\n\nThe Model Monitor at https:\u002F\u002Fmonitor.pollinations.ai shows **all real-world traffic**, including:\n\n- **401 errors**: Anonymous users without API keys (most common)\n- **402 errors**: Users with insufficient pollen balance or exhausted API key budget\n- **403 errors**: Users denied access to specific models (API key restrictions)\n- **400 errors**: Invalid request parameters (e.g., `openai-audio` without `modalities` param)\n- **429 errors**: Rate-limited requests\n- **500\u002F504 errors**: Actual backend failures (investigate these)\n\nWhen you test manually with a valid secret key (`sk_`), you bypass auth\u002Fquota issues, so models appear to work fine.\n\n**Key insight**: High 401\u002F402\u002F403\u002F400 rates are **expected** from real-world usage. Focus investigation on 500\u002F504 errors.\n\n---\n\n# Data Flow Architecture\n\n```\nUser Request → enter.pollinations.ai (Cloudflare Worker)\n                    ↓\n              Logs to Cloudflare Workers Observability\n                    ↓\n              Events stored in D1 database\n                    ↓\n              Batched to Tinybird (async, 100-500 events)\n                    ↓\n              Model Monitor queries Tinybird (model_health.pipe)\n```\n\n**Structured Logging**: enter.pollinations.ai uses LogTape with:\n- `requestId`: Unique per request (passed to downstream via `x-request-id` header)\n- `status`, `body`: Full error response from downstream services\n- Context: `method`, `routePath`, `userAgent`, `ipAddress`\n\n---\n\n# Quick Diagnostics\n\n## 1. Check Model Monitor\nView current model health at: https:\u002F\u002Fmonitor.pollinations.ai\n\n## 2. Query Recent Errors from D1 Database\n```bash\n# Via enter.pollinations.ai worker (requires wrangler)\ncd enter.pollinations.ai\nnpx wrangler d1 execute pollinations-db --remote --command \"SELECT model_requested, response_status, error_message, COUNT(*) as count FROM event WHERE response_status >= 400 AND created_at > datetime('now', '-1 hour') GROUP BY model_requested, response_status, error_message ORDER BY count DESC LIMIT 20\"\n```\n\n## 3. Capture Live Logs\n\n### enter.pollinations.ai (Cloudflare Worker)\n```bash\ncd enter.pollinations.ai\nwrangler tail --format json | tee logs.jsonl\n# Or with formatting:\nwrangler tail --format json | npx tsx scripts\u002Fformat-logs.ts\n```\n\n### gen.pollinations.ai (image + text gateway)\nImage and text generation now run inside the gen Cloudflare Worker (the legacy EC2 `image-pollinations` and `text-pollinations` services are decommissioned). Use `wrangler tail` from `gen.pollinations.ai\u002F`:\n```bash\ncd gen.pollinations.ai\nwrangler tail --format json | tee gen-logs.jsonl\n```\n\n### Legacy anonymous image (OVH)\nAnonymous traffic to `image.pollinations.ai` still terminates on the OVH host:\n```bash\n# Real-time logs\nssh -i ~\u002F.ssh\u002Fid_rsa_ovh ubuntu@57.130.31.42 \"sudo journalctl -u image-pollinations -f\"\n\n# Last 3 minutes\nssh -i ~\u002F.ssh\u002Fid_rsa_ovh ubuntu@57.130.31.42 \"sudo journalctl -u image-pollinations --since '3 minutes ago' --no-pager\" > legacy-image-logs.txt\n```\n\n---\n\n# Common Error Patterns\n\n## Azure Content Safety DNS Failure\n**Error**: `getaddrinfo ENOTFOUND gptimagemain1-resource.cognitiveservices.azure.com`\n**Cause**: Azure Content Safety resource deleted or misconfigured\n**Impact**: Fail-open (content proceeds without safety check)\n**Fix**: Create new Azure Content Safety resource and update `.env`:\n```\nAZURE_CONTENT_SAFETY_ENDPOINT=https:\u002F\u002F\u003Cnew-resource>.cognitiveservices.azure.com\u002F\nAZURE_CONTENT_SAFETY_API_KEY=\u003Cnew-key>\n```\n\n## Azure Kontext Content Filter\n**Error**: `Content rejected due to sexual\u002Fhate\u002Fviolence content detection`\n**Cause**: Azure's content moderation blocking prompts\u002Fimages\n**Impact**: 400 error returned to user\n**Fix**: User error - prompt violates content policy\n\n## Vertex AI Invalid Image\n**Error**: `Provided image is not valid`\n**Cause**: User passing unsupported image URL (e.g., Google Drive links)\n**Impact**: 400 error returned to user\n**Fix**: User error - need direct image URL\n\n## Translation Service Down\n**Error**: `No active translate servers available`\n**Cause**: Translation service unavailable\n**Impact**: Prompts not translated (non-fatal)\n**Fix**: Check translation service status\n\n## OpenAI Audio Invalid Voice\n**Error**: `Invalid value for audio.voice`\n**Cause**: User requesting unsupported voice name\n**Impact**: 400 error returned to user\n**Fix**: User error - use supported voices: alloy, echo, fable, onyx, nova, shimmer, coral, verse, ballad, ash, sage, etc.\n\n## Oversized Text Seed Surfaced as 500\n**Error**: `'seed' must be Integer`, `invalid request error`, or a generic upstream 500\n**Cause**: A client sent a seed above signed INT32 max (`2147483647`) to a strict provider\n**Impact**: The provider may misclassify invalid client input as 500, inflating model health errors\n**Fix**: Reject oversized seeds as 400 at gateway validation; group incidents by user, API key, and request shape before treating them as a model outage\n\n## Veo No Video Data\n**Error**: `No video data in response`\n**Cause**: Vertex AI returned empty video response\n**Impact**: 500 error\n**Fix**: Check Vertex AI quota\u002Fstatus, may be transient\n\n---\n\n# Environment Variables to Check\n\nImage and text env vars now live in the gen Worker secrets (`gen.pollinations.ai\u002Fsecrets\u002F{dev,staging,prod}.vars.json`, SOPS-encrypted). Decrypt to inspect:\n```bash\nsops -d gen.pollinations.ai\u002Fsecrets\u002Fprod.vars.json | jq 'keys[] | select(test(\"AZURE|GOOGLE|CLOUDFLARE|OPENAI\"))'\n```\n\nKey variables:\n- `AZURE_CONTENT_SAFETY_ENDPOINT` - Azure Content Safety API endpoint\n- `AZURE_CONTENT_SAFETY_API_KEY` - Azure Content Safety API key\n- `GOOGLE_PROJECT_ID` - Google Cloud project for Vertex AI\n- `AZURE_MYCELI_PROD_SWEDEN_API_KEY` - Shared Azure API key (Kontext, GPT Image, GPT Image 1.5)\n\n---\n\n# Updating Secrets\n\nSecrets are stored encrypted with SOPS:\n- `gen.pollinations.ai\u002Fsecrets\u002F{dev,staging,prod}.vars.json`\n- `enter.pollinations.ai\u002Fsecrets\u002F{dev,staging,prod}.vars.json`\n\nTo update:\n```bash\n# Decrypt, edit, re-encrypt\nsops gen.pollinations.ai\u002Fsecrets\u002Fprod.vars.json\n\n# Deploy to the gen Worker (secrets ship with the deploy)\ncd gen.pollinations.ai && npm run deploy\n```\n\n---\n\n# Log Analysis Commands\n\n```bash\n# Count errors by type (against captured wrangler-tail JSON)\njq -r '.logs[]?.message[]? \u002F\u002F .message? \u002F\u002F empty' gen-logs.jsonl | grep -oE \"(Azure Flux Kontext|Vertex AI|No active translate|getaddrinfo ENOTFOUND)\" | sort | uniq -c | sort -rn\n\n# Find content filter rejections\njq -r '.logs[]?.message[]? \u002F\u002F .message? \u002F\u002F empty' gen-logs.jsonl | grep -i \"Content rejected\" | sort | uniq -c\n```\n\n---\n\n# Model-Specific Debugging\n\n| Model | Backend | Common Issues |\n|-------|---------|---------------|\n| `flux` | Azure\u002FReplicate | Rate limits, content filter |\n| `kontext` | Azure Flux Kontext | Content filter (strict) |\n| `nanobanana` | Vertex AI Gemini | Invalid image URLs, content filter |\n| `seedream-pro` | ByteDance ARK | NSFW filter, API key issues |\n| `veo` | Vertex AI | Quota, empty responses |\n| `openai-audio` | Azure OpenAI | Invalid voice names |\n| `deepseek` | DeepSeek API | Rate limits, API key |\n\n---\n\n# Cloudflare Workers Observability API\n\nThe enter.pollinations.ai worker has structured logging enabled. You can query logs programmatically via the Cloudflare Workers Observability API.\n\n## Prerequisites\n\n### 1. Get Account ID\n```bash\n# From wrangler.toml\ngrep account_id enter.pollinations.ai\u002Fwrangler.toml\n\n# Or from existing .env\ngrep CLOUDFLARE_ACCOUNT_ID image.pollinations.ai\u002F.env\n```\n\n### 2. Create API Token with Workers Observability Permission\n\n**Via Cloudflare Dashboard:**\n1. Go to https:\u002F\u002Fdash.cloudflare.com\u002Fprofile\u002Fapi-tokens\n2. Click **Create Token**\n3. Click **Create Custom Token**\n4. Configure:\n   - **Token name**: `Workers Observability Read`\n   - **Permissions**: \n     - Account → Workers Scripts → Read\n     - Account → Workers Observability → Edit (required for query API)\n   - **Account Resources**: Include → Your Account\n5. Click **Continue to summary** → **Create Token**\n6. Copy the token immediately (shown only once)\n\n### 3. Store Token Securely\n\nThe token is stored in SOPS-encrypted secrets:\n- **Location**: `enter.pollinations.ai\u002Fsecrets\u002Fenv.json`\n- **Key**: `CLOUDFLARE_OBSERVABILITY_TOKEN`\n\nTo add\u002Fupdate:\n```bash\n# Step 1: Decrypt to temp file\ncd \u002Fpath\u002Fto\u002Fpollinations\nsops -d enter.pollinations.ai\u002Fsecrets\u002Fenv.json > \u002Ftmp\u002Fenv.json\n\n# Step 2: Add the token (use jq)\njq '. + {\"CLOUDFLARE_OBSERVABILITY_TOKEN\": \"your_token\"}' \u002Ftmp\u002Fenv.json > \u002Ftmp\u002Fenv_updated.json\n\n# Step 3: Re-encrypt (must rename to match .sops.yaml pattern)\ncp \u002Ftmp\u002Fenv_updated.json \u002Ftmp\u002Fenv.json\nsops -e \u002Ftmp\u002Fenv.json > enter.pollinations.ai\u002Fsecrets\u002Fenv.json\n\n# Step 4: Cleanup\nrm \u002Ftmp\u002Fenv.json \u002Ftmp\u002Fenv_updated.json\n\n# Verify\nsops -d enter.pollinations.ai\u002Fsecrets\u002Fenv.json | jq 'keys'\n```\n\n**Note**: The `.sops.yaml` config requires filenames matching `env.json$` pattern.\n\n## API Endpoint\n\n```\nPOST https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F{account_id}\u002Fworkers\u002Fobservability\u002Ftelemetry\u002Fquery\n```\n\n## Query Examples\n\n### Setup: Get Credentials from SOPS\n\n```bash\n# Extract credentials from encrypted secrets\nACCOUNT_ID=$(sops -d enter.pollinations.ai\u002Fsecrets\u002Fenv.json | jq -r '.CLOUDFLARE_ACCOUNT_ID')\nAPI_TOKEN=$(sops -d enter.pollinations.ai\u002Fsecrets\u002Fenv.json | jq -r '.CLOUDFLARE_OBSERVABILITY_TOKEN')\n```\n\n### List Available Log Keys (Working)\n\nThis endpoint works and shows what fields are available:\n\n```bash\ncurl -s \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F$ACCOUNT_ID\u002Fworkers\u002Fobservability\u002Ftelemetry\u002Fkeys\" \\\n  -H \"Authorization: Bearer $API_TOKEN\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\"timeframe\": {\"from\": '$(( $(date +%s) - 86400 ))'000, \"to\": '$(date +%s)'000}, \"datasets\": [\"workers\"]}' | jq '.result[:10]'\n```\n\n### Query Recent Errors (Last 15 Minutes)\n\n**Note**: The `\u002Fquery` endpoint requires a saved `queryId`. For ad-hoc queries, use the Cloudflare Dashboard Query Builder or `wrangler tail`.\n\n```bash\n# This format requires a saved query ID\n\n# Query errors with status >= 400\ncurl -s \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F$ACCOUNT_ID\u002Fworkers\u002Fobservability\u002Ftelemetry\u002Fquery\" \\\n  -H \"Authorization: Bearer $API_TOKEN\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"timeframe\": {\n      \"from\": '$(( $(date +%s) - 900 ))'000,\n      \"to\": '$(date +%s)'000\n    },\n    \"parameters\": {\n      \"datasets\": [\"workers\"],\n      \"filters\": [\n        {\"key\": \"$workers.scriptName\", \"operation\": \"eq\", \"type\": \"string\", \"value\": \"enter-pollinations-ai\"},\n        {\"key\": \"$metadata.statusCode\", \"operation\": \"gte\", \"type\": \"number\", \"value\": 400}\n      ],\n      \"calculations\": [{\"operator\": \"count\"}],\n      \"groupBys\": [\n        {\"type\": \"string\", \"value\": \"$metadata.statusCode\"},\n        {\"type\": \"string\", \"value\": \"$metadata.error\"}\n      ],\n      \"limit\": 50\n    }\n  }' | jq '.result.events.events[:20]'\n```\n\n### Query Errors by Model\n\n```bash\ncurl -s \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F$ACCOUNT_ID\u002Fworkers\u002Fobservability\u002Ftelemetry\u002Fquery\" \\\n  -H \"Authorization: Bearer $API_TOKEN\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"timeframe\": {\n      \"from\": '$(( $(date +%s) - 3600 ))'000,\n      \"to\": '$(date +%s)'000\n    },\n    \"parameters\": {\n      \"datasets\": [\"workers\"],\n      \"filters\": [\n        {\"key\": \"$workers.scriptName\", \"operation\": \"eq\", \"type\": \"string\", \"value\": \"enter-pollinations-ai\"},\n        {\"key\": \"$metadata.statusCode\", \"operation\": \"gte\", \"type\": \"number\", \"value\": 400}\n      ],\n      \"calculations\": [{\"operator\": \"count\"}],\n      \"groupBys\": [\n        {\"type\": \"string\", \"value\": \"model\"},\n        {\"type\": \"string\", \"value\": \"$metadata.statusCode\"}\n      ],\n      \"limit\": 100\n    }\n  }' | jq '.result.calculations[0].aggregates'\n```\n\n### Get Raw Error Events with Full Details\n\n```bash\ncurl -s \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F$ACCOUNT_ID\u002Fworkers\u002Fobservability\u002Ftelemetry\u002Fquery\" \\\n  -H \"Authorization: Bearer $API_TOKEN\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"timeframe\": {\n      \"from\": '$(( $(date +%s) - 900 ))'000,\n      \"to\": '$(date +%s)'000\n    },\n    \"parameters\": {\n      \"datasets\": [\"workers\"],\n      \"filters\": [\n        {\"key\": \"$workers.scriptName\", \"operation\": \"eq\", \"type\": \"string\", \"value\": \"enter-pollinations-ai\"},\n        {\"key\": \"$metadata.statusCode\", \"operation\": \"gte\", \"type\": \"number\", \"value\": 500}\n      ],\n      \"limit\": 20\n    }\n  }' | jq '.result.events.events[] | {\n    timestamp: .timestamp,\n    statusCode: .\"$metadata\".statusCode,\n    error: .\"$metadata\".error,\n    message: .\"$metadata\".message,\n    requestId: .\"$workers\".requestId,\n    url: .\"$metadata\".url\n  }'\n```\n\n### List Available Log Keys\n\n```bash\ncurl -s \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F$ACCOUNT_ID\u002Fworkers\u002Fobservability\u002Ftelemetry\u002Fkeys\" \\\n  -H \"Authorization: Bearer $API_TOKEN\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"timeframe\": {\n      \"from\": '$(( $(date +%s) - 3600 ))'000,\n      \"to\": '$(date +%s)'000\n    },\n    \"datasets\": [\"workers\"],\n    \"filters\": [\n      {\"key\": \"$workers.scriptName\", \"operation\": \"eq\", \"type\": \"string\", \"value\": \"enter-pollinations-ai\"}\n    ]\n  }' | jq '.result.keys'\n```\n\n## Structured Logging in enter.pollinations.ai\n\nThe worker uses LogTape for structured logging with these key fields:\n\n- **requestId**: Unique ID per request (first 8 chars shown in logs)\n- **method**: HTTP method (GET, POST)\n- **routePath**: Request URL\n- **status**: Response status code\n- **duration**: Request duration in ms\n\nDownstream errors are logged with:\n```typescript\nlog.warn(\"Chat completions error {status}: {body}\", {\n    status: response.status,\n    body: responseText,\n});\n```\n\n## Tinybird Analytics (Alternative)\n\nFor aggregated model health stats, query Tinybird directly.\n\n> **⚠️ Use the prod read token from SOPS — do NOT use `.tinyb`.** The `.tinyb` in `enter.pollinations.ai\u002Fobservability\u002F` points to the **staging** workspace (`pollinations_enter_staging`), which has ~no real traffic, so prod queries come back empty. Get the prod token instead:\n> ```bash\n> TB=$(sops -d enter.pollinations.ai\u002Fsecrets\u002Fprod.vars.json | jq -r '.TINYBIRD_READ_TOKEN')\n> ```\n> This single token works for **both** pipes (`\u002Fv0\u002Fpipes\u002F...`) and raw SQL (`\u002Fv0\u002Fsql`) against the prod workspace (`pollinations_enter`). The public read token in `apps\u002Fmodel-monitor\u002Fsrc\u002Fhooks\u002FuseModelMonitor.js` also works for pipes but is rotated periodically — pull it live, never hardcode (the one previously pinned in this skill went stale).\n\n```bash\nH=\"https:\u002F\u002Fapi.europe-west2.gcp.tinybird.co\"\n\n# Get model health stats — pass minutes (default pipe window is short; use 240 for last 4h)\ncurl -s \"$H\u002Fv0\u002Fpipes\u002Fmodel_health.json?token=$TB&minutes=240\" | jq '.data'\n\n# Detailed server-side error breakdown (full messages, upstream status\u002Fbody, user attribution)\ncurl -s \"$H\u002Fv0\u002Fpipes\u002Frecent_server_errors.json?token=$TB&minutes=240&limit=500\" -o \u002Ftmp\u002Ferrs.json\n```\n\n**`model_health` columns** (note: NOT `error_count`\u002F`error_rate`): `model`, `event_type`, `provider`, `model_used`, `total_requests`, `status_2xx`, `errors_4xx`, `errors_5xx`, `last_error_at`, `latency_p50_ms`, `latency_p95_ms`, `avg_latency_ms`, `last_request_at`. Sort by `errors_5xx` to find backend issues.\n\n**`recent_server_errors`** is the go-to pipe for root-causing (defined in `enter.pollinations.ai\u002Fobservability\u002Fendpoints\u002Frecent_server_errors.pipe`, params `minutes` default 1440, `limit` default 200). It returns `timestamp, status, upstream_status, upstream_host, upstream_body, message, error_code, error_class, model_requested, route_path, request_inputs, user_id, user_tier, api_key_id`. There is **no** `model_errors` pipe.\n\n> **JSON quirk**: `recent_server_errors` rows contain raw newlines in `stack`\u002F`message`, which break `jq`. Parse with Python instead: `python3 -c \"import json; d=json.load(open('\u002Ftmp\u002Ferrs.json'),strict=False); ...\"`.\n\n> **Reading 5xx**: `upstream_status` reveals the true cause. `502 (up 429)` = provider throttle (e.g. Bedrock \"Too many tokens\" — account-level TPM quota, often a peak-traffic spike across many users, not one abuser). `502 (up 403)` from `api.openai.com` with `unsupported_country_region_territory` = Cloudflare egress PoP in an OpenAI-blocked country. `500 (up 500)` from Vertex\u002FxAI = provider-side transient (\"high load\"\u002F\"Internal error\") — no action.\n\n---\n\n# Debugging Workflow\n\n1. **Check Model Monitor** - https:\u002F\u002Fmonitor.pollinations.ai\n   - Identify which models have high error rates\n   - Note the error code breakdown (401, 402, 403, 400, 500, etc.)\n\n2. **Query Cloudflare Logs** - Use the API queries above\n   - Get raw error events with full details\n   - Look for patterns in error messages\n   - Group by `user_id`, `api_key_id`, route, and sanitized `request_inputs` before calling the pattern a model-wide outage\n   - A concentrated burst from one caller can be invalid input even when the upstream reports 500\n\n3. **Correlate with Request ID** - If you have a specific request ID:\n   ```bash\n   # Filter by request ID\n   curl -s \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F$ACCOUNT_ID\u002Fworkers\u002Fobservability\u002Ftelemetry\u002Fquery\" \\\n     -H \"Authorization: Bearer $API_TOKEN\" \\\n     -H \"Content-Type: application\u002Fjson\" \\\n     -d '{\n       \"timeframe\": {\"from\": '$(( $(date +%s) - 86400 ))'000, \"to\": '$(date +%s)'000},\n       \"parameters\": {\n         \"datasets\": [\"workers\"],\n         \"filters\": [\n           {\"key\": \"$workers.requestId\", \"operation\": \"eq\", \"type\": \"string\", \"value\": \"REQUEST_ID_HERE\"}\n         ],\n         \"limit\": 100\n       }\n     }' | jq '.result.events.events'\n   ```\n\n4. **Check Gateway Logs** - Tail the gen Worker (image + text both run here):\n   ```bash\n   cd gen.pollinations.ai && wrangler tail --format json | tee gen-logs.jsonl\n   ```\n\n5. **Test Model Directly** - Verify if model is actually broken:\n   ```bash\n   TOKEN=$(grep ENTER_API_TOKEN_REMOTE enter.pollinations.ai\u002F.testingtokens | cut -d= -f2)\n   \n   # Test text model\n   curl -s 'https:\u002F\u002Fgen.pollinations.ai\u002Fv1\u002Fchat\u002Fcompletions' \\\n     -H \"Authorization: Bearer $TOKEN\" \\\n     -H 'Content-Type: application\u002Fjson' \\\n     -d '{\"model\": \"MODEL_NAME\", \"messages\": [{\"role\": \"user\", \"content\": \"Test\"}]}' \\\n     -w \"\\nHTTP: %{http_code}\\n\"\n   \n   # Test image model\n   curl -s 'https:\u002F\u002Fgen.pollinations.ai\u002Fimage\u002Ftest?model=MODEL_NAME&width=256&height=256' \\\n     -H \"Authorization: Bearer $TOKEN\" \\\n     -w \"\\nHTTP: %{http_code}\\n\" -o \u002Fdev\u002Fnull\n   ```\n\n---\n\n# Current Status & Limitations\n\n## Cloudflare Observability API\n\n**What works:**\n- `\u002Ftelemetry\u002Fkeys` - List available log fields ✅\n- `\u002Ftelemetry\u002Fvalues` - Get unique values for a field ✅\n- Token stored in SOPS: `enter.pollinations.ai\u002Fsecrets\u002Fenv.json` ✅\n\n**Limitations:**\n- `\u002Ftelemetry\u002Fquery` requires a saved `queryId` from the dashboard\n- For ad-hoc queries, use **Cloudflare Dashboard** → Workers & Pages → pollinations-enter → Observability → Investigate\n- Or use `wrangler tail` for real-time logs\n\n## Alternative: Tinybird (Recommended for Aggregates)\n\nTinybird provides pre-aggregated model health stats and raw event data.\n\n### Token Locations\n\n- **Prod read token (use this)**: `enter.pollinations.ai\u002Fsecrets\u002Fprod.vars.json` → `TINYBIRD_READ_TOKEN` (via SOPS). Works for both pipes and raw `\u002Fv0\u002Fsql` against prod (`pollinations_enter`).\n- **Public read token** (pipes only, rotates): `apps\u002Fmodel-monitor\u002Fsrc\u002Fhooks\u002FuseModelMonitor.js`.\n- **`.tinyb`** = **staging** workspace (`pollinations_enter_staging`) — empty of prod traffic. Only use for staging-specific debugging.\n\n### Basic Queries\n\n```bash\n# Prod read token from SOPS — works for pipes AND raw SQL\nTB=$(sops -d enter.pollinations.ai\u002Fsecrets\u002Fprod.vars.json | jq -r '.TINYBIRD_READ_TOKEN')\n\n# Get model health (last 4h)\ncurl -s \"https:\u002F\u002Fapi.europe-west2.gcp.tinybird.co\u002Fv0\u002Fpipes\u002Fmodel_health.json?token=$TB&minutes=240\" | jq '.data'\n```\n\n### Raw SQL Queries\n\nThe prod `TINYBIRD_READ_TOKEN` above can query the raw `generation_event_v2` datasource directly via `\u002Fv0\u002Fsql` (verified). Reuse `$TB`:\n\n```bash\n# Find users with frequent 403 errors (last 24 hours)\ncurl -s \"https:\u002F\u002Fapi.europe-west2.gcp.tinybird.co\u002Fv0\u002Fsql?token=$TB\" \\\n  --data-urlencode \"q=SELECT user_id, user_github_username, user_tier, count() as error_403_count\nFROM generation_event_v2\nWHERE response_status = 403\n  AND start_time > now() - interval 24 hour\n  AND user_id != ''\n  AND user_id != 'undefined'\nGROUP BY user_id, user_github_username, user_tier\nORDER BY error_403_count DESC\nLIMIT 20\"\n\n# Find users with 500 errors (actual backend issues)\ncurl -s \"https:\u002F\u002Fapi.europe-west2.gcp.tinybird.co\u002Fv0\u002Fsql?token=$TB\" \\\n  --data-urlencode \"q=SELECT user_github_username, model_requested, error_message, count() as error_count \nFROM generation_event_v2\nWHERE response_status >= 500 \n  AND start_time > now() - interval 24 hour \nGROUP BY user_github_username, model_requested, error_message \nORDER BY error_count DESC \nLIMIT 20\"\n\n# Check specific user's recent errors\ncurl -s \"https:\u002F\u002Fapi.europe-west2.gcp.tinybird.co\u002Fv0\u002Fsql?token=$TB\" \\\n  --data-urlencode \"q=SELECT start_time, response_status, model_requested, error_message \nFROM generation_event_v2\nWHERE user_github_username = 'USERNAME_HERE' \n  AND start_time > now() - interval 24 hour \nORDER BY start_time DESC \nLIMIT 50\"\n```\n\n### Datasource Schema\n\nThe `generation_event_v2` datasource is defined in `enter.pollinations.ai\u002Fobservability\u002Fdatasources\u002Fgeneration_event_v2.datasource` and includes:\n- `user_id`, `user_github_username`, `user_tier`\n- `response_status`, `error_message`, `error_response_code`\n- `model_requested`, `model_used`\n- `total_price`, `total_cost`\n- `start_time`, `end_time`, `response_time`\n\n---\n\n# Scripts\n\nHelper scripts for common debugging tasks. Run from repo root.\n\n## Find Users with 403 Errors (Quota Issues)\n\n```bash\n# Find users with >10 403 errors in last 24 hours\n.claude\u002Fskills\u002Fmodel-debugging\u002Fscripts\u002Ffind-403-users.sh 24 10\n```\n\n## Find 500 Errors (Backend Issues)\n\n```bash\n# Find 500+ errors grouped by user\u002Fmodel\u002Fmessage\n.claude\u002Fskills\u002Fmodel-debugging\u002Fscripts\u002Ffind-500-errors.sh 24\n```\n\n## Check Specific User's Errors\n\n```bash\n# See a user's recent errors\n.claude\u002Fskills\u002Fmodel-debugging\u002Fscripts\u002Fcheck-user-errors.sh superbrainai 24\n```\n\n---\n\n# Notes\n\n- **401 errors**: User authentication issues (no API key) - **expected from anonymous traffic**\n- **402 errors**: Pollen\u002Fbilling issues (user ran out of credits or key budget) - **expected**\n- **403 errors**: Permission issues (model not allowed for API key) - **expected**\n- **400 errors**: Usually user input errors (bad prompts, invalid params) - **expected**\n- **500 errors**: Backend\u002Finfrastructure issues - **investigate these**\n- **504 errors**: Timeouts (model too slow or hung) - **investigate these**\n\n---\n\n# Tested Models (All Working as of 2025-12-22)\n\n| Model | Type | Endpoint | Status |\n|-------|------|----------|--------|\n| `openai` | text | POST \u002Fv1\u002Fchat\u002Fcompletions | ✅ |\n| `openai-fast` | text | POST \u002Fv1\u002Fchat\u002Fcompletions | ✅ |\n| `openai-large` | text | POST \u002Fv1\u002Fchat\u002Fcompletions | ✅ |\n| `openai-audio` | text | GET \u002Ftext\u002F{prompt}?model=openai-audio&voice=alloy | ✅ (MP3) |\n| `claude` | text | POST \u002Fv1\u002Fchat\u002Fcompletions | ✅ |\n| `gemini-fast` | text | POST \u002Fv1\u002Fchat\u002Fcompletions | ✅ |\n| `flux` | image | GET \u002Fimage\u002F{prompt} | ✅ |\n| `nanobanana-pro` | image | GET \u002Fimage\u002F{prompt} | ✅ |\n| `seedream-pro` | image | GET \u002Fimage\u002F{prompt} | ✅ |\n| `seedance-pro` | video | GET \u002Fimage\u002F{prompt} | ✅ (MP4) |\n",{"name":5,"description":7},"wan-aspect-ratio-fix","0cf9c44b2cb868c1431babf98007a269e7a2b66b",".claude\u002Fskills\u002Fmodel-debugging","https:\u002F\u002Fgithub.com\u002FAPPLEPIE6969\u002FPollinations","https:\u002F\u002Fgithub.com\u002FAPPLEPIE6969","2026-07-26 00:50:46",{"source":8,"skillName":5,"commands":59,"prompt":69,"zipAvailable":70},[60,63,66],{"tool":61,"command":62},"npx","npx skills add APPLEPIE6969\u002FPollinations --skill model-debugging",{"tool":64,"command":65},"bunx","bunx skills add APPLEPIE6969\u002FPollinations --skill model-debugging",{"tool":67,"command":68},"pnpm","pnpm dlx skills add APPLEPIE6969\u002FPollinations --skill model-debugging","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FAPPLEPIE6969\u002FPollinations\u002Fblob\u002Fwan-aspect-ratio-fix\u002F.claude\u002Fskills\u002Fmodel-debugging\u002FSKILL.md",true,[72,78,82,86],{"path":73,"resourceType":74,"resourceTypeDesc":75,"sizeBytes":76,"sourceHtmlUrl":77},"scripts\u002Fcheck-user-errors.sh","SCRIPT","脚本",1011,"https:\u002F\u002Fgithub.com\u002FAPPLEPIE6969\u002FPollinations\u002Fblob\u002Fwan-aspect-ratio-fix\u002F.claude\u002Fskills\u002Fmodel-debugging\u002Fscripts\u002Fcheck-user-errors.sh",{"path":79,"resourceType":74,"resourceTypeDesc":75,"sizeBytes":80,"sourceHtmlUrl":81},"scripts\u002Ffind-402-users.sh",1041,"https:\u002F\u002Fgithub.com\u002FAPPLEPIE6969\u002FPollinations\u002Fblob\u002Fwan-aspect-ratio-fix\u002F.claude\u002Fskills\u002Fmodel-debugging\u002Fscripts\u002Ffind-402-users.sh",{"path":83,"resourceType":74,"resourceTypeDesc":75,"sizeBytes":84,"sourceHtmlUrl":85},"scripts\u002Ffind-403-users.sh",1018,"https:\u002F\u002Fgithub.com\u002FAPPLEPIE6969\u002FPollinations\u002Fblob\u002Fwan-aspect-ratio-fix\u002F.claude\u002Fskills\u002Fmodel-debugging\u002Fscripts\u002Ffind-403-users.sh",{"path":87,"resourceType":74,"resourceTypeDesc":75,"sizeBytes":88,"sourceHtmlUrl":89},"scripts\u002Ffind-500-errors.sh",985,"https:\u002F\u002Fgithub.com\u002FAPPLEPIE6969\u002FPollinations\u002Fblob\u002Fwan-aspect-ratio-fix\u002F.claude\u002Fskills\u002Fmodel-debugging\u002Fscripts\u002Ffind-500-errors.sh",[91,119,139,164,187,202],{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":95,"categories":96,"keywords":99,"publishedAt":47},"01M6R5KBQ6ARHS4YP4XZX06DJC","polli","Generate images, text, audio, video, and transcribe speech via the Pollinations API using the polli CLI. Use when asked to generate media, call pollinations.ai, check pollen balance, list models, manage API keys, inspect quests, manage my-models, or run polli commands.","https:\u002F\u002Fgithub.com\u002FAPPLEPIE6969\u002FPollinations\u002Fblob\u002Fwan-aspect-ratio-fix\u002Fpackages\u002Fpolli-cli\u002FSKILL.md",[97,98],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},{"categoryNo":23,"categoryName":24,"taxonomyType":20,"taxonomyTypeDesc":21},[93,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118],"cli","generate","run","speech","api","Video","Media","Audio","Bash","text","list","images","commands","balance","quests","my","inspect","api keys","when",{"skillNo":120,"name":121,"nameZh":6,"description":122,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":123,"categories":124,"keywords":125,"publishedAt":47},"01M6R5K5FEQFK9BRNY2S06M73S","founder-meditation","When something goes wrong — build failures, crashes, errors, test failures, deployment issues — generate a short calming meditation with TTS audio to reassure the user that everything will be okay.","https:\u002F\u002Fgithub.com\u002FAPPLEPIE6969\u002FPollinations\u002Fblob\u002Fwan-aspect-ratio-fix\u002Fapps\u002Fopenclaw\u002Fskills\u002Ffounder-meditation\u002FSKILL.md",[],[126,127,128,129,101,130,131,104,107,108,132,133,134,135,136,40,137,44,138,118],"founder","founder meditation","test","deployment","issues","https","tts","curl","short","user","build","with","Wrong",{"skillNo":140,"name":141,"nameZh":6,"description":142,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":143,"categories":144,"keywords":148,"publishedAt":47},"01M6R5JT4Q4B2SE36ZP416AJ4Q","web-research","Query Pollinations text API with web-search models (gemini-search, perplexity-fast, nomnom, etc.). Use when you need web search grounded answers via Pollinations.","https:\u002F\u002Fgithub.com\u002FAPPLEPIE6969\u002FPollinations\u002Fblob\u002Fwan-aspect-ratio-fix\u002F.claude\u002Fskills\u002Fweb-research\u002FSKILL.md",[145],{"categoryNo":146,"categoryName":147,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000021","资料检索与研究",[149,150,151,152,153,154,155,104,156,109,157,158,40,159,160,161,137,162,163,118],"research","web","web research","gemini","search","fast","perplexity","web search","use","models","web-search","query","via","you","nomnom",{"skillNo":165,"name":166,"nameZh":6,"description":167,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":168,"categories":169,"keywords":170,"publishedAt":47},"01M6R5JMK3CJN4VEKVX23745BR","voting-status","Create and post ASCII art voting status diagrams to GitHub issues and Discord. Use when asked to update voting, show voting results, or announce voting status.","https:\u002F\u002Fgithub.com\u002FAPPLEPIE6969\u002FPollinations\u002Fblob\u002Fwan-aspect-ratio-fix\u002F.claude\u002Fskills\u002Fvoting-status\u002FSKILL.md",[],[171,172,173,174,175,176,177,178,130,179,157,180,181,182,183,184,185,186,118],"status","voting","voting status","discord","github","create","update","github issues","results","ascii art","diagrams","Art","show","ascii","post","announce",{"skillNo":188,"name":189,"nameZh":6,"description":190,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":191,"categories":192,"keywords":196,"publishedAt":47},"01M6R5J6ZK52VS1783EJH36137","tinybird-deploy","Deploy Tinybird pipes and datasources for enter.pollinations.ai observability with the Tinybird Forward CLI.","https:\u002F\u002Fgithub.com\u002FAPPLEPIE6969\u002FPollinations\u002Fblob\u002Fwan-aspect-ratio-fix\u002F.claude\u002Fskills\u002Ftinybird-deploy\u002FSKILL.md",[193],{"categoryNo":194,"categoryName":195,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000011","运维与部署",[197,198,199,100,200,40,137,201],"deploy","tinybird","tinybird deploy","observability","for",{"skillNo":203,"name":204,"nameZh":6,"description":205,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":206,"categories":207,"keywords":208,"publishedAt":47},"01M6R5HYNSNJBPMC3SQ7PZVM5Y","sync-production","Sync the main branch into production via the GitHub merges API. Pre-checks for merge conflicts and bails out with a clear reason if the merge isn't clean. Use when asked to merge main into production, deploy main to production, or sync production.","https:\u002F\u002Fgithub.com\u002FAPPLEPIE6969\u002FPollinations\u002Fblob\u002Fwan-aspect-ratio-fix\u002F.claude\u002Fskills\u002Fsync-production\u002FSKILL.md",[],[209,210,211,175,212,197,104,213,157,214,215,216,217,218,161,137,201,219,118],"sync","Production","sync production","merge","branch","merge conflicts","conflicts","clean","main","clear","reason"]