[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6AD9C2NA8BE48P04WSRM5D7":3,"skill-same:01M6AD9C2NA8BE48P04WSRM5D7":65},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":36,"linkOnly":37,"markdownContent":38,"markdownBody":39,"frontMatter":40,"refName":41,"commitSha":42,"skillDirectory":43,"forksCount":44,"repositoryHtmlUrl":45,"repositoryHomepageUrl":46,"ownerHtmlUrl":47,"ownerBlogUrl":48,"ownerDescription":49,"updatedAt":50,"install":51,"resources":64},"01M6AD9C2NA8BE48P04WSRM5D7","bigquery-pipeline-audit",null,"Audits Python + BigQuery pipelines for cost safety, idempotency, and production readiness. Returns a structured report with exact patch locations.","github\u002Fawesome-copilot","github","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F9919?v=4","01M6AC83DS133TPXFSP1DHNCC6","GitHub","01M6AD3PBDYPK7NVFJNJ66Y8RS",41081,"MIT","https:\u002F\u002Fgithub.com\u002Fgithub\u002Fawesome-copilot\u002Fblob\u002Fmain\u002Fskills\u002Fbigquery-pipeline-audit\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35],"bigquery","audit","bigquery pipeline audit","pipeline","returns","for","with","readiness","structured","production readiness","pipelines","patch","cost","python","Safety","Production","report","2026-11-10 10:12:30",false,"---\nname: bigquery-pipeline-audit\ndescription: 'Audits Python + BigQuery pipelines for cost safety, idempotency, and production readiness. Returns a structured report with exact patch locations.'\n---\n\n# BigQuery Pipeline Audit: Cost, Safety and Production Readiness\n\nYou are a senior data engineer reviewing a Python + BigQuery pipeline script.\nYour goals: catch runaway costs before they happen, ensure reruns do not corrupt\ndata, and make sure failures are visible.\n\nAnalyze the codebase and respond in the structure below (A to F + Final).\nReference exact function names and line locations. Suggest minimal fixes, not\nrewrites.\n\n---\n\n## A) COST EXPOSURE: What will actually get billed?\n\nLocate every BigQuery job trigger (`client.query`, `load_table_from_*`,\n`extract_table`, `copy_table`, DDL\u002FDML via query) and every external call\n(APIs, LLM calls, storage writes).\n\nFor each, answer:\n- Is this inside a loop, retry block, or async gather?\n- What is the realistic worst-case call count?\n- For each `client.query`, is `QueryJobConfig.maximum_bytes_billed` set?\n  For load, extract, and copy jobs, is the scope bounded and counted against MAX_JOBS?\n- Is the same SQL and params being executed more than once in a single run?\n  Flag repeated identical queries and suggest query hashing plus temp table caching.\n\n**Flag immediately if:**\n- Any BQ query runs once per date or once per entity in a loop\n- Worst-case BQ job count exceeds 20\n- `maximum_bytes_billed` is missing on any `client.query` call\n\n---\n\n## B) DRY RUN AND EXECUTION MODES\n\nVerify a `--mode` flag exists with at least `dry_run` and `execute` options.\n\n- `dry_run` must print the plan and estimated scope with zero billed BQ execution\n  (BigQuery dry-run estimation via job config is allowed) and zero external API or LLM calls\n- `execute` requires explicit confirmation for prod (`--env=prod --confirm`)\n- Prod must not be the default environment\n\nIf missing, propose a minimal `argparse` patch with safe defaults.\n\n---\n\n## C) BACKFILL AND LOOP DESIGN\n\n**Hard fail if:** the script runs one BQ query per date or per entity in a loop.\n\nCheck that date-range backfills use one of:\n1. A single set-based query with `GENERATE_DATE_ARRAY`\n2. A staging table loaded with all dates then one join query\n3. Explicit chunks with a hard `MAX_CHUNKS` cap\n\nAlso check:\n- Is the date range bounded by default (suggest 14 days max without `--override`)?\n- If the script crashes mid-run, is it safe to re-run without double-writing?\n- For backdated simulations, verify data is read from time-consistent snapshots\n  (`FOR SYSTEM_TIME AS OF`, partitioned as-of tables, or dated snapshot tables).\n  Flag any read from a \"latest\" or unversioned table when running in backdated mode.\n\nSuggest a concrete rewrite if the current approach is row-by-row.\n\n---\n\n## D) QUERY SAFETY AND SCAN SIZE\n\nFor each query, check:\n- **Partition filter** is on the raw column, not `DATE(ts)`, `CAST(...)`, or\n  any function that prevents pruning\n- **No `SELECT *`**: only columns actually used downstream\n- **Joins will not explode**: verify join keys are unique or appropriately scoped\n  and flag any potential many-to-many\n- **Expensive operations** (`REGEXP`, `JSON_EXTRACT`, UDFs) only run after\n  partition filtering, not on full table scans\n\nProvide a specific SQL fix for any query that fails these checks.\n\n---\n\n## E) SAFE WRITES AND IDEMPOTENCY\n\nIdentify every write operation. Flag plain `INSERT`\u002Fappend with no dedup logic.\n\nEach write should use one of:\n1. `MERGE` on a deterministic key (e.g., `entity_id + date + model_version`)\n2. Write to a staging table scoped to the run, then swap or merge into final\n3. Append-only with a dedupe view:\n   `QUALIFY ROW_NUMBER() OVER (PARTITION BY \u003Ckey>) = 1`\n\nAlso check:\n- Will a re-run create duplicate rows?\n- Is the write disposition (`WRITE_TRUNCATE` vs `WRITE_APPEND`) intentional\n  and documented?\n- Is `run_id` being used as part of the merge or dedupe key? If so, flag it.\n  `run_id` should be stored as a metadata column, not as part of the uniqueness\n  key, unless you explicitly want multi-run history.\n\nState the recommended approach and the exact dedup key for this codebase.\n\n---\n\n## F) OBSERVABILITY: Can you debug a failure?\n\nVerify:\n- Failures raise exceptions and abort with no silent `except: pass` or warn-only\n- Each BQ job logs: job ID, bytes processed or billed when available,\n  slot milliseconds, and duration\n- A run summary is logged or written at the end containing:\n  `run_id, env, mode, date_range, tables written, total BQ jobs, total bytes`\n- `run_id` is present and consistent across all log lines\n\nIf `run_id` is missing, propose a one-line fix:\n`run_id = run_id or datetime.utcnow().strftime('%Y%m%dT%H%M%S')`\n\n---\n\n## Final\n\n**1. PASS \u002F FAIL** with specific reasons per section (A to F).\n**2. Patch list** ordered by risk, referencing exact functions to change.\n**3. If FAIL: Top 3 cost risks** with a rough worst-case estimate\n(e.g., \"loop over 90 dates x 3 retries = 270 BQ jobs\").\n","# BigQuery Pipeline Audit: Cost, Safety and Production Readiness\n\nYou are a senior data engineer reviewing a Python + BigQuery pipeline script.\nYour goals: catch runaway costs before they happen, ensure reruns do not corrupt\ndata, and make sure failures are visible.\n\nAnalyze the codebase and respond in the structure below (A to F + Final).\nReference exact function names and line locations. Suggest minimal fixes, not\nrewrites.\n\n---\n\n## A) COST EXPOSURE: What will actually get billed?\n\nLocate every BigQuery job trigger (`client.query`, `load_table_from_*`,\n`extract_table`, `copy_table`, DDL\u002FDML via query) and every external call\n(APIs, LLM calls, storage writes).\n\nFor each, answer:\n- Is this inside a loop, retry block, or async gather?\n- What is the realistic worst-case call count?\n- For each `client.query`, is `QueryJobConfig.maximum_bytes_billed` set?\n  For load, extract, and copy jobs, is the scope bounded and counted against MAX_JOBS?\n- Is the same SQL and params being executed more than once in a single run?\n  Flag repeated identical queries and suggest query hashing plus temp table caching.\n\n**Flag immediately if:**\n- Any BQ query runs once per date or once per entity in a loop\n- Worst-case BQ job count exceeds 20\n- `maximum_bytes_billed` is missing on any `client.query` call\n\n---\n\n## B) DRY RUN AND EXECUTION MODES\n\nVerify a `--mode` flag exists with at least `dry_run` and `execute` options.\n\n- `dry_run` must print the plan and estimated scope with zero billed BQ execution\n  (BigQuery dry-run estimation via job config is allowed) and zero external API or LLM calls\n- `execute` requires explicit confirmation for prod (`--env=prod --confirm`)\n- Prod must not be the default environment\n\nIf missing, propose a minimal `argparse` patch with safe defaults.\n\n---\n\n## C) BACKFILL AND LOOP DESIGN\n\n**Hard fail if:** the script runs one BQ query per date or per entity in a loop.\n\nCheck that date-range backfills use one of:\n1. A single set-based query with `GENERATE_DATE_ARRAY`\n2. A staging table loaded with all dates then one join query\n3. Explicit chunks with a hard `MAX_CHUNKS` cap\n\nAlso check:\n- Is the date range bounded by default (suggest 14 days max without `--override`)?\n- If the script crashes mid-run, is it safe to re-run without double-writing?\n- For backdated simulations, verify data is read from time-consistent snapshots\n  (`FOR SYSTEM_TIME AS OF`, partitioned as-of tables, or dated snapshot tables).\n  Flag any read from a \"latest\" or unversioned table when running in backdated mode.\n\nSuggest a concrete rewrite if the current approach is row-by-row.\n\n---\n\n## D) QUERY SAFETY AND SCAN SIZE\n\nFor each query, check:\n- **Partition filter** is on the raw column, not `DATE(ts)`, `CAST(...)`, or\n  any function that prevents pruning\n- **No `SELECT *`**: only columns actually used downstream\n- **Joins will not explode**: verify join keys are unique or appropriately scoped\n  and flag any potential many-to-many\n- **Expensive operations** (`REGEXP`, `JSON_EXTRACT`, UDFs) only run after\n  partition filtering, not on full table scans\n\nProvide a specific SQL fix for any query that fails these checks.\n\n---\n\n## E) SAFE WRITES AND IDEMPOTENCY\n\nIdentify every write operation. Flag plain `INSERT`\u002Fappend with no dedup logic.\n\nEach write should use one of:\n1. `MERGE` on a deterministic key (e.g., `entity_id + date + model_version`)\n2. Write to a staging table scoped to the run, then swap or merge into final\n3. Append-only with a dedupe view:\n   `QUALIFY ROW_NUMBER() OVER (PARTITION BY \u003Ckey>) = 1`\n\nAlso check:\n- Will a re-run create duplicate rows?\n- Is the write disposition (`WRITE_TRUNCATE` vs `WRITE_APPEND`) intentional\n  and documented?\n- Is `run_id` being used as part of the merge or dedupe key? If so, flag it.\n  `run_id` should be stored as a metadata column, not as part of the uniqueness\n  key, unless you explicitly want multi-run history.\n\nState the recommended approach and the exact dedup key for this codebase.\n\n---\n\n## F) OBSERVABILITY: Can you debug a failure?\n\nVerify:\n- Failures raise exceptions and abort with no silent `except: pass` or warn-only\n- Each BQ job logs: job ID, bytes processed or billed when available,\n  slot milliseconds, and duration\n- A run summary is logged or written at the end containing:\n  `run_id, env, mode, date_range, tables written, total BQ jobs, total bytes`\n- `run_id` is present and consistent across all log lines\n\nIf `run_id` is missing, propose a one-line fix:\n`run_id = run_id or datetime.utcnow().strftime('%Y%m%dT%H%M%S')`\n\n---\n\n## Final\n\n**1. PASS \u002F FAIL** with specific reasons per section (A to F).\n**2. Patch list** ordered by risk, referencing exact functions to change.\n**3. If FAIL: Top 3 cost risks** with a rough worst-case estimate\n(e.g., \"loop over 90 dates x 3 retries = 270 BQ jobs\").\n",{"name":5,"description":7},"main","5269f7ea2d6c3a9bfda5ed878b6726cf509121fd","skills\u002Fbigquery-pipeline-audit",5344,"https:\u002F\u002Fgithub.com\u002Fgithub\u002Fawesome-copilot","https:\u002F\u002Fawesome-copilot.github.com\u002F","https:\u002F\u002Fgithub.com\u002Fgithub","https:\u002F\u002Fgithub.com\u002Fabout","How people build software.","2026-11-13 02:05:36",{"source":8,"skillName":5,"commands":52,"prompt":62,"zipAvailable":63},[53,56,59],{"tool":54,"command":55},"npx","npx skills add github\u002Fawesome-copilot --skill bigquery-pipeline-audit",{"tool":57,"command":58},"bunx","bunx skills add github\u002Fawesome-copilot --skill bigquery-pipeline-audit",{"tool":60,"command":61},"pnpm","pnpm dlx skills add github\u002Fawesome-copilot --skill bigquery-pipeline-audit","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fgithub\u002Fawesome-copilot\u002Fblob\u002Fmain\u002Fskills\u002Fbigquery-pipeline-audit\u002FSKILL.md",true,[],[66,100,126,156,190,207],{"skillNo":67,"name":68,"nameZh":6,"description":69,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":70,"categories":71,"keywords":80,"publishedAt":36},"01M70KXZNX9ES38N2A8M58ZZMZ","supabase-rls-policies","Write, review, and test Supabase Row Level Security (RLS) policies so that one user cannot read or change another user's rows. Use when creating or changing tables, policies, views, or security definer functions in a Supabase project, when reviewing a migration that touches RLS, or when asked to prove user isolation with pgTAP tests that sign in as two users.","https:\u002F\u002Fgithub.com\u002Fgithub\u002Fawesome-copilot\u002Fblob\u002Fmain\u002Fskills\u002Fsupabase-rls-policies\u002FSKILL.md",[72,77],{"categoryNo":73,"categoryName":74,"taxonomyType":75,"taxonomyTypeDesc":76},"01JBSEEDCAT000000000000008","测试与质量","SCENARIO","使用场景",{"categoryNo":78,"categoryName":79,"taxonomyType":75,"taxonomyTypeDesc":76},"01JBSEEDCAT000000000000023","Skill与智能体开发",[81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,25],"supabase","rls","supabase rls","supabase rls policies","project","when","review","change","read","write","security","isolation","tables","as","tests","prove","use","users","functions",{"skillNo":101,"name":102,"nameZh":6,"description":103,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":104,"categories":105,"keywords":107,"publishedAt":36},"01M70KWA5YWXDV6V8DZ19E3T2Y","skill-distribution-audit","Audit a skill collection before release or investigate installer omissions, archived\u002Ftemplate leakage, duplicate identities, and manifest drift. Reconcile the intended public source set, distribution manifest, and actual discovery output for a specified consumer version. Includes a deterministic set-comparison helper.","https:\u002F\u002Fgithub.com\u002Fgithub\u002Fawesome-copilot\u002Fblob\u002Fmain\u002Fskills\u002Fskill-distribution-audit\u002FSKILL.md",[106],{"categoryNo":78,"categoryName":79,"taxonomyType":75,"taxonomyTypeDesc":76},[108,109,20,110,111,112,24,113,114,115,116,117,118,119,120,121,122,123,124,125],"distribution","skill distribution audit","SET","deterministic","installer","discovery","comparison","helper","manifest","version","collection","drift","output","investigate","duplicate","template","release","reconcile",{"skillNo":127,"name":128,"nameZh":6,"description":129,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":130,"sourceHtmlUrl":131,"categories":132,"keywords":138,"publishedAt":36},"01M70KCJPFVW7PPQ49CC5R8JGK","logicapps-custom-code-net10-migration","Migrate Azure Logic Apps Standard local-function custom-code projects from .NET 8 to .NET 10. Use for legacy Logic Apps custom-code workspace upgrades, project\u002Fpackage configuration, custom-worker settings, publication, and integrated VS Code debugging. Not for ordinary .NET applications, standalone Azure Functions, Consumption workflows, or cloud deployment.","https:\u002F\u002Fgithub.com\u002Fgithub\u002Fawesome-copilot\u002Fblob\u002Fmain\u002Fskills\u002Flogicapps-custom-code-net10-migration\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fgithub\u002Fawesome-copilot\u002Fblob\u002Fmain\u002Fskills\u002Flogicapps-custom-code-net10-migration\u002FSKILL.md",[133],{"categoryNo":134,"categoryName":135,"taxonomyType":136,"taxonomyTypeDesc":137},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",[139,140,141,142,143,144,24,85,145,146,147,148,149,150,151,97,152,153,154,155],"custom","code","logicapps custom code net10 migration","migration","projects","standalone","deployment","applications","local","azure","worker","upgrades","package","apps","Legacy","debugging","workflows",{"skillNo":157,"name":158,"nameZh":6,"description":159,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":160,"sourceHtmlUrl":161,"categories":162,"keywords":167,"publishedAt":36},"01M6AEBN37RQMXJ3298PEY7PG1","winmd-api-search","Find and explore Windows desktop APIs. Use when building features that need platform capabilities — camera, file access, notifications, UI controls, AI\u002FML, sensors, networking, etc. Discovers the right API for a task and retrieves full type details (methods, properties, events, enumeration values).","https:\u002F\u002Fgithub.com\u002Fgithub\u002Fawesome-copilot\u002Fblob\u002Fmain\u002Fskills\u002Fwinmd-api-search\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fgithub\u002Fawesome-copilot\u002Fblob\u002Fmain\u002Fskills\u002Fwinmd-api-search\u002FSKILL.md",[163,166],{"categoryNo":164,"categoryName":165,"taxonomyType":75,"taxonomyTypeDesc":76},"01JBSEEDCAT000000000000010","后端与接口",{"categoryNo":134,"categoryName":135,"taxonomyType":136,"taxonomyTypeDesc":137},[168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,86,183,184,185,186,187,24,188,189],"winmd api search","search","api","windows","desktop","file access","enumeration","ui","ml","file","networking","capabilities","notifications","explore","access","features","full","camera","events","platform","methods","controls",{"skillNo":191,"name":192,"nameZh":6,"description":193,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":194,"licenseFileUrl":195,"sourceHtmlUrl":196,"categories":197,"keywords":203,"publishedAt":36},"01M6AE571R3SC1ZGE202B3NB5S","resemble-detect","Deepfake detection and media safety — detect AI-generated audio, images, video, and text, trace synthesis sources, apply watermarks, verify speaker identity, and analyze media intelligence using Resemble AI","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fgithub\u002Fawesome-copilot\u002Fblob\u002Fmain\u002Fskills\u002Fresemble-detect\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fgithub\u002Fawesome-copilot\u002Fblob\u002Fmain\u002Fskills\u002Fresemble-detect\u002FSKILL.md",[198,199,202],{"categoryNo":164,"categoryName":165,"taxonomyType":75,"taxonomyTypeDesc":76},{"categoryNo":200,"categoryName":201,"taxonomyType":75,"taxonomyTypeDesc":76},"01JBSEEDCAT000000000000016","图像与多媒体",{"categoryNo":78,"categoryName":79,"taxonomyType":75,"taxonomyTypeDesc":76},[204,110,177,205,206],"resemble detect","https","identity",{"skillNo":208,"name":209,"nameZh":6,"description":210,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":194,"licenseFileUrl":211,"sourceHtmlUrl":212,"categories":213,"keywords":218,"publishedAt":36},"01M6AE30BSK2HCY4TQQ22Z8JRD","quality-playbook","Run a complete quality engineering audit on any codebase. Derives behavioral requirements from the code, generates spec-traced functional tests, runs a three-pass code review with regression tests, executes a multi-model spec audit (Council of Three), and produces a consolidated bug report with TDD-verified patches. Finds the 35% of real defects that structural code review alone cannot catch. Works with any language. Trigger on 'quality playbook', 'spec audit', 'Council of Three', 'fitness-to-purpose', or 'coverage theater'.","https:\u002F\u002Fgithub.com\u002Fgithub\u002Fawesome-copilot\u002Fblob\u002Fmain\u002Fskills\u002Fquality-playbook\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fgithub\u002Fawesome-copilot\u002Fblob\u002Fmain\u002Fskills\u002Fquality-playbook\u002FSKILL.md",[214,217],{"categoryNo":215,"categoryName":216,"taxonomyType":75,"taxonomyTypeDesc":76},"01JBSEEDCAT000000000000007","代码审查",{"categoryNo":78,"categoryName":79,"taxonomyType":75,"taxonomyTypeDesc":76},[219,220,221,222,140,87,9,20,223,224,205,35,225,226,227,228,229,230,231,232,233,234,95,235,236,237,238],"playbook","quality playbook","quality","code review","tdd","run","spec","bug report","council","engineering","functional","spec audit","bug","verified","multi-model","three","coverage","model","regression","structural"]