[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M78JJ92HS90GF1Q2PXAS3ZDS":3,"skill-same:01M78JJ92HS90GF1Q2PXAS3ZDS":74},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":27,"commitSha":28,"skillDirectory":29,"forksCount":30,"repositoryHtmlUrl":31,"repositoryHomepageUrl":6,"ownerHtmlUrl":32,"ownerBlogUrl":33,"ownerDescription":34,"updatedAt":35,"install":36,"resources":49},"01M78JJ92HS90GF1Q2PXAS3ZDS","bigquery-migration-metadata-extractor",null,"Extracts database catalog metadata and query logs from data warehouses by running the DWH migration extractor locally or on a temporary Google Cloud VM through the BigQuery Migration Cloud Extractor (credentials held in Secret Manager), and inspects extracted `.zip` archives to report object and volume statistics. Use when extracting database metadata or query logs from Snowflake, Redshift, Teradata, Oracle, PostgreSQL, SQL Server, Hive, etc., deciding between local and cloud (Cloud Extractor) extraction, or inspecting extracted dwh-migration-*.zip archives to print post-extraction statistics. Don't use for SQL dialect translation (use bigquery-migration-sql-translator), exporting table data to Parquet or CSV files (such as Snowflake `COPY INTO` unloads) or loading it into BigQuery (use the BigQuery Data Transfer Service or bigquery-basics), or validating migrated data.","google\u002Fskills","google","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F1342004?v=4","01M698VYEFDM0VJBZDYYJ2T83S","Google","01M698WR0H76ENHE587T9EJT89",24235,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fbigquery-migration-metadata-extractor\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: bigquery-migration-metadata-extractor\nmetadata:\n  version: \"1.0.0\"\n  category: BigDataAndAnalytics\n  publisher: google\ndescription: >-\n  Extracts database catalog metadata and query logs from data warehouses by\n  running the DWH migration extractor locally or on a temporary Google Cloud VM\n  through the BigQuery Migration Cloud Extractor (credentials held in Secret\n  Manager), and inspects extracted `.zip` archives to report object and volume\n  statistics. Use when extracting database metadata or query logs from\n  Snowflake, Redshift, Teradata, Oracle, PostgreSQL, SQL Server, Hive, etc.,\n  deciding between local and cloud (Cloud Extractor) extraction, or inspecting\n  extracted dwh-migration-*.zip archives to print post-extraction statistics.\n  Don't use for SQL dialect translation (use bigquery-migration-sql-translator),\n  exporting table data to Parquet or CSV files (such as Snowflake `COPY INTO`\n  unloads) or loading it into BigQuery (use the BigQuery Data Transfer Service\n  or bigquery-basics), or validating migrated data.\n---\n\n# BigQuery Migration Metadata Extraction\n\nExtraction runs the DWH migration extractor (`dwh-migration-dumper`) through two\ngcloud commands to collect catalog DDL and query history—not table row data. If\nthe user wants to migrate table rows (such as Snowflake `COPY INTO` unloads),\nask whether they prefer the [BigQuery Data Transfer Service][bq_dts] (including\nthe [Snowflake connector][snowflake_dts]) or exporting to Cloud Storage and\nloading with `bq load`. Both extract commands are in the **alpha** release\ntrack:\n\nCommand                                        | Use for\n:--------------------------------------------- | :------\n`gcloud alpha bq migration metadata extract`   | Catalog metadata (metadata connectors)\n`gcloud alpha bq migration query-logs extract` | Query history (`*-logs` connectors)\n\nBoth take the same arguments. gcloud flags come first; the extractor's own flags\ncome after `--` and are passed to the extractor unchanged:\n\n```text\ngcloud alpha bq migration {metadata|query-logs} extract \\\n    --connector=CONNECTOR --output=gs:\u002F\u002FBUCKET\u002FPATH\u002F \\\n    [--run-mode=local|cloud] [cloud flags] \\\n    -- [EXTRACTOR FLAGS] [PASSWORD]\n```\n\n*   `--connector` and `--output` are required gcloud flags. `--output` is a\n    Cloud Storage path (`gs:\u002F\u002F...`); local mode also accepts a local file or\n    directory path. This skill uses Cloud Storage in both run modes.\n*   `--run-mode=local` (the default) runs the extractor on this machine.\n    `--run-mode=cloud` runs it on a temporary Compute Engine VM (the **BigQuery\n    Migration Cloud Extractor**).\n*   Database flags (`--host`, `--port`, `--user`, `--database`, `--schema`,\n    `--warehouse`, `--role`, `--driver`, `--assessment`, `--query-log-*`, ...)\n    go after `--`. Putting them before `--` is an error.\n*   How the password is passed depends on the run mode (Section 4): an extractor\n    response file (`@FILE`) in local mode, and `--password` read from a file in\n    cloud mode. Never type a literal password.\n\n> [!IMPORTANT]\n> The commands are in the alpha track. If gcloud reports an invalid choice or\n> unknown command, ask the user before updating the Google Cloud CLI, then\n> retry. Update it with `gcloud components update`, or, if gcloud was installed\n> with a system package manager such as `apt` or `yum` (which disables the\n> component manager), upgrade the `google-cloud-cli` package with that package\n> manager.\n\n## Usage Attribution\n\n**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you\nMUST read and follow the instructions in\n[Usage Attribution](references\u002Fattribution.md).\n\n## Before you begin\n\n1.  **Google Cloud CLI, authentication, project, region, and zone**: Ensure the\n    [Google Cloud CLI][gcloud_install] is installed, verify that the Google\n    Cloud project has an active Cloud Billing account, authenticate with user\n    and Application Default Credentials, and configure the target project, a\n    default region such as `us-central1` (check regional service availability at\n    [Google Cloud locations][cloud_locations]), and a zone in that region. Cloud\n    mode takes the VM's zone from `--zone`, then `compute\u002Fzone`, then\n    `us-central1-a`; it does not read `compute\u002Fregion`:\n\n    ```bash\n    gcloud auth login\n    gcloud auth application-default login\n    gcloud config set project {project_id}\n    gcloud config set compute\u002Fregion {region}\n    gcloud config set compute\u002Fzone {zone}\n    ```\n\n2.  **Output bucket**: This skill writes the ZIP archives to a Cloud Storage\n    bucket (`--output`) that the project owns and that is not public, in both\n    run modes. In both modes the command checks the bucket before it starts\n    the extractor, so the caller needs `storage.buckets.list` on the bucket's\n    project and `storage.buckets.getIamPolicy` on the bucket unless public\n    access prevention is enforced on it. In local mode the Application Default\n    Credentials also need to create objects in the bucket.\n\n3.  **Run-mode prerequisites** (ask the user before installing software or\n    enabling APIs):\n\n    *   **Local mode (`--run-mode=local`)**: Java 11 or later (JRE 11+) on\n        `PATH`, the `bqms` gcloud component, vendor JDBC driver JARs\n        (`--driver`) when required, and local network access to the database;\n        see the [local extraction guide](references\u002Flocal_extraction.md).\n        Without a terminal the extract command cannot prompt to install\n        `bqms` and fails, so once the user agrees, install it first with\n        `gcloud components install bqms --quiet` (for a gcloud installed with\n        `apt` or `yum`, install the `google-cloud-cli-bqms` package instead).\n    *   **Cloud mode (`--run-mode=cloud`)**:\n        *   **APIs**: Compute Engine (`compute.googleapis.com`), Secret Manager\n            (`secretmanager.googleapis.com`), Cloud Storage\n            (`storage.googleapis.com`), and Cloud Logging\n            (`logging.googleapis.com`).\n        *   **Network and subnet**: A VPC `--network` and `--subnet` (default\n            `default`) with routing to the database, and the VM's egress IP\n            range allowlisted by the database. The VM also needs internet\n            access to download the extractor and reach Google APIs: keep the\n            default external IP, or with `--no-address` add Cloud NAT in the\n            VM's region. A Cloud VPN or Cloud Interconnect route to the\n            database does not replace it.\n        *   **IAM permissions**: The caller needs roles to create the VM and\n            the secret, for example Compute Admin (`roles\u002Fcompute.admin`) and\n            Secret Manager Admin (`roles\u002Fsecretmanager.admin`), plus the\n            bucket permissions in item 2. With a prepared\n            `--gce-service-account`, the caller also needs Service Account User\n            (`roles\u002Fiam.serviceAccountUser`) on that account, and the account\n            itself needs the roles listed in the\n            [cloud extraction guide](references\u002Fcloud_extraction.md).\n        *   **Default service account**: Without `--gce-service-account`, every\n            run creates or reuses\n            `bqms-cloud-extractor@{project_id}.iam.gserviceaccount.com` and\n            grants it nine project roles, including `roles\u002Feditor`,\n            `roles\u002Fcompute.admin`, and `roles\u002Fcloudkms.cryptoKeyDecrypter`.\n            This needs Service Account Admin (`roles\u002Fiam.serviceAccountAdmin`)\n            and Project IAM Admin (`roles\u002Fresourcemanager.projectIamAdmin`).\n            Without a terminal, gcloud accepts its own \"create it?\" prompt, so\n            get the user's explicit approval before you omit\n            `--gce-service-account`; see the\n            [cloud extraction guide](references\u002Fcloud_extraction.md).\n\n4.  **Cost and cleanup**: Review\n    [Google Cloud Pricing](https:\u002F\u002Fcloud.google.com\u002Fpricing) for Compute Engine,\n    Secret Manager, and Cloud Storage. After a cloud run, delete the run's\n    Secret Manager secret (Section 5).\n\n## 1. Choose the Run Mode\n\n&nbsp;                   | `--run-mode=local` (default)                             | `--run-mode=cloud` (Cloud Extractor)\n:----------------------- | :------------------------------------------------------- | :-----------------------------------\nWhere the extractor runs | This machine (needs a Java runtime)                      | Temporary Compute Engine VM\nNetwork path to the DB   | This machine's network (VPN, on-prem, bastion)           | The VM's VPC; the DB must accept connections from it\nJDBC drivers             | Any, including vendor JARs passed with `--driver`        | Only drivers bundled with the extractor; the VM has no vendor JARs and cannot read a local `--driver` path\nCredentials              | `@FILE` response file read by the local extractor        | `--password` (read from a file) stored with the other extractor arguments in a Secret Manager secret that the VM reads\nFailed extractor tasks   | Partial ZIP still lands in `--output`                    | No ZIP is uploaded; the run fails\nRead next                | [local extraction guide](references\u002Flocal_extraction.md) | [cloud extraction guide](references\u002Fcloud_extraction.md)\n\nPick **local** when any of these is true:\n\n*   The connector needs a JDBC driver that the cloud VM does not have:\n    `teradata`, `teradata-logs`, `teradata14-logs`, `oracle`, `oracle-logs`,\n    `oracle-stats`, `sqlserver`, `mysql`, `netezza`, `vertica`, `vertica-logs`.\n    Locally, the `bqms` launcher downloads the Oracle, SQL Server, and MySQL\n    drivers itself (the local extractor ships without JDBC drivers, so the\n    launcher also downloads the Snowflake, Redshift, PostgreSQL, and Greenplum\n    drivers); Teradata, Netezza, and Vertica need `--driver` (Section 3).\n*   The database is only reachable from the user's network (on-prem, VPN,\n    private endpoint with no route from Google Cloud).\n*   The user cannot create Compute Engine VMs, service accounts, or secrets in\n    the project.\n*   `--host` is `localhost`, `127.0.0.1`, `::1`, or a private address with no\n    route from a Google Cloud VPC. Never use `--run-mode=cloud` for these; the\n    VM cannot reach them (on the VM, `127.0.0.1` is the VM itself, even if the\n    user says the address is routed). Ask for a routable address, or, if the\n    user already accepted a fallback, run locally with the same `--output` and\n    drop the cloud-only flags.\n\nPick **cloud** when the database is reachable from Google Cloud (Snowflake,\nRedshift, BigQuery, cloud-hosted PostgreSQL\u002FGreenplum, Hive metastore) and the\nuser wants nothing installed locally.\n\nTell the user which run mode you chose and why, for example that cloud mode\ncannot load vendor JDBC drivers or reach a database that only the user's network\ncan reach.\n\n## 2. Connectors and Commands\n\nEach connector extracts **either** catalog metadata **or** query logs, so it\nbelongs to exactly one command. Run both commands to get both.\n\nSource                | `metadata extract` connector                                                                          | `query-logs extract` connector\n:-------------------- | :---------------------------------------------------------------------------------------------------- | :-----------------------------\nSnowflake             | `snowflake` (or `snowflake-account-usage-metadata`, `snowflake-information-schema-metadata`)          | `snowflake-logs` (or `snowflake-account-usage-logs`, `snowflake-information-schema-logs`)\nAmazon Redshift       | `redshift`                                                                                            | `redshift-raw-logs` (or `redshift-logs`, `redshift-serverless-logs`)\nTeradata              | `teradata`                                                                                            | `teradata-logs` (version 15+), `teradata14-logs`\nOracle                | `oracle` (DDL), `oracle-stats` (workload statistics for assessment; window set by `--query-log-days`) | `oracle-logs`\nGoogle BigQuery       | `bigquery`                                                                                            | `bigquery-logs`\nGreenplum             | `greenplum`                                                                                           | `greenplum-logs`\nVertica               | `vertica`                                                                                             | `vertica-logs`\nPostgreSQL            | `postgresql`                                                                                          | none\nSQL Server \u002F Azure    | `sqlserver`                                                                                           | none\nMySQL                 | `mysql`                                                                                               | none\nNetezza               | `netezza`                                                                                             | none\nApache Hive metastore | `hiveql`                                                                                              | none\n\n> [!TIP]\n> Catalog metadata is fast (minutes) and is all that SQL translation,\n> schema creation, Parquet export, and static lineage need. Query logs take\n> longer and are only needed for workload assessment or lineage of queries that\n> are not stored as views (for example dbt or Airflow `INSERT ... SELECT` jobs).\n> Always bound log extraction with `--query-log-start`\u002F`--query-log-end` or\n> `--query-log-days` (after `--`).\n\n## 3. Flags\n\ngcloud flags (before `--`):\n\nFlag                                          | Run mode   | Meaning\n:-------------------------------------------- | :--------- | :------\n`--connector`                                 | both       | Required. Must match the command (Section 2).\n`--output`                                    | both       | Required. `gs:\u002F\u002FBUCKET\u002FPATH\u002F` (or a `.zip` object path). Local mode also accepts a local file or directory path.\n`--run-mode`                                  | both       | `local` (default) or `cloud`.\n`--expected-owner`                            | both       | Project that must own the `--output` bucket (defaults to the current project).\n`--endpoint-mode`                             | both       | Standard gcloud regional endpoint selection; leave unset unless the user asks.\n`--project`                                   | both       | Standard gcloud project flag.\n`--zone`, `--secret`, `--network`, and others | cloud only | See the [cloud extraction guide](references\u002Fcloud_extraction.md).\n\nExtractor flags (after `--`):\n\nFlag                                                       | Meaning\n:--------------------------------------------------------- | :------\n`--host`, `--port`, `--url`                                | Server address, or a full JDBC URL that overrides host and port.\n`--user`, `@FILE`                                          | User, and a response file holding the password (Section 4).\n`--database`, `--schema`                                   | Comma-separated lists to limit what is extracted. `postgresql` extracts only the first `--database` and ignores `--schema`; run it once per database, each with its own `--output`.\n`--warehouse`, `--role`                                    | Snowflake session settings (set per session; never change user defaults).\n`--assessment`                                             | Extra data for BigQuery Migration Assessment (omit `--database` and `--schema` when passing `--assessment` to `snowflake`, which rejects combining them). Only changes the output of `teradata`, `teradata-logs`, `redshift`, `redshift-raw-logs`, `snowflake`, `snowflake-logs`, `hiveql`, and `greenplum-logs`; other connectors ignore it.\n`--query-log-start`, `--query-log-end`, `--query-log-days` | Query log window for `*-logs` connectors.\n`--driver`                                                 | Vendor JDBC driver JAR(s) (required for `teradata*`, `vertica*`, and `netezza`; optional override when the local `bqms` launcher downloads the driver). Local mode only.\n\n## 4. Credentials\n\nNever type a literal password on the command line, and never ask the user to\npaste it into the chat. Ask the user to run the password-file command below in\ntheir own terminal, with `DB_PASSWORD` set there (for example with\n`read -rs DB_PASSWORD && export DB_PASSWORD` in `bash` or `zsh`), or to create\nthe file another way, and to give you only its path. The commands stop with an\nerror instead of writing an empty password when `DB_PASSWORD` is unset. Which\nmechanism to use depends on the run mode.\n\n**Local mode: `@FILE` response file.** The local extractor reads `@FILE`\narguments as YAML response files and turns each key into a flag, so a file with\n`password: ...` supplies `--password` without putting it on the command line.\nWrite it so only the user can read it, with the value single-quoted (any `'`\ndoubled) so special characters survive. This form works in `bash`, `zsh`, and\n`sh` and assumes a single-line password:\n\n```bash\n: \"${DB_PASSWORD:?DB_PASSWORD is not set}\" && umask 077 && printf \"password: '%s'\\n\" \"$(printf '%s' \"$DB_PASSWORD\" | sed \"s\u002F'\u002F''\u002Fg\")\" > .\u002Fextractor-secret.yaml\n```\n\nPass `@.\u002Fextractor-secret.yaml` after `--` (for a file outside the current\ndirectory use an absolute path such as `@$HOME\u002Fpath\u002Ffile.yaml`; `@~\u002F...` is not\nexpanded).\n\n**Cloud mode: `--password` read from a file.** The CLI stores the extractor\narguments in Secret Manager exactly as given, and the VM runs the extractor with\nthem. The VM has none of your local files, so an `@FILE` argument fails there.\nPut the password in a file only the user can read and pass it with command\nsubstitution after `--`:\n\n```bash\n: \"${DB_PASSWORD:?DB_PASSWORD is not set}\" && umask 077 && printf '%s' \"${DB_PASSWORD}\" > .\u002Fdb_password\n```\n\n```text\n-- ... --password=\"$(cat .\u002Fdb_password)\"\n```\n\nThe value then lives in the run's Secret Manager secret (delete it after the\nrun, Section 5) and is visible in this machine's process list while gcloud runs,\nso use a machine only the user can access.\n\nDelete the password file when done if you created it; leave a file the user\nsupplied in place.\n\n## 5. Quick Start\n\nLocal (Teradata on-prem with the vendor driver):\n\n```bash\ngcloud alpha bq migration metadata extract \\\n    --connector=teradata \\\n    --output=\"gs:\u002F\u002F{bucket}\u002Fteradata_metadata\u002F\" \\\n    -- \\\n    --host=\"{teradata_host}\" \\\n    --user=\"{db_user}\" \\\n    --driver=\"{path_to}\u002Fterajdbc4.jar\" \\\n    --assessment \\\n    @.\u002Fextractor-secret.yaml\n```\n\nCloud (Snowflake through the Cloud Extractor):\n\n```bash\ngcloud alpha bq migration metadata extract \\\n    --connector=snowflake \\\n    --output=\"gs:\u002F\u002F{bucket}\u002Fsnowflake_metadata\u002F{run_label}\u002F\" \\\n    --run-mode=cloud \\\n    --project=\"{project_id}\" \\\n    --zone=\"{zone}\" \\\n    --secret=\"bqms-extractor-creds-{run_label}\" \\\n    -- \\\n    --host=\"{account_identifier}.snowflakecomputing.com\" \\\n    --user=\"{db_user}\" \\\n    --database=\"{db_name}\" \\\n    --warehouse=\"{db_warehouse}\" \\\n    --role=\"{db_role}\" \\\n    --password=\"$(cat .\u002Fdb_password)\"\n```\n\n`--secret` is optional. Without it the CLI creates\n`bqms-extractor-creds-run-EPOCH_SECONDS`, prints its name, and labels it\n`goog-bqms-extractor-run-id`. If you pass `--secret`, use a new `{run_label}`\nfor every run: for a secret that already exists the CLI only adds a version,\nand deleting that secret later would also delete its earlier versions.\n\nGive every cloud run an `--output` prefix that holds no `.zip` yet, such as a\nnew `{run_label}` folder. gcloud reports success as soon as it finds any\nnon-empty `.zip` under `--output`, without checking when it was written. An\nolder archive there (from an earlier metadata, query-log, or local run) ends\nthe command at once with `Extraction complete` while the new VM is still\nstarting, and deleting the secret then breaks the real run. If the user names a\nprefix, use it as given after checking that `gcloud storage ls\n\"gs:\u002F\u002F{bucket}\u002F{path}\u002F\"` lists no `.zip`; otherwise propose a new sub-prefix.\n\nEvery answer that runs or recommends an extraction must include these steps, in\norder, with their commands:\n\n1.  Check the exit code and task summary (Section 6, step 1).\n2.  Verify the output with the statistics template (Section 6, step 2): `cp\n    \u003CSKILL_DIR>\u002Fassets\u002Fextraction_stats_template.py .\u002Fextraction_stats.py &&\n    python3 .\u002Fextraction_stats.py --input \"gs:\u002F\u002F{bucket}\u002F{path}\u002F\"`.\n3.  After a cloud run, delete the run's Secret Manager secret once steps 1 and 2\n    are done (or right away if the run failed), and say why: it still holds the\n    database password. Delete the `--secret` name if you passed one: `gcloud\n    secrets delete bqms-extractor-creds-{run_label} --project={project_id}\n    --quiet`. Otherwise delete the auto-named\n    `bqms-extractor-creds-run-EPOCH_SECONDS` secret the command printed, or find\n    it with `gcloud secrets list\n    --filter=\"labels.goog-bqms-extractor-run-id:*\"`. Only delete secrets created\n    for this run; if the user had you reuse an existing secret, record its\n    newest version before the run and destroy only a version this run added.\n    See the cleanup section of the\n    [cloud extraction guide](references\u002Fcloud_extraction.md).\n4.  Delete `extractor-secret.yaml` or `db_password` if you created it; leave a\n    file the user supplied in place.\n\n## 6. Check the Result\n\n1.  **Exit code and task summary.** The extractor exits 1 when any required task\n    fails. What that means depends on the run mode:\n\n    *   **Local:** gcloud reports `BigQuery Migration Extractor failed with exit\n        code 1`, but the extractor has already written the ZIP to `--output`,\n        with a `*.exception.txt` entry for each failure. A ZIP present after\n        this error is a partial success, not a reason to rerun.\n    *   **Cloud:** the VM uploads the ZIP only when the extractor exits 0, so\n        `Extraction complete` means success only if `--output` held no `.zip`\n        before the run (Section 5). If any required task fails, nothing is\n        uploaded and gcloud reports `Extraction finished, but no non-empty\n        artifact was found`. Always tell the user both next steps: read the\n        VM's logs to see why the extractor failed, and then either fix the\n        cause and rerun, or rerun with `--run-mode=local`, which keeps the\n        partial ZIP. gcloud prints a `gcloud logging read\n        'resource.type=\"gce_instance\" AND resource.labels.instance_id=\"...\"'`\n        command only when it finds error entries; otherwise query the logs by\n        the VM name from the `Instance Link:` line (see Monitoring in the\n        [cloud extraction guide](references\u002Fcloud_extraction.md)). Also delete\n        the run's secret (Section 5).\n\n    For a local partial ZIP, read the `N TASKS SUCCEEDED \u002F N TASKS FAILED`\n    summary and the exception entries (`unzip -l {archive}.zip\n    '*.exception.txt'` lists the failed tasks; each file is long, so read just\n    the error with `unzip -p {archive}.zip '{task}.exception.txt' | grep -m 3 -E\n    '^\\s*message:'`). A failure is harmless when a version-specific fallback of\n    the same task succeeded; the fallback has a version suffix, for example\n    `functions.csv.exception.txt` next to a populated `functions-11.csv` on\n    PostgreSQL 11 and later.\n2.  **Archive integrity and statistics.** Copy the\n    [extraction statistics template][extraction_stats_template]\n    (`\u003CSKILL_DIR>\u002Fassets\u002Fextraction_stats_template.py`, where `\u003CSKILL_DIR>` is\n    the directory that contains this `SKILL.md`) into the working directory and\n    run it on the ZIP, its folder, or the GCS prefix:\n\n    ```bash\n    cp \u003CSKILL_DIR>\u002Fassets\u002Fextraction_stats_template.py .\u002Fextraction_stats.py\n    python3 .\u002Fextraction_stats.py --input \"gs:\u002F\u002F{bucket}\u002Fsnowflake_metadata\u002F\"\n    ```\n\n    If the `gs:\u002F\u002F` form fails, copy the ZIP locally with `gcloud storage cp` and\n    run the script on the local file.\n\n    It rejects incomplete archives (22-byte placeholders) and reports the\n    extractor telemetry status, counts of databases, schemas, base tables,\n    views, and columns, total base table row counts and byte sizes, per-schema\n    and per-table breakdowns, and the column data type distribution. See the\n    [statistics report specification](references\u002Fextraction_stats_report.md)\n    for the full report metrics and the `# CUSTOMIZE:` extension points (for\n    example, editing `_EXCLUDED_SYSTEM_SCHEMAS` or `_should_include_schema()` to\n    drop a `STAGING_TMP` schema).\n\n> [!NOTE]\n> The script computes object, row, and byte counts from the Snowflake, Redshift\n> (`svv_*`), and Teradata (`dbc.*V`) catalog layouts, and table, view, and\n> top-level column counts from the BigQuery `tables.jsonl` (rows and sizes show\n> as `N\u002FA` there because that connector does not record them). It reads the run\n> status from `dumper-telemetry.jsonl` when the archive has one. For other\n> layouts (for example PostgreSQL, Oracle, or the Hive `tables.jsonl`) it still\n> validates the archive and reports the connector and host, but object counts\n> are zero and the run status can show `UNKNOWN`. On PostgreSQL 11 and later it\n> shows `FAILURE` because of the harmless `functions.csv` and `aggregates.csv`\n> fallbacks (Section 6, step 1). Then use the task summary printed by the\n> extract command for the run status, and count objects from the connector's\n> own files (Section 7).\n\n## 7. Output Archives\n\nUnless `--output` names a `.zip` object, the extractor names the ZIP\n`dwh-migration-{connector}-metadata.zip` for `metadata extract` connectors and\n`dwh-migration-{connector}.zip` for `query-logs extract` connectors, whose names\nalready end in `-logs`: `snowflake` writes\n`dwh-migration-snowflake-metadata.zip` and `snowflake-logs` writes\n`dwh-migration-snowflake-logs.zip`. A few names differ (`oracle-stats` writes\n`dwh-migration-oracle-stats.zip`; `hiveql`, and query logs extracted with\n`--assessment`, add a `-YYYYMMDDTHHMMSS` timestamp), so list `--output` to find\nthe archive. Every archive contains `compilerworks-metadata.yaml` (extractor\nversion and arguments) plus connector-specific CSVs:\n\n```text\ndwh-migration-snowflake-metadata.zip\n├── databases-au.csv            # Databases and owners\n├── schemata-au.csv             # Schemas\n├── tables-au.csv               # Tables, row counts, byte sizes\n├── columns-au.csv              # Columns, ordinal positions, types, nullability\n├── views-au.csv                # View DDL\n├── functions-au.csv            # User-defined and external functions\n├── external_tables.csv         # External stage tables\n├── snowflake.yaml              # Assessment flag and warehouse count\n├── compilerworks-metadata.yaml # Extractor version and flags\n└── dumper-telemetry.jsonl      # Task timings\n\ndwh-migration-snowflake-logs.zip\n├── query_history_2026-08-05T000000Z.csv   # One file per rotation window\n├── ...\n└── compilerworks-metadata.yaml\n\ndwh-migration-postgresql-metadata.zip\n├── database.csv                # Databases on the server\n├── is_columns_private.csv      # Columns of user tables and views\n├── pg_views_private.csv        # User view DDL\n├── functions-11.csv            # Functions on search_path (11+; drop pg_catalog)\n├── aggregates-11.csv, types.csv, casts.csv\n├── is_columns_generic.csv, pg_views_generic.csv   # System catalog objects\n├── pg_user.csv                 # Roles and privileges (passwords masked)\n├── *.csv.exception.txt         # One per failed task\n└── compilerworks-{metadata.yaml,arguments.txt,version.txt,format.txt}\n```\n\nIn PostgreSQL archives the `*_private.csv` files hold user objects and the\n`*_generic.csv` files hold system objects. `is_columns_private.csv` includes\nview columns; subtract the views in `pg_views_private.csv` to count base tables.\n`functions-11.csv`, `aggregates-11.csv`, and `types.csv` list only objects\nvisible on the session `search_path` (`\"$user\", public` by default), so other\nschemas' functions and types are missing. To include them, replace `--host`,\n`--port`, and `--database` with\n`--url=\"jdbc:postgresql:\u002F\u002F{host}:{port}\u002F{db_name}?currentSchema={schema1},{schema2},public\"`.\n\nArchives never contain the database password: to confirm, check that\n`compilerworks-arguments.txt` and `compilerworks-metadata.yaml` have no\n`password` entry (scanning for the password value is unreliable when it is a\ncommon word), but they do contain object DDL and role lists, and\n`dumper-telemetry.jsonl` is copied from\n`~\u002F.cache\u002FDWH-Dumper\u002Fall-dumper-telemetry.jsonl`, so it can hold connectors,\nhosts, and errors from earlier extractor runs on the same machine. Treat them\nas sensitive and keep them in a private bucket.\n\n## 8. Downstream BigQuery Workflows\n\nEach cloud run writes to its own `--output` prefix (Section 5). To hand the\narchives over together, copy the finished ZIPs into one shared prefix, for\nexample `gcloud storage cp \"gs:\u002F\u002F{bucket}\u002F{run_prefix}\u002F*.zip\"\n\"gs:\u002F\u002F{bucket}\u002Fassessment\u002F\"`, and never use that shared prefix as a cloud\n`--output`. Then:\n\n1.  **BigQuery Migration Assessment**: direct the user to **BigQuery >\n    Migration > Assessment** in the Google Cloud Console and give it the GCS\n    prefix that holds the metadata (extracted with `--assessment`) and logs.\n2.  **Batch SQL translation**: attach the metadata ZIP as schema context in\n    **BigQuery > Migration > SQL translation**, or through the BigQuery\n    Migration API \u002F BQMS MCP tools (see `bigquery-migration-sql-translator`).\n3.  **Table data transfer (out of scope)**: this skill extracts catalog metadata\n    and query logs only. If the user asks to export or load table rows (such as\n    Snowflake `COPY INTO` unloads), ask whether they want to use the\n    [BigQuery Data Transfer Service][bq_dts] (including the\n    [Snowflake transfer connector][snowflake_dts]) or export to Parquet\u002FCSV in\n    Cloud Storage and load with `bq load`.\n\n[bq_dts]: https:\u002F\u002Fdocs.cloud.google.com\u002Fbigquery\u002Fdocs\u002Fdts-introduction.md.txt\n[cloud_locations]: https:\u002F\u002Fcloud.google.com\u002Fabout\u002Flocations\n[extraction_stats_template]: assets\u002Fextraction_stats_template.py\n[gcloud_install]: https:\u002F\u002Fcloud.google.com\u002Fsdk\u002Fdocs\u002Finstall\n[snowflake_dts]: https:\u002F\u002Fdocs.cloud.google.com\u002Fbigquery\u002Fdocs\u002Fmigration\u002Fsnowflake-transfer.md.txt\n","# BigQuery Migration Metadata Extraction\n\nExtraction runs the DWH migration extractor (`dwh-migration-dumper`) through two\ngcloud commands to collect catalog DDL and query history—not table row data. If\nthe user wants to migrate table rows (such as Snowflake `COPY INTO` unloads),\nask whether they prefer the [BigQuery Data Transfer Service][bq_dts] (including\nthe [Snowflake connector][snowflake_dts]) or exporting to Cloud Storage and\nloading with `bq load`. Both extract commands are in the **alpha** release\ntrack:\n\nCommand                                        | Use for\n:--------------------------------------------- | :------\n`gcloud alpha bq migration metadata extract`   | Catalog metadata (metadata connectors)\n`gcloud alpha bq migration query-logs extract` | Query history (`*-logs` connectors)\n\nBoth take the same arguments. gcloud flags come first; the extractor's own flags\ncome after `--` and are passed to the extractor unchanged:\n\n```text\ngcloud alpha bq migration {metadata|query-logs} extract \\\n    --connector=CONNECTOR --output=gs:\u002F\u002FBUCKET\u002FPATH\u002F \\\n    [--run-mode=local|cloud] [cloud flags] \\\n    -- [EXTRACTOR FLAGS] [PASSWORD]\n```\n\n*   `--connector` and `--output` are required gcloud flags. `--output` is a\n    Cloud Storage path (`gs:\u002F\u002F...`); local mode also accepts a local file or\n    directory path. This skill uses Cloud Storage in both run modes.\n*   `--run-mode=local` (the default) runs the extractor on this machine.\n    `--run-mode=cloud` runs it on a temporary Compute Engine VM (the **BigQuery\n    Migration Cloud Extractor**).\n*   Database flags (`--host`, `--port`, `--user`, `--database`, `--schema`,\n    `--warehouse`, `--role`, `--driver`, `--assessment`, `--query-log-*`, ...)\n    go after `--`. Putting them before `--` is an error.\n*   How the password is passed depends on the run mode (Section 4): an extractor\n    response file (`@FILE`) in local mode, and `--password` read from a file in\n    cloud mode. Never type a literal password.\n\n> [!IMPORTANT]\n> The commands are in the alpha track. If gcloud reports an invalid choice or\n> unknown command, ask the user before updating the Google Cloud CLI, then\n> retry. Update it with `gcloud components update`, or, if gcloud was installed\n> with a system package manager such as `apt` or `yum` (which disables the\n> component manager), upgrade the `google-cloud-cli` package with that package\n> manager.\n\n## Usage Attribution\n\n**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you\nMUST read and follow the instructions in\n[Usage Attribution](references\u002Fattribution.md).\n\n## Before you begin\n\n1.  **Google Cloud CLI, authentication, project, region, and zone**: Ensure the\n    [Google Cloud CLI][gcloud_install] is installed, verify that the Google\n    Cloud project has an active Cloud Billing account, authenticate with user\n    and Application Default Credentials, and configure the target project, a\n    default region such as `us-central1` (check regional service availability at\n    [Google Cloud locations][cloud_locations]), and a zone in that region. Cloud\n    mode takes the VM's zone from `--zone`, then `compute\u002Fzone`, then\n    `us-central1-a`; it does not read `compute\u002Fregion`:\n\n    ```bash\n    gcloud auth login\n    gcloud auth application-default login\n    gcloud config set project {project_id}\n    gcloud config set compute\u002Fregion {region}\n    gcloud config set compute\u002Fzone {zone}\n    ```\n\n2.  **Output bucket**: This skill writes the ZIP archives to a Cloud Storage\n    bucket (`--output`) that the project owns and that is not public, in both\n    run modes. In both modes the command checks the bucket before it starts\n    the extractor, so the caller needs `storage.buckets.list` on the bucket's\n    project and `storage.buckets.getIamPolicy` on the bucket unless public\n    access prevention is enforced on it. In local mode the Application Default\n    Credentials also need to create objects in the bucket.\n\n3.  **Run-mode prerequisites** (ask the user before installing software or\n    enabling APIs):\n\n    *   **Local mode (`--run-mode=local`)**: Java 11 or later (JRE 11+) on\n        `PATH`, the `bqms` gcloud component, vendor JDBC driver JARs\n        (`--driver`) when required, and local network access to the database;\n        see the [local extraction guide](references\u002Flocal_extraction.md).\n        Without a terminal the extract command cannot prompt to install\n        `bqms` and fails, so once the user agrees, install it first with\n        `gcloud components install bqms --quiet` (for a gcloud installed with\n        `apt` or `yum`, install the `google-cloud-cli-bqms` package instead).\n    *   **Cloud mode (`--run-mode=cloud`)**:\n        *   **APIs**: Compute Engine (`compute.googleapis.com`), Secret Manager\n            (`secretmanager.googleapis.com`), Cloud Storage\n            (`storage.googleapis.com`), and Cloud Logging\n            (`logging.googleapis.com`).\n        *   **Network and subnet**: A VPC `--network` and `--subnet` (default\n            `default`) with routing to the database, and the VM's egress IP\n            range allowlisted by the database. The VM also needs internet\n            access to download the extractor and reach Google APIs: keep the\n            default external IP, or with `--no-address` add Cloud NAT in the\n            VM's region. A Cloud VPN or Cloud Interconnect route to the\n            database does not replace it.\n        *   **IAM permissions**: The caller needs roles to create the VM and\n            the secret, for example Compute Admin (`roles\u002Fcompute.admin`) and\n            Secret Manager Admin (`roles\u002Fsecretmanager.admin`), plus the\n            bucket permissions in item 2. With a prepared\n            `--gce-service-account`, the caller also needs Service Account User\n            (`roles\u002Fiam.serviceAccountUser`) on that account, and the account\n            itself needs the roles listed in the\n            [cloud extraction guide](references\u002Fcloud_extraction.md).\n        *   **Default service account**: Without `--gce-service-account`, every\n            run creates or reuses\n            `bqms-cloud-extractor@{project_id}.iam.gserviceaccount.com` and\n            grants it nine project roles, including `roles\u002Feditor`,\n            `roles\u002Fcompute.admin`, and `roles\u002Fcloudkms.cryptoKeyDecrypter`.\n            This needs Service Account Admin (`roles\u002Fiam.serviceAccountAdmin`)\n            and Project IAM Admin (`roles\u002Fresourcemanager.projectIamAdmin`).\n            Without a terminal, gcloud accepts its own \"create it?\" prompt, so\n            get the user's explicit approval before you omit\n            `--gce-service-account`; see the\n            [cloud extraction guide](references\u002Fcloud_extraction.md).\n\n4.  **Cost and cleanup**: Review\n    [Google Cloud Pricing](https:\u002F\u002Fcloud.google.com\u002Fpricing) for Compute Engine,\n    Secret Manager, and Cloud Storage. After a cloud run, delete the run's\n    Secret Manager secret (Section 5).\n\n## 1. Choose the Run Mode\n\n&nbsp;                   | `--run-mode=local` (default)                             | `--run-mode=cloud` (Cloud Extractor)\n:----------------------- | :------------------------------------------------------- | :-----------------------------------\nWhere the extractor runs | This machine (needs a Java runtime)                      | Temporary Compute Engine VM\nNetwork path to the DB   | This machine's network (VPN, on-prem, bastion)           | The VM's VPC; the DB must accept connections from it\nJDBC drivers             | Any, including vendor JARs passed with `--driver`        | Only drivers bundled with the extractor; the VM has no vendor JARs and cannot read a local `--driver` path\nCredentials              | `@FILE` response file read by the local extractor        | `--password` (read from a file) stored with the other extractor arguments in a Secret Manager secret that the VM reads\nFailed extractor tasks   | Partial ZIP still lands in `--output`                    | No ZIP is uploaded; the run fails\nRead next                | [local extraction guide](references\u002Flocal_extraction.md) | [cloud extraction guide](references\u002Fcloud_extraction.md)\n\nPick **local** when any of these is true:\n\n*   The connector needs a JDBC driver that the cloud VM does not have:\n    `teradata`, `teradata-logs`, `teradata14-logs`, `oracle`, `oracle-logs`,\n    `oracle-stats`, `sqlserver`, `mysql`, `netezza`, `vertica`, `vertica-logs`.\n    Locally, the `bqms` launcher downloads the Oracle, SQL Server, and MySQL\n    drivers itself (the local extractor ships without JDBC drivers, so the\n    launcher also downloads the Snowflake, Redshift, PostgreSQL, and Greenplum\n    drivers); Teradata, Netezza, and Vertica need `--driver` (Section 3).\n*   The database is only reachable from the user's network (on-prem, VPN,\n    private endpoint with no route from Google Cloud).\n*   The user cannot create Compute Engine VMs, service accounts, or secrets in\n    the project.\n*   `--host` is `localhost`, `127.0.0.1`, `::1`, or a private address with no\n    route from a Google Cloud VPC. Never use `--run-mode=cloud` for these; the\n    VM cannot reach them (on the VM, `127.0.0.1` is the VM itself, even if the\n    user says the address is routed). Ask for a routable address, or, if the\n    user already accepted a fallback, run locally with the same `--output` and\n    drop the cloud-only flags.\n\nPick **cloud** when the database is reachable from Google Cloud (Snowflake,\nRedshift, BigQuery, cloud-hosted PostgreSQL\u002FGreenplum, Hive metastore) and the\nuser wants nothing installed locally.\n\nTell the user which run mode you chose and why, for example that cloud mode\ncannot load vendor JDBC drivers or reach a database that only the user's network\ncan reach.\n\n## 2. Connectors and Commands\n\nEach connector extracts **either** catalog metadata **or** query logs, so it\nbelongs to exactly one command. Run both commands to get both.\n\nSource                | `metadata extract` connector                                                                          | `query-logs extract` connector\n:-------------------- | :---------------------------------------------------------------------------------------------------- | :-----------------------------\nSnowflake             | `snowflake` (or `snowflake-account-usage-metadata`, `snowflake-information-schema-metadata`)          | `snowflake-logs` (or `snowflake-account-usage-logs`, `snowflake-information-schema-logs`)\nAmazon Redshift       | `redshift`                                                                                            | `redshift-raw-logs` (or `redshift-logs`, `redshift-serverless-logs`)\nTeradata              | `teradata`                                                                                            | `teradata-logs` (version 15+), `teradata14-logs`\nOracle                | `oracle` (DDL), `oracle-stats` (workload statistics for assessment; window set by `--query-log-days`) | `oracle-logs`\nGoogle BigQuery       | `bigquery`                                                                                            | `bigquery-logs`\nGreenplum             | `greenplum`                                                                                           | `greenplum-logs`\nVertica               | `vertica`                                                                                             | `vertica-logs`\nPostgreSQL            | `postgresql`                                                                                          | none\nSQL Server \u002F Azure    | `sqlserver`                                                                                           | none\nMySQL                 | `mysql`                                                                                               | none\nNetezza               | `netezza`                                                                                             | none\nApache Hive metastore | `hiveql`                                                                                              | none\n\n> [!TIP]\n> Catalog metadata is fast (minutes) and is all that SQL translation,\n> schema creation, Parquet export, and static lineage need. Query logs take\n> longer and are only needed for workload assessment or lineage of queries that\n> are not stored as views (for example dbt or Airflow `INSERT ... SELECT` jobs).\n> Always bound log extraction with `--query-log-start`\u002F`--query-log-end` or\n> `--query-log-days` (after `--`).\n\n## 3. Flags\n\ngcloud flags (before `--`):\n\nFlag                                          | Run mode   | Meaning\n:-------------------------------------------- | :--------- | :------\n`--connector`                                 | both       | Required. Must match the command (Section 2).\n`--output`                                    | both       | Required. `gs:\u002F\u002FBUCKET\u002FPATH\u002F` (or a `.zip` object path). Local mode also accepts a local file or directory path.\n`--run-mode`                                  | both       | `local` (default) or `cloud`.\n`--expected-owner`                            | both       | Project that must own the `--output` bucket (defaults to the current project).\n`--endpoint-mode`                             | both       | Standard gcloud regional endpoint selection; leave unset unless the user asks.\n`--project`                                   | both       | Standard gcloud project flag.\n`--zone`, `--secret`, `--network`, and others | cloud only | See the [cloud extraction guide](references\u002Fcloud_extraction.md).\n\nExtractor flags (after `--`):\n\nFlag                                                       | Meaning\n:--------------------------------------------------------- | :------\n`--host`, `--port`, `--url`                                | Server address, or a full JDBC URL that overrides host and port.\n`--user`, `@FILE`                                          | User, and a response file holding the password (Section 4).\n`--database`, `--schema`                                   | Comma-separated lists to limit what is extracted. `postgresql` extracts only the first `--database` and ignores `--schema`; run it once per database, each with its own `--output`.\n`--warehouse`, `--role`                                    | Snowflake session settings (set per session; never change user defaults).\n`--assessment`                                             | Extra data for BigQuery Migration Assessment (omit `--database` and `--schema` when passing `--assessment` to `snowflake`, which rejects combining them). Only changes the output of `teradata`, `teradata-logs`, `redshift`, `redshift-raw-logs`, `snowflake`, `snowflake-logs`, `hiveql`, and `greenplum-logs`; other connectors ignore it.\n`--query-log-start`, `--query-log-end`, `--query-log-days` | Query log window for `*-logs` connectors.\n`--driver`                                                 | Vendor JDBC driver JAR(s) (required for `teradata*`, `vertica*`, and `netezza`; optional override when the local `bqms` launcher downloads the driver). Local mode only.\n\n## 4. Credentials\n\nNever type a literal password on the command line, and never ask the user to\npaste it into the chat. Ask the user to run the password-file command below in\ntheir own terminal, with `DB_PASSWORD` set there (for example with\n`read -rs DB_PASSWORD && export DB_PASSWORD` in `bash` or `zsh`), or to create\nthe file another way, and to give you only its path. The commands stop with an\nerror instead of writing an empty password when `DB_PASSWORD` is unset. Which\nmechanism to use depends on the run mode.\n\n**Local mode: `@FILE` response file.** The local extractor reads `@FILE`\narguments as YAML response files and turns each key into a flag, so a file with\n`password: ...` supplies `--password` without putting it on the command line.\nWrite it so only the user can read it, with the value single-quoted (any `'`\ndoubled) so special characters survive. This form works in `bash`, `zsh`, and\n`sh` and assumes a single-line password:\n\n```bash\n: \"${DB_PASSWORD:?DB_PASSWORD is not set}\" && umask 077 && printf \"password: '%s'\\n\" \"$(printf '%s' \"$DB_PASSWORD\" | sed \"s\u002F'\u002F''\u002Fg\")\" > .\u002Fextractor-secret.yaml\n```\n\nPass `@.\u002Fextractor-secret.yaml` after `--` (for a file outside the current\ndirectory use an absolute path such as `@$HOME\u002Fpath\u002Ffile.yaml`; `@~\u002F...` is not\nexpanded).\n\n**Cloud mode: `--password` read from a file.** The CLI stores the extractor\narguments in Secret Manager exactly as given, and the VM runs the extractor with\nthem. The VM has none of your local files, so an `@FILE` argument fails there.\nPut the password in a file only the user can read and pass it with command\nsubstitution after `--`:\n\n```bash\n: \"${DB_PASSWORD:?DB_PASSWORD is not set}\" && umask 077 && printf '%s' \"${DB_PASSWORD}\" > .\u002Fdb_password\n```\n\n```text\n-- ... --password=\"$(cat .\u002Fdb_password)\"\n```\n\nThe value then lives in the run's Secret Manager secret (delete it after the\nrun, Section 5) and is visible in this machine's process list while gcloud runs,\nso use a machine only the user can access.\n\nDelete the password file when done if you created it; leave a file the user\nsupplied in place.\n\n## 5. Quick Start\n\nLocal (Teradata on-prem with the vendor driver):\n\n```bash\ngcloud alpha bq migration metadata extract \\\n    --connector=teradata \\\n    --output=\"gs:\u002F\u002F{bucket}\u002Fteradata_metadata\u002F\" \\\n    -- \\\n    --host=\"{teradata_host}\" \\\n    --user=\"{db_user}\" \\\n    --driver=\"{path_to}\u002Fterajdbc4.jar\" \\\n    --assessment \\\n    @.\u002Fextractor-secret.yaml\n```\n\nCloud (Snowflake through the Cloud Extractor):\n\n```bash\ngcloud alpha bq migration metadata extract \\\n    --connector=snowflake \\\n    --output=\"gs:\u002F\u002F{bucket}\u002Fsnowflake_metadata\u002F{run_label}\u002F\" \\\n    --run-mode=cloud \\\n    --project=\"{project_id}\" \\\n    --zone=\"{zone}\" \\\n    --secret=\"bqms-extractor-creds-{run_label}\" \\\n    -- \\\n    --host=\"{account_identifier}.snowflakecomputing.com\" \\\n    --user=\"{db_user}\" \\\n    --database=\"{db_name}\" \\\n    --warehouse=\"{db_warehouse}\" \\\n    --role=\"{db_role}\" \\\n    --password=\"$(cat .\u002Fdb_password)\"\n```\n\n`--secret` is optional. Without it the CLI creates\n`bqms-extractor-creds-run-EPOCH_SECONDS`, prints its name, and labels it\n`goog-bqms-extractor-run-id`. If you pass `--secret`, use a new `{run_label}`\nfor every run: for a secret that already exists the CLI only adds a version,\nand deleting that secret later would also delete its earlier versions.\n\nGive every cloud run an `--output` prefix that holds no `.zip` yet, such as a\nnew `{run_label}` folder. gcloud reports success as soon as it finds any\nnon-empty `.zip` under `--output`, without checking when it was written. An\nolder archive there (from an earlier metadata, query-log, or local run) ends\nthe command at once with `Extraction complete` while the new VM is still\nstarting, and deleting the secret then breaks the real run. If the user names a\nprefix, use it as given after checking that `gcloud storage ls\n\"gs:\u002F\u002F{bucket}\u002F{path}\u002F\"` lists no `.zip`; otherwise propose a new sub-prefix.\n\nEvery answer that runs or recommends an extraction must include these steps, in\norder, with their commands:\n\n1.  Check the exit code and task summary (Section 6, step 1).\n2.  Verify the output with the statistics template (Section 6, step 2): `cp\n    \u003CSKILL_DIR>\u002Fassets\u002Fextraction_stats_template.py .\u002Fextraction_stats.py &&\n    python3 .\u002Fextraction_stats.py --input \"gs:\u002F\u002F{bucket}\u002F{path}\u002F\"`.\n3.  After a cloud run, delete the run's Secret Manager secret once steps 1 and 2\n    are done (or right away if the run failed), and say why: it still holds the\n    database password. Delete the `--secret` name if you passed one: `gcloud\n    secrets delete bqms-extractor-creds-{run_label} --project={project_id}\n    --quiet`. Otherwise delete the auto-named\n    `bqms-extractor-creds-run-EPOCH_SECONDS` secret the command printed, or find\n    it with `gcloud secrets list\n    --filter=\"labels.goog-bqms-extractor-run-id:*\"`. Only delete secrets created\n    for this run; if the user had you reuse an existing secret, record its\n    newest version before the run and destroy only a version this run added.\n    See the cleanup section of the\n    [cloud extraction guide](references\u002Fcloud_extraction.md).\n4.  Delete `extractor-secret.yaml` or `db_password` if you created it; leave a\n    file the user supplied in place.\n\n## 6. Check the Result\n\n1.  **Exit code and task summary.** The extractor exits 1 when any required task\n    fails. What that means depends on the run mode:\n\n    *   **Local:** gcloud reports `BigQuery Migration Extractor failed with exit\n        code 1`, but the extractor has already written the ZIP to `--output`,\n        with a `*.exception.txt` entry for each failure. A ZIP present after\n        this error is a partial success, not a reason to rerun.\n    *   **Cloud:** the VM uploads the ZIP only when the extractor exits 0, so\n        `Extraction complete` means success only if `--output` held no `.zip`\n        before the run (Section 5). If any required task fails, nothing is\n        uploaded and gcloud reports `Extraction finished, but no non-empty\n        artifact was found`. Always tell the user both next steps: read the\n        VM's logs to see why the extractor failed, and then either fix the\n        cause and rerun, or rerun with `--run-mode=local`, which keeps the\n        partial ZIP. gcloud prints a `gcloud logging read\n        'resource.type=\"gce_instance\" AND resource.labels.instance_id=\"...\"'`\n        command only when it finds error entries; otherwise query the logs by\n        the VM name from the `Instance Link:` line (see Monitoring in the\n        [cloud extraction guide](references\u002Fcloud_extraction.md)). Also delete\n        the run's secret (Section 5).\n\n    For a local partial ZIP, read the `N TASKS SUCCEEDED \u002F N TASKS FAILED`\n    summary and the exception entries (`unzip -l {archive}.zip\n    '*.exception.txt'` lists the failed tasks; each file is long, so read just\n    the error with `unzip -p {archive}.zip '{task}.exception.txt' | grep -m 3 -E\n    '^\\s*message:'`). A failure is harmless when a version-specific fallback of\n    the same task succeeded; the fallback has a version suffix, for example\n    `functions.csv.exception.txt` next to a populated `functions-11.csv` on\n    PostgreSQL 11 and later.\n2.  **Archive integrity and statistics.** Copy the\n    [extraction statistics template][extraction_stats_template]\n    (`\u003CSKILL_DIR>\u002Fassets\u002Fextraction_stats_template.py`, where `\u003CSKILL_DIR>` is\n    the directory that contains this `SKILL.md`) into the working directory and\n    run it on the ZIP, its folder, or the GCS prefix:\n\n    ```bash\n    cp \u003CSKILL_DIR>\u002Fassets\u002Fextraction_stats_template.py .\u002Fextraction_stats.py\n    python3 .\u002Fextraction_stats.py --input \"gs:\u002F\u002F{bucket}\u002Fsnowflake_metadata\u002F\"\n    ```\n\n    If the `gs:\u002F\u002F` form fails, copy the ZIP locally with `gcloud storage cp` and\n    run the script on the local file.\n\n    It rejects incomplete archives (22-byte placeholders) and reports the\n    extractor telemetry status, counts of databases, schemas, base tables,\n    views, and columns, total base table row counts and byte sizes, per-schema\n    and per-table breakdowns, and the column data type distribution. See the\n    [statistics report specification](references\u002Fextraction_stats_report.md)\n    for the full report metrics and the `# CUSTOMIZE:` extension points (for\n    example, editing `_EXCLUDED_SYSTEM_SCHEMAS` or `_should_include_schema()` to\n    drop a `STAGING_TMP` schema).\n\n> [!NOTE]\n> The script computes object, row, and byte counts from the Snowflake, Redshift\n> (`svv_*`), and Teradata (`dbc.*V`) catalog layouts, and table, view, and\n> top-level column counts from the BigQuery `tables.jsonl` (rows and sizes show\n> as `N\u002FA` there because that connector does not record them). It reads the run\n> status from `dumper-telemetry.jsonl` when the archive has one. For other\n> layouts (for example PostgreSQL, Oracle, or the Hive `tables.jsonl`) it still\n> validates the archive and reports the connector and host, but object counts\n> are zero and the run status can show `UNKNOWN`. On PostgreSQL 11 and later it\n> shows `FAILURE` because of the harmless `functions.csv` and `aggregates.csv`\n> fallbacks (Section 6, step 1). Then use the task summary printed by the\n> extract command for the run status, and count objects from the connector's\n> own files (Section 7).\n\n## 7. Output Archives\n\nUnless `--output` names a `.zip` object, the extractor names the ZIP\n`dwh-migration-{connector}-metadata.zip` for `metadata extract` connectors and\n`dwh-migration-{connector}.zip` for `query-logs extract` connectors, whose names\nalready end in `-logs`: `snowflake` writes\n`dwh-migration-snowflake-metadata.zip` and `snowflake-logs` writes\n`dwh-migration-snowflake-logs.zip`. A few names differ (`oracle-stats` writes\n`dwh-migration-oracle-stats.zip`; `hiveql`, and query logs extracted with\n`--assessment`, add a `-YYYYMMDDTHHMMSS` timestamp), so list `--output` to find\nthe archive. Every archive contains `compilerworks-metadata.yaml` (extractor\nversion and arguments) plus connector-specific CSVs:\n\n```text\ndwh-migration-snowflake-metadata.zip\n├── databases-au.csv            # Databases and owners\n├── schemata-au.csv             # Schemas\n├── tables-au.csv               # Tables, row counts, byte sizes\n├── columns-au.csv              # Columns, ordinal positions, types, nullability\n├── views-au.csv                # View DDL\n├── functions-au.csv            # User-defined and external functions\n├── external_tables.csv         # External stage tables\n├── snowflake.yaml              # Assessment flag and warehouse count\n├── compilerworks-metadata.yaml # Extractor version and flags\n└── dumper-telemetry.jsonl      # Task timings\n\ndwh-migration-snowflake-logs.zip\n├── query_history_2026-08-05T000000Z.csv   # One file per rotation window\n├── ...\n└── compilerworks-metadata.yaml\n\ndwh-migration-postgresql-metadata.zip\n├── database.csv                # Databases on the server\n├── is_columns_private.csv      # Columns of user tables and views\n├── pg_views_private.csv        # User view DDL\n├── functions-11.csv            # Functions on search_path (11+; drop pg_catalog)\n├── aggregates-11.csv, types.csv, casts.csv\n├── is_columns_generic.csv, pg_views_generic.csv   # System catalog objects\n├── pg_user.csv                 # Roles and privileges (passwords masked)\n├── *.csv.exception.txt         # One per failed task\n└── compilerworks-{metadata.yaml,arguments.txt,version.txt,format.txt}\n```\n\nIn PostgreSQL archives the `*_private.csv` files hold user objects and the\n`*_generic.csv` files hold system objects. `is_columns_private.csv` includes\nview columns; subtract the views in `pg_views_private.csv` to count base tables.\n`functions-11.csv`, `aggregates-11.csv`, and `types.csv` list only objects\nvisible on the session `search_path` (`\"$user\", public` by default), so other\nschemas' functions and types are missing. To include them, replace `--host`,\n`--port`, and `--database` with\n`--url=\"jdbc:postgresql:\u002F\u002F{host}:{port}\u002F{db_name}?currentSchema={schema1},{schema2},public\"`.\n\nArchives never contain the database password: to confirm, check that\n`compilerworks-arguments.txt` and `compilerworks-metadata.yaml` have no\n`password` entry (scanning for the password value is unreliable when it is a\ncommon word), but they do contain object DDL and role lists, and\n`dumper-telemetry.jsonl` is copied from\n`~\u002F.cache\u002FDWH-Dumper\u002Fall-dumper-telemetry.jsonl`, so it can hold connectors,\nhosts, and errors from earlier extractor runs on the same machine. Treat them\nas sensitive and keep them in a private bucket.\n\n## 8. Downstream BigQuery Workflows\n\nEach cloud run writes to its own `--output` prefix (Section 5). To hand the\narchives over together, copy the finished ZIPs into one shared prefix, for\nexample `gcloud storage cp \"gs:\u002F\u002F{bucket}\u002F{run_prefix}\u002F*.zip\"\n\"gs:\u002F\u002F{bucket}\u002Fassessment\u002F\"`, and never use that shared prefix as a cloud\n`--output`. Then:\n\n1.  **BigQuery Migration Assessment**: direct the user to **BigQuery >\n    Migration > Assessment** in the Google Cloud Console and give it the GCS\n    prefix that holds the metadata (extracted with `--assessment`) and logs.\n2.  **Batch SQL translation**: attach the metadata ZIP as schema context in\n    **BigQuery > Migration > SQL translation**, or through the BigQuery\n    Migration API \u002F BQMS MCP tools (see `bigquery-migration-sql-translator`).\n3.  **Table data transfer (out of scope)**: this skill extracts catalog metadata\n    and query logs only. If the user asks to export or load table rows (such as\n    Snowflake `COPY INTO` unloads), ask whether they want to use the\n    [BigQuery Data Transfer Service][bq_dts] (including the\n    [Snowflake transfer connector][snowflake_dts]) or export to Parquet\u002FCSV in\n    Cloud Storage and load with `bq load`.\n\n[bq_dts]: https:\u002F\u002Fdocs.cloud.google.com\u002Fbigquery\u002Fdocs\u002Fdts-introduction.md.txt\n[cloud_locations]: https:\u002F\u002Fcloud.google.com\u002Fabout\u002Flocations\n[extraction_stats_template]: assets\u002Fextraction_stats_template.py\n[gcloud_install]: https:\u002F\u002Fcloud.google.com\u002Fsdk\u002Fdocs\u002Finstall\n[snowflake_dts]: https:\u002F\u002Fdocs.cloud.google.com\u002Fbigquery\u002Fdocs\u002Fmigration\u002Fsnowflake-transfer.md.txt\n",{"name":5,"metadata":24,"description":7},{"version":25,"category":26,"publisher":9},"1.0.0","BigDataAndAnalytics","main","4c6800d20130b5a3233e414609f1241013cec8cc","skills\u002Fcloud\u002Fbigquery-migration-metadata-extractor",2014,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills","https:\u002F\u002Fgithub.com\u002Fgoogle","https:\u002F\u002Fopensource.google\u002F","Google ❤️ Open Source","2026-11-14 23:31:00",{"source":8,"skillName":5,"commands":37,"prompt":47,"zipAvailable":48},[38,41,44],{"tool":39,"command":40},"npx","npx skills add google\u002Fskills --skill bigquery-migration-metadata-extractor",{"tool":42,"command":43},"bunx","bunx skills add google\u002Fskills --skill bigquery-migration-metadata-extractor",{"tool":45,"command":46},"pnpm","pnpm dlx skills add google\u002Fskills --skill bigquery-migration-metadata-extractor","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fbigquery-migration-metadata-extractor\u002FSKILL.md",true,[50,56,62,66,70],{"path":51,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":54,"sourceHtmlUrl":55},"assets\u002Fextraction_stats_template.py","SCRIPT","脚本",50785,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fbigquery-migration-metadata-extractor\u002Fassets\u002Fextraction_stats_template.py",{"path":57,"resourceType":58,"resourceTypeDesc":59,"sizeBytes":60,"sourceHtmlUrl":61},"references\u002Fattribution.md","REFERENCE","参考文档",1963,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fbigquery-migration-metadata-extractor\u002Freferences\u002Fattribution.md",{"path":63,"resourceType":58,"resourceTypeDesc":59,"sizeBytes":64,"sourceHtmlUrl":65},"references\u002Fcloud_extraction.md",19955,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fbigquery-migration-metadata-extractor\u002Freferences\u002Fcloud_extraction.md",{"path":67,"resourceType":58,"resourceTypeDesc":59,"sizeBytes":68,"sourceHtmlUrl":69},"references\u002Fextraction_stats_report.md",9930,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fbigquery-migration-metadata-extractor\u002Freferences\u002Fextraction_stats_report.md",{"path":71,"resourceType":58,"resourceTypeDesc":59,"sizeBytes":72,"sourceHtmlUrl":73},"references\u002Flocal_extraction.md",9096,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fbigquery-migration-metadata-extractor\u002Freferences\u002Flocal_extraction.md",[75,82,89,96,103,110],{"skillNo":76,"name":77,"nameZh":6,"description":78,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":79,"categories":80,"keywords":81,"publishedAt":19},"01M7DQ9K7JF4T7NJZ2K1MQZ44Q","cloud-operator-share-context","Shares Google Cloud operating context the user or their team already have — environments, projects, services and regions, product preferences, best practices, architecture guidance, runbooks — found in agent instruction files, local memory, skills and workflows, and the repository's GCP docs, with the operator agent's memory after the user reviews the list. Use when the user runs this skill, accepts the cloud-operator or cloud-operator-setup skill's offer to share their existing Google Cloud context, or asks to share, import or save that context or a GCP fact for the operator agent. Don't use for ordinary GCP requests, which go to the cloud-operator skill.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fcloud-operator-share-context\u002FSKILL.md",[],[],{"skillNo":83,"name":84,"nameZh":6,"description":85,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":86,"categories":87,"keywords":88,"publishedAt":19},"01M7DQ9ERFGGQCFAWQW0KRKYBY","cloud-operator-setup","Sets up the operator agent in a Google Cloud project via the autocloud MCP server: picks the operator project, creates or checks the user's operator agent, saves the project, and offers to share the user's existing Google Cloud operating context with the operator agent's memory. Use when the user runs this skill, asks to set up the operator or create their operator agent, or chooses setup when the cloud-operator skill asks. Don't use for ordinary GCP requests or installing gcloud.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fcloud-operator-setup\u002FSKILL.md",[],[],{"skillNo":90,"name":91,"nameZh":6,"description":92,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":93,"categories":94,"keywords":95,"publishedAt":19},"01M7B4RHQCM0DK2XQ7S7ZXVN8D","developer-device-platform-device-run","Runs automated, non-interactive jobs on remote Android devices with Developer Device Platform (DDP) Device Run using `gcloud beta device-run sessions`: native Android executables or shell scripts (for example CLI tools, custom runners, test harnesses or benchmarks) and Android instrumentation tests, on one or many devices, then waits for completion and collects per-device outputs and logs. Use when running a binary or test suite on remote devices without an interactive connection. Don't use it for interactive device reservations, adb tunnels or screen streaming (use developer-device-platform-basics), or for local devices.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fdeveloper-device-platform-device-run\u002FSKILL.md",[],[],{"skillNo":97,"name":98,"nameZh":6,"description":99,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":100,"categories":101,"keywords":102,"publishedAt":19},"01M7B4R9YXMRW7RZP9Q9FMZJ2P","compute-vm-creation","Guides the planning, configuration, and provisioning of standalone Google Compute Engine virtual machines (VMs) and bare metal (-metal) instances using gcloud CLI commands or direct MCP tool execution. Use ONLY when the user wants to create, configure, or provision a new standalone Compute Engine VM or bare metal instance from conversational intents (hardware sizing, region selection, OS image, networking, security defaults). DO NOT read or use for Google Kubernetes Engine (GKE) Autopilot or Standard clusters, Kubernetes node pools or Pods, container orchestration, managing existing VMs, Managed Instance Groups (MIGs), or Instance Templates.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fcompute-vm-creation\u002FSKILL.md",[],[],{"skillNo":104,"name":105,"nameZh":6,"description":106,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":107,"categories":108,"keywords":109,"publishedAt":19},"01M7B4QD02N0YBVT0838Z5K5VC","agent-platform-autorater-customization","Customizes an autorater (evaluation metric) for an AI agent so its grades match the user's own judgment, starting from the agent's real trajectories. Use when a user wants a metric tailored to their agent, wants to turn failing trajectories into rubrics, says an existing autorater grades their agent wrong, or wants an autorater to agree with human labels (autorater hillclimbing, judge calibration). Don't use for quick evals with built-in metrics or for improving the agent itself (use agent-platform-eval-flywheel).","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-autorater-customization\u002FSKILL.md",[],[],{"skillNo":111,"name":112,"nameZh":6,"description":113,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":114,"categories":115,"keywords":116,"publishedAt":19},"01M78JP22K73KK96JH0JNDT5C4","iap-agent-platform-troubleshooting","Diagnoses Identity-Aware Proxy (IAP) denials of outbound requests from Gemini Enterprise Agent Platform agents that run on Agent Runtime and send traffic through Agent Gateway. Use when an agent request fails with \"Egress request is not authorized\", when Cloud Audit Logs show an IAP egressViaIAP decision with granted false, when IAP audit entries are missing while debugging agent egress, or when Access policies, IAP allow policies, policy bindings, or Agent Registry entries might block agent egress. Don't use for Gemini model IAM permission errors (such as aiplatform.endpoints.predict), Agent Platform API errors, or Agent Runtime deployment or 500 errors (use agent-platform-troubleshooting instead). Don't use for setting up, creating, or connecting an Agent Gateway, or for IAP web apps, IAP TCP forwarding, or SSH.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fiap-agent-platform-troubleshooting\u002FSKILL.md",[],[]]