[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M78JJJ5CSDTKX3HRH34K37Z8":3,"skill-same:01M78JJJ5CSDTKX3HRH34K37Z8":88},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":27,"commitSha":28,"skillDirectory":29,"forksCount":30,"repositoryHtmlUrl":31,"repositoryHomepageUrl":6,"ownerHtmlUrl":32,"ownerBlogUrl":33,"ownerDescription":34,"updatedAt":35,"install":36,"resources":49},"01M78JJJ5CSDTKX3HRH34K37Z8","bigquery-migration-validator",null,"Validates that tables migrated to BigQuery match their source (Teradata, Snowflake, Oracle, Redshift, Databricks, PostgreSQL, SQL Server, MySQL, MariaDB, Hive, Impala, Vertica) with the BigQuery Migration Service (BQMS) validation tool: schema and data type comparison, column aggregate comparison, and row-level data comparison. Use when generating or editing a validation config, running a validation, choosing sampling or partitioning, fixing connection or driver errors, or investigating validation results in BigQuery and Cloud Storage to decide which mismatches to fix. Don't use for SQL translation (use bigquery-migration-sql-translator), extracting source database catalog metadata or query logs, generating SQL scripts (such as COPY INTO) or exporting or unloading source tables to Parquet on Cloud Storage, loading staged files into BigQuery (such as bq load), or general BigQuery query failures and tuning (use bigquery-troubleshooting or bigquery-optimization).","google\u002Fskills","google","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F1342004?v=4","01M698VYEFDM0VJBZDYYJ2T83S","Google","01M698WR0H76ENHE587T9EJT89",24235,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fbigquery-migration-validator\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: bigquery-migration-validator\nmetadata:\n  version: \"1.0.0\"\n  category: BigDataAndAnalytics\n  publisher: google\ndescription: >-\n  Validates that tables migrated to BigQuery match their source (Teradata,\n  Snowflake, Oracle, Redshift, Databricks, PostgreSQL, SQL Server, MySQL,\n  MariaDB, Hive, Impala, Vertica) with the BigQuery Migration Service (BQMS)\n  validation tool: schema and data type comparison, column aggregate\n  comparison, and row-level data comparison. Use when generating or editing a\n  validation config, running a validation, choosing sampling or partitioning,\n  fixing connection or driver errors, or investigating validation results in\n  BigQuery and Cloud Storage to decide which mismatches to fix. Don't use for\n  SQL translation (use bigquery-migration-sql-translator), extracting source\n  database catalog metadata or query logs, generating SQL scripts (such as COPY\n  INTO) or exporting or unloading source tables to Parquet on Cloud Storage,\n  loading staged files into BigQuery (such as bq load), or general BigQuery\n  query failures and tuning (use bigquery-troubleshooting or\n  bigquery-optimization).\n---\n\n# BigQuery Migration Validator\n\nThe validation tool compares a source table with its migrated BigQuery table. It\nruns in two halves:\n\n1.  **Client (this machine)**: Reads the config, queries the source database,\n    and stages a row sample, source aggregates, and run metadata in Cloud\n    Storage. It then starts a BigQuery Migration workflow for each table and\n    exits.\n2.  **Backend (Google Cloud, asynchronous)**: Loads the staged sample into\n    BigQuery, compares it with the target table, and writes the results to\n    BigQuery and Cloud Storage.\n\nResults do not exist when the command exits. Wait for the workflow to finish\nbefore reading them (Section 4).\n\n## Usage attribution\n\n**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you\nMUST read and follow the instructions in\n[Usage Attribution](references\u002Fattribution.md).\n\n## Before you begin\n\n1.  **Google Cloud CLI and project**: Ensure the\n    [Google Cloud CLI][gcloud_install] is installed, verify that the Google\n    Cloud project has an active Cloud Billing account, and configure the target\n    project (check regional service availability at\n    [Google Cloud locations][cloud_locations] when choosing `--location` for\n    validation workflows):\n\n    ```bash\n    gcloud config set project PROJECT_ID\n    ```\n\n2.  **Enable the APIs** (ask the user before enabling APIs): BigQuery\n    (`bigquery.googleapis.com`), BigQuery Migration\n    (`bigquerymigration.googleapis.com`), Cloud Storage\n    (`storage.googleapis.com`), and Secret Manager\n    (`secretmanager.googleapis.com`, when reading source passwords from Secret\n    Manager):\n\n    ```bash\n    gcloud services enable \\\n        bigquery.googleapis.com \\\n        bigquerymigration.googleapis.com \\\n        storage.googleapis.com\n    # Only when source passwords come from Secret Manager:\n    gcloud services enable secretmanager.googleapis.com\n    ```\n\n3.  **Local runtime, authentication, and IAM permissions** (ask before\n    installing software):\n\n    *   **Runtime and drivers**: Java 11 or later (JRE 11+) first on `PATH`, the\n        `bqms` gcloud component (installed on first use or with\n        `gcloud components install bqms`), local network access to the source\n        database, and vendor JDBC driver JARs for Teradata (`terajdbc4.jar`),\n        Vertica (`vertica-jdbc.jar`), Impala, and DuckDB in `.\u002Fdrivers` or\n        `BQMS_DRIVER_DIR` when required (other JDBC drivers download on first\n        use; see [connections and drivers](references\u002Fconnections.md)).\n    *   **Google Cloud credentials and IAM**: Authenticate the Google Cloud CLI\n        and Application Default Credentials with `gcloud auth login` and\n        `gcloud auth application-default login` (or set\n        `GOOGLE_APPLICATION_CREDENTIALS` for a service account). The identity\n        needs write access to the staging Cloud Storage bucket\n        (`roles\u002Fstorage.objectAdmin`), permission to create BigQuery Migration\n        workflows (`roles\u002Fbigquerymigration.editor`), and permission to run\n        BigQuery jobs and write validation results in the target project and,\n        if set, the `results.projectId` project (`roles\u002Fbigquery.dataEditor`\n        and `roles\u002Fbigquery.jobUser`). With Secret Manager passwords, it also\n        needs `roles\u002Fsecretmanager.secretAccessor` on the secret, or\n        `roles\u002Fiam.serviceAccountTokenCreator` on the service account named in\n        `secretImpersonateServiceAccount` (which then needs the accessor role).\n\n4.  **Cost and cleanup**: Review\n    [Google Cloud Pricing](https:\u002F\u002Fcloud.google.com\u002Fpricing) for BigQuery, Cloud\n    Storage, and Secret Manager. Staged samples in Cloud Storage contain source\n    rows. When validation is complete, ask the user before deleting the staged\n    run directory (a `gcs_paths` entry in `run_state.json`) and any per-run\n    tables they no longer need (list them as in the\n    [results reference](references\u002Fresults.md), Section 5). Deleting the run\n    directory also deletes the backend reports stored there (`results.json`,\n    `validation_summary.txt`, `column_details.txt`, `queries.sql`) and\n    `sample_metadata.json`, so `run_history.gcs_uri` no longer resolves;\n    say so and offer to copy the reports locally first:\n\n    ```bash\n    gcloud storage cp GCS_RUN_PATH\u002Fresults.json \\\n        GCS_RUN_PATH\u002Fvalidation_summary.txt GCS_RUN_PATH\u002Fcolumn_details.txt \\\n        GCS_RUN_PATH\u002Fqueries.sql LOCAL_DIR\u002F\n    gcloud storage rm --recursive GCS_RUN_PATH\n    bq rm -f -t RESULTS_PROJECT:DATASET.TABLE_NAME\n    ```\n\n## 1. What gets compared\n\nPillar        | `--validation-types` | Compares                                                                                                                    | Coverage\n:------------ | :------------------- | :-------------------------------------------------------------------------------------------------------------------------- | :-------\n**Schema**    | `schema`             | Column presence, mapping, and data type compatibility (one of 9 compatibility notes)                                        | All columns\n**Aggregate** | `aggregate`          | Column statistics per type (`COUNT`, `SUM`, `AVG`, `MIN`, `MAX`, `SUM_LENGTH`, ...), computed per partition, then rolled up | 100% of rows (pushed down)\n**Data**      | `data`               | Cell-by-cell values joined on comparison keys, with a distance per mismatched cell                                          | Sampled or full rows\n\nWith `--validation-types` omitted, all three run. With `--validation-types` set\n(for example `--validation-types=schema,aggregate`), only the listed types run.\nQuery (code) validation is not available in this tool.\n\nEach pillar catches what the others miss. Aggregates cover every row cheaply but\ncannot see offsetting errors (one row +5, another -5). Row sampling sees those\nbut only in the sample. Run both for sign-off.\n\n## 2. Quick start\n\nBoth validation commands run in the **alpha** release track through the Google\nCloud CLI, which calls the `bqms` component (Java 11+ required). Arguments after\n`--` are forwarded unchanged to the validation client.\n\n> [!IMPORTANT]\n> The commands are in the alpha track. If gcloud reports an invalid choice or\n> unknown command, ask the user before running `gcloud components update` (or\n> `gcloud components install alpha bqms`). If the Google Cloud CLI was\n> installed with a system package manager such as `apt` or `yum`, the\n> component manager is disabled; have the user upgrade `google-cloud-cli`\n> through that package manager instead. If the command is still unavailable,\n> stop and point the user to the [CLI reference](references\u002Fcli.md) and the\n> [installation guide][gcloud_install] rather than retrying.\n\n```bash\n# 1. Generate per-table configs offline from extractor metadata or BQMS\n#    translation output (writes .\u002Fvalidation_config\u002F).\ngcloud alpha bq migration validation configure \\\n    --metadata=dwh-migration-oracle-metadata.zip \\\n    --target-dataset=warehouse \\\n    --output=validation_config\n\n# 2. Edit validation_config\u002Fconnections.yaml (check the source dialect;\n#    set the target dataset) and settings.yaml (googleCloudStorage.gcsPath),\n#    then run schema and aggregate checks first.\ngcloud alpha bq migration validation run validation_config\u002F \\\n    --location=us --validation-types=schema,aggregate\n\n# 3. Run row-level comparison, optionally on a subset of tables.\ngcloud alpha bq migration validation run validation_config\u002F \\\n    --location=us --validation-types=data --tables=orders\n```\n\nThen wait for the workflows (Section 4) and investigate (Section 5).\n\nEvery answer that runs or recommends `validation run` must tell the user that\nthe comparison finishes later in a BigQuery Migration workflow, that exit code 0\ndoes not mean the data matched, and how to check the workflow before reading\nresults. Users who skip this read an empty or stale results dataset as a pass.\n\n## 3. Configure\n\nSettings come from YAML (modular `validation_config\u002F` directory or one\n`validation-config.yaml`), then CLI flags. The essentials:\n\n*   `connections`: a `source` and a BigQuery `target`. The target needs\n    `projectId` and `dataset`; without `dataset` the run fails, even when every\n    `targetTable.name` is fully qualified. `configure` can scaffold the wrong\n    source `dialect` (for example `snowflake` for PostgreSQL metadata), so\n    check it and its properties before the first run. Connection names are\n    letters and digits only; a name with `_` or `-` fails config validation. If\n    the user asks for such a name, drop the separators (for example `sales-db`\n    becomes `salesdb`) and tell them why, because the name also forms the\n    environment variable prefix. Keep secrets out of YAML: use\n    `BQMS_VALIDATION_\u003CCONNECTION>_PASSWORD` or a Secret Manager reference (see\n    [connections and drivers](references\u002Fconnections.md)). Never ask the user\n    to type a password into the conversation and never echo one. For commands\n    you run yourself, use a Secret Manager reference (`passwordSecret` or\n    `BQMS_VALIDATION_\u003CCONNECTION>_PASSWORD_SECRET`); give plaintext `export`\n    lines only for the user to run in their own terminal.\n*   `googleCloudStorage.gcsPath`: required staging bucket (`gs:\u002F\u002Fbucket\u002Fpath`,\n    no trailing slash). Results artifacts also land here.\n*   Per table: `sourceTable` and `targetTable` (each with `name` and, under\n    `sourceTable`, `comparisonKeys`: the columns that uniquely identify a row\n    for the validation join when the table has no primary key — explain that\n    `comparisonKeys` only control how validation matches rows, so no database\n    primary key or schema change is needed; `configure` fills them in only\n    when the metadata includes primary keys), `columnSettings` (exclusions,\n    transforms, tolerances), `sampleSize`, and `partitioning`.\n*   `results.dataset` and optional `results.projectId`: BigQuery dataset and\n    project for results (defaults `validation_results` and the target\n    connection's project). Query results where these point.\n\nDetails: [configuration reference](references\u002Fconfiguration.md).\n\n## 4. Run and wait\n\n1.  Run `gcloud alpha bq migration validation run`. Exit code 1 means a config\n    or source-side failure; read the error and fix it. Exit code 0 means the\n    samples were staged (or that usage was printed for a bad forwarded flag;\n    check the output). Mismatches never change the exit code.\n2.  Confirm a workflow started. The client prints a workflow name per table and\n    writes `run_state.json` (run ID, Cloud Storage paths, workflow names,\n    results dataset) to `--output-dir` (default `\u002Ftmp\u002Fvalidation_outputs`,\n    passed after `--`). If it instead prints \"Could not initiate BigQuery\n    Migration validation workflow\", fix the cause it reports (usually\n    permissions or the API not enabled) and re-run, or create the workflow from\n    the console link it prints.\n3.  Poll until the workflow finishes:\n\n    ```bash\n    gcloud bq migration-workflows describe WORKFLOW_NAME\n    ```\n\n    Or check `run_history.status` in the results dataset; `IN_PROGRESS` means\n    the backend is still working.\n\n## 5. Investigate results\n\nResults are in three places. Pick by question, not by habit:\n\nSurface                                        | Audience       | Grain                                          | Use for\n:--------------------------------------------- | :------------- | :--------------------------------------------- | :------\nBigQuery results dataset                       | Agents, humans | Run, slice (partition), column, aggregate, row | All investigation; the only source for per-partition and per-row detail\nCloud Storage `results.json`                   | Agents         | One run (whole table)                          | Quick status of one table without BigQuery access; execution times\n`validation_summary.txt`, `column_details.txt` | Humans         | One run                                        | Show the user; do not parse\nWeb UI                                         | Humans         | All of the above, visual                       | Point the user to it if the printed link is a real URL\n\nInvestigate by zooming in one level at a time, stopping as soon as the cause is\nclear:\n\n1.  **All tables**: Latest run per table with status, row deltas, and mismatch\n    counts. Errors first, then the largest mismatch counts.\n2.  **One table, by pillar**: Schema compatibility, then failed aggregates, then\n    missing rows, then per-column mismatch statistics.\n3.  **Partitions**: Per-slice metrics show whether mismatches are local (one day\n    or range: a failed load) or everywhere (a systematic conversion).\n4.  **Rows**: Group mismatched cells by column and distance, then read a few\n    rows to confirm the pattern.\n5.  **Fix and re-run narrowly**: Pass `--tables` and `-- --where`, then compare\n    the new run with the old one.\n\nThen zoom back out: say which tables pass, which have explained differences, and\nwhich need a fix in the migration itself. A narrower re-run (some\n`--validation-types`, or `--where`) becomes the table's latest run without\ncovering the rest; a table passes only when every pillar and the whole table\npassed.\n\nQueries, field names, and decision rules for each level:\n[investigation workflow](references\u002Finvestigation.md). What every status, field,\nand table means: [results reference](references\u002Fresults.md).\n\n## 6. Reference guides\n\nGuide                                                         | Read when\n:------------------------------------------------------------ | :--------\n[CLI reference](references\u002Fcli.md)                            | Installing, running, flags, exit codes, workflow status\n[Configuration reference](references\u002Fconfiguration.md)        | Writing or auditing YAML, keys, transforms, tolerances\n[Connections and drivers](references\u002Fconnections.md)          | Credentials, Secret Manager, JDBC drivers, JDBC URLs\n[Sampling and partitioning](references\u002Fperformance_tuning.md) | Sample sizes, partitioned runs, wide tables, concurrency\n[Results reference](references\u002Fresults.md)                    | Statuses, `results.json`, BigQuery tables and views, distances\n[Investigation workflow](references\u002Finvestigation.md)         | Step-by-step drill-down queries and decision rules\n[Dialects and types](references\u002Fdialect_mapping.md)           | Default aggregates, type compatibility, dialect quirks\n[Troubleshooting](references\u002Ftroubleshooting.md)              | Mismatch signatures and their config fixes, run errors\n\n## 7. Operating principles\n\n1.  **Read-only**: Validation only runs `SELECT` on the source and target. Never\n    modify source or target data to make a validation pass.\n2.  **Mismatches are findings, not failures**: A run that finishes with\n    mismatches (`COMPLETED_WITH_MISMATCHES`) succeeded and found differences.\n    Reserve \"failed\" for `ERROR` runs and non-zero exit codes.\n3.  **Explain before excluding**: Do not add `exclude`, tolerances, or\n    transforms to hide a difference until the row-level evidence shows it is a\n    representation difference (padding, precision, empty string versus NULL) and\n    not lost or wrong data. Say which one it is.\n4.  **Protect row data**: Mismatch tables hold real customer values. Query\n    aggregates and small samples (`LIMIT`), and do not paste bulk row values\n    into the conversation unless the user asks.\n5.  **Edit the user's config in place**: When the user names a config file, read\n    it, change only what the task needs, and keep everything else as written:\n    table names (including dataset or schema prefixes), `comparisonKeys`,\n    filters, and settings. Do not normalize or shorten existing names. A changed\n    `targetTable.name` points the comparison at a different table, and a dropped\n    `comparisonKeys` changes what counts as a missing row, so either silently\n    changes what is validated. If something existing looks wrong, say so instead\n    of changing it.\n\n[cloud_locations]: https:\u002F\u002Fcloud.google.com\u002Fabout\u002Flocations\n[gcloud_install]: https:\u002F\u002Fcloud.google.com\u002Fsdk\u002Fdocs\u002Finstall\n","# BigQuery Migration Validator\n\nThe validation tool compares a source table with its migrated BigQuery table. It\nruns in two halves:\n\n1.  **Client (this machine)**: Reads the config, queries the source database,\n    and stages a row sample, source aggregates, and run metadata in Cloud\n    Storage. It then starts a BigQuery Migration workflow for each table and\n    exits.\n2.  **Backend (Google Cloud, asynchronous)**: Loads the staged sample into\n    BigQuery, compares it with the target table, and writes the results to\n    BigQuery and Cloud Storage.\n\nResults do not exist when the command exits. Wait for the workflow to finish\nbefore reading them (Section 4).\n\n## Usage attribution\n\n**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you\nMUST read and follow the instructions in\n[Usage Attribution](references\u002Fattribution.md).\n\n## Before you begin\n\n1.  **Google Cloud CLI and project**: Ensure the\n    [Google Cloud CLI][gcloud_install] is installed, verify that the Google\n    Cloud project has an active Cloud Billing account, and configure the target\n    project (check regional service availability at\n    [Google Cloud locations][cloud_locations] when choosing `--location` for\n    validation workflows):\n\n    ```bash\n    gcloud config set project PROJECT_ID\n    ```\n\n2.  **Enable the APIs** (ask the user before enabling APIs): BigQuery\n    (`bigquery.googleapis.com`), BigQuery Migration\n    (`bigquerymigration.googleapis.com`), Cloud Storage\n    (`storage.googleapis.com`), and Secret Manager\n    (`secretmanager.googleapis.com`, when reading source passwords from Secret\n    Manager):\n\n    ```bash\n    gcloud services enable \\\n        bigquery.googleapis.com \\\n        bigquerymigration.googleapis.com \\\n        storage.googleapis.com\n    # Only when source passwords come from Secret Manager:\n    gcloud services enable secretmanager.googleapis.com\n    ```\n\n3.  **Local runtime, authentication, and IAM permissions** (ask before\n    installing software):\n\n    *   **Runtime and drivers**: Java 11 or later (JRE 11+) first on `PATH`, the\n        `bqms` gcloud component (installed on first use or with\n        `gcloud components install bqms`), local network access to the source\n        database, and vendor JDBC driver JARs for Teradata (`terajdbc4.jar`),\n        Vertica (`vertica-jdbc.jar`), Impala, and DuckDB in `.\u002Fdrivers` or\n        `BQMS_DRIVER_DIR` when required (other JDBC drivers download on first\n        use; see [connections and drivers](references\u002Fconnections.md)).\n    *   **Google Cloud credentials and IAM**: Authenticate the Google Cloud CLI\n        and Application Default Credentials with `gcloud auth login` and\n        `gcloud auth application-default login` (or set\n        `GOOGLE_APPLICATION_CREDENTIALS` for a service account). The identity\n        needs write access to the staging Cloud Storage bucket\n        (`roles\u002Fstorage.objectAdmin`), permission to create BigQuery Migration\n        workflows (`roles\u002Fbigquerymigration.editor`), and permission to run\n        BigQuery jobs and write validation results in the target project and,\n        if set, the `results.projectId` project (`roles\u002Fbigquery.dataEditor`\n        and `roles\u002Fbigquery.jobUser`). With Secret Manager passwords, it also\n        needs `roles\u002Fsecretmanager.secretAccessor` on the secret, or\n        `roles\u002Fiam.serviceAccountTokenCreator` on the service account named in\n        `secretImpersonateServiceAccount` (which then needs the accessor role).\n\n4.  **Cost and cleanup**: Review\n    [Google Cloud Pricing](https:\u002F\u002Fcloud.google.com\u002Fpricing) for BigQuery, Cloud\n    Storage, and Secret Manager. Staged samples in Cloud Storage contain source\n    rows. When validation is complete, ask the user before deleting the staged\n    run directory (a `gcs_paths` entry in `run_state.json`) and any per-run\n    tables they no longer need (list them as in the\n    [results reference](references\u002Fresults.md), Section 5). Deleting the run\n    directory also deletes the backend reports stored there (`results.json`,\n    `validation_summary.txt`, `column_details.txt`, `queries.sql`) and\n    `sample_metadata.json`, so `run_history.gcs_uri` no longer resolves;\n    say so and offer to copy the reports locally first:\n\n    ```bash\n    gcloud storage cp GCS_RUN_PATH\u002Fresults.json \\\n        GCS_RUN_PATH\u002Fvalidation_summary.txt GCS_RUN_PATH\u002Fcolumn_details.txt \\\n        GCS_RUN_PATH\u002Fqueries.sql LOCAL_DIR\u002F\n    gcloud storage rm --recursive GCS_RUN_PATH\n    bq rm -f -t RESULTS_PROJECT:DATASET.TABLE_NAME\n    ```\n\n## 1. What gets compared\n\nPillar        | `--validation-types` | Compares                                                                                                                    | Coverage\n:------------ | :------------------- | :-------------------------------------------------------------------------------------------------------------------------- | :-------\n**Schema**    | `schema`             | Column presence, mapping, and data type compatibility (one of 9 compatibility notes)                                        | All columns\n**Aggregate** | `aggregate`          | Column statistics per type (`COUNT`, `SUM`, `AVG`, `MIN`, `MAX`, `SUM_LENGTH`, ...), computed per partition, then rolled up | 100% of rows (pushed down)\n**Data**      | `data`               | Cell-by-cell values joined on comparison keys, with a distance per mismatched cell                                          | Sampled or full rows\n\nWith `--validation-types` omitted, all three run. With `--validation-types` set\n(for example `--validation-types=schema,aggregate`), only the listed types run.\nQuery (code) validation is not available in this tool.\n\nEach pillar catches what the others miss. Aggregates cover every row cheaply but\ncannot see offsetting errors (one row +5, another -5). Row sampling sees those\nbut only in the sample. Run both for sign-off.\n\n## 2. Quick start\n\nBoth validation commands run in the **alpha** release track through the Google\nCloud CLI, which calls the `bqms` component (Java 11+ required). Arguments after\n`--` are forwarded unchanged to the validation client.\n\n> [!IMPORTANT]\n> The commands are in the alpha track. If gcloud reports an invalid choice or\n> unknown command, ask the user before running `gcloud components update` (or\n> `gcloud components install alpha bqms`). If the Google Cloud CLI was\n> installed with a system package manager such as `apt` or `yum`, the\n> component manager is disabled; have the user upgrade `google-cloud-cli`\n> through that package manager instead. If the command is still unavailable,\n> stop and point the user to the [CLI reference](references\u002Fcli.md) and the\n> [installation guide][gcloud_install] rather than retrying.\n\n```bash\n# 1. Generate per-table configs offline from extractor metadata or BQMS\n#    translation output (writes .\u002Fvalidation_config\u002F).\ngcloud alpha bq migration validation configure \\\n    --metadata=dwh-migration-oracle-metadata.zip \\\n    --target-dataset=warehouse \\\n    --output=validation_config\n\n# 2. Edit validation_config\u002Fconnections.yaml (check the source dialect;\n#    set the target dataset) and settings.yaml (googleCloudStorage.gcsPath),\n#    then run schema and aggregate checks first.\ngcloud alpha bq migration validation run validation_config\u002F \\\n    --location=us --validation-types=schema,aggregate\n\n# 3. Run row-level comparison, optionally on a subset of tables.\ngcloud alpha bq migration validation run validation_config\u002F \\\n    --location=us --validation-types=data --tables=orders\n```\n\nThen wait for the workflows (Section 4) and investigate (Section 5).\n\nEvery answer that runs or recommends `validation run` must tell the user that\nthe comparison finishes later in a BigQuery Migration workflow, that exit code 0\ndoes not mean the data matched, and how to check the workflow before reading\nresults. Users who skip this read an empty or stale results dataset as a pass.\n\n## 3. Configure\n\nSettings come from YAML (modular `validation_config\u002F` directory or one\n`validation-config.yaml`), then CLI flags. The essentials:\n\n*   `connections`: a `source` and a BigQuery `target`. The target needs\n    `projectId` and `dataset`; without `dataset` the run fails, even when every\n    `targetTable.name` is fully qualified. `configure` can scaffold the wrong\n    source `dialect` (for example `snowflake` for PostgreSQL metadata), so\n    check it and its properties before the first run. Connection names are\n    letters and digits only; a name with `_` or `-` fails config validation. If\n    the user asks for such a name, drop the separators (for example `sales-db`\n    becomes `salesdb`) and tell them why, because the name also forms the\n    environment variable prefix. Keep secrets out of YAML: use\n    `BQMS_VALIDATION_\u003CCONNECTION>_PASSWORD` or a Secret Manager reference (see\n    [connections and drivers](references\u002Fconnections.md)). Never ask the user\n    to type a password into the conversation and never echo one. For commands\n    you run yourself, use a Secret Manager reference (`passwordSecret` or\n    `BQMS_VALIDATION_\u003CCONNECTION>_PASSWORD_SECRET`); give plaintext `export`\n    lines only for the user to run in their own terminal.\n*   `googleCloudStorage.gcsPath`: required staging bucket (`gs:\u002F\u002Fbucket\u002Fpath`,\n    no trailing slash). Results artifacts also land here.\n*   Per table: `sourceTable` and `targetTable` (each with `name` and, under\n    `sourceTable`, `comparisonKeys`: the columns that uniquely identify a row\n    for the validation join when the table has no primary key — explain that\n    `comparisonKeys` only control how validation matches rows, so no database\n    primary key or schema change is needed; `configure` fills them in only\n    when the metadata includes primary keys), `columnSettings` (exclusions,\n    transforms, tolerances), `sampleSize`, and `partitioning`.\n*   `results.dataset` and optional `results.projectId`: BigQuery dataset and\n    project for results (defaults `validation_results` and the target\n    connection's project). Query results where these point.\n\nDetails: [configuration reference](references\u002Fconfiguration.md).\n\n## 4. Run and wait\n\n1.  Run `gcloud alpha bq migration validation run`. Exit code 1 means a config\n    or source-side failure; read the error and fix it. Exit code 0 means the\n    samples were staged (or that usage was printed for a bad forwarded flag;\n    check the output). Mismatches never change the exit code.\n2.  Confirm a workflow started. The client prints a workflow name per table and\n    writes `run_state.json` (run ID, Cloud Storage paths, workflow names,\n    results dataset) to `--output-dir` (default `\u002Ftmp\u002Fvalidation_outputs`,\n    passed after `--`). If it instead prints \"Could not initiate BigQuery\n    Migration validation workflow\", fix the cause it reports (usually\n    permissions or the API not enabled) and re-run, or create the workflow from\n    the console link it prints.\n3.  Poll until the workflow finishes:\n\n    ```bash\n    gcloud bq migration-workflows describe WORKFLOW_NAME\n    ```\n\n    Or check `run_history.status` in the results dataset; `IN_PROGRESS` means\n    the backend is still working.\n\n## 5. Investigate results\n\nResults are in three places. Pick by question, not by habit:\n\nSurface                                        | Audience       | Grain                                          | Use for\n:--------------------------------------------- | :------------- | :--------------------------------------------- | :------\nBigQuery results dataset                       | Agents, humans | Run, slice (partition), column, aggregate, row | All investigation; the only source for per-partition and per-row detail\nCloud Storage `results.json`                   | Agents         | One run (whole table)                          | Quick status of one table without BigQuery access; execution times\n`validation_summary.txt`, `column_details.txt` | Humans         | One run                                        | Show the user; do not parse\nWeb UI                                         | Humans         | All of the above, visual                       | Point the user to it if the printed link is a real URL\n\nInvestigate by zooming in one level at a time, stopping as soon as the cause is\nclear:\n\n1.  **All tables**: Latest run per table with status, row deltas, and mismatch\n    counts. Errors first, then the largest mismatch counts.\n2.  **One table, by pillar**: Schema compatibility, then failed aggregates, then\n    missing rows, then per-column mismatch statistics.\n3.  **Partitions**: Per-slice metrics show whether mismatches are local (one day\n    or range: a failed load) or everywhere (a systematic conversion).\n4.  **Rows**: Group mismatched cells by column and distance, then read a few\n    rows to confirm the pattern.\n5.  **Fix and re-run narrowly**: Pass `--tables` and `-- --where`, then compare\n    the new run with the old one.\n\nThen zoom back out: say which tables pass, which have explained differences, and\nwhich need a fix in the migration itself. A narrower re-run (some\n`--validation-types`, or `--where`) becomes the table's latest run without\ncovering the rest; a table passes only when every pillar and the whole table\npassed.\n\nQueries, field names, and decision rules for each level:\n[investigation workflow](references\u002Finvestigation.md). What every status, field,\nand table means: [results reference](references\u002Fresults.md).\n\n## 6. Reference guides\n\nGuide                                                         | Read when\n:------------------------------------------------------------ | :--------\n[CLI reference](references\u002Fcli.md)                            | Installing, running, flags, exit codes, workflow status\n[Configuration reference](references\u002Fconfiguration.md)        | Writing or auditing YAML, keys, transforms, tolerances\n[Connections and drivers](references\u002Fconnections.md)          | Credentials, Secret Manager, JDBC drivers, JDBC URLs\n[Sampling and partitioning](references\u002Fperformance_tuning.md) | Sample sizes, partitioned runs, wide tables, concurrency\n[Results reference](references\u002Fresults.md)                    | Statuses, `results.json`, BigQuery tables and views, distances\n[Investigation workflow](references\u002Finvestigation.md)         | Step-by-step drill-down queries and decision rules\n[Dialects and types](references\u002Fdialect_mapping.md)           | Default aggregates, type compatibility, dialect quirks\n[Troubleshooting](references\u002Ftroubleshooting.md)              | Mismatch signatures and their config fixes, run errors\n\n## 7. Operating principles\n\n1.  **Read-only**: Validation only runs `SELECT` on the source and target. Never\n    modify source or target data to make a validation pass.\n2.  **Mismatches are findings, not failures**: A run that finishes with\n    mismatches (`COMPLETED_WITH_MISMATCHES`) succeeded and found differences.\n    Reserve \"failed\" for `ERROR` runs and non-zero exit codes.\n3.  **Explain before excluding**: Do not add `exclude`, tolerances, or\n    transforms to hide a difference until the row-level evidence shows it is a\n    representation difference (padding, precision, empty string versus NULL) and\n    not lost or wrong data. Say which one it is.\n4.  **Protect row data**: Mismatch tables hold real customer values. Query\n    aggregates and small samples (`LIMIT`), and do not paste bulk row values\n    into the conversation unless the user asks.\n5.  **Edit the user's config in place**: When the user names a config file, read\n    it, change only what the task needs, and keep everything else as written:\n    table names (including dataset or schema prefixes), `comparisonKeys`,\n    filters, and settings. Do not normalize or shorten existing names. A changed\n    `targetTable.name` points the comparison at a different table, and a dropped\n    `comparisonKeys` changes what counts as a missing row, so either silently\n    changes what is validated. If something existing looks wrong, say so instead\n    of changing it.\n\n[cloud_locations]: https:\u002F\u002Fcloud.google.com\u002Fabout\u002Flocations\n[gcloud_install]: https:\u002F\u002Fcloud.google.com\u002Fsdk\u002Fdocs\u002Finstall\n",{"name":5,"metadata":24,"description":7},{"version":25,"category":26,"publisher":9},"1.0.0","BigDataAndAnalytics","main","4c6800d20130b5a3233e414609f1241013cec8cc","skills\u002Fcloud\u002Fbigquery-migration-validator",2014,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills","https:\u002F\u002Fgithub.com\u002Fgoogle","https:\u002F\u002Fopensource.google\u002F","Google ❤️ Open Source","2026-11-14 23:31:00",{"source":8,"skillName":5,"commands":37,"prompt":47,"zipAvailable":48},[38,41,44],{"tool":39,"command":40},"npx","npx skills add google\u002Fskills --skill bigquery-migration-validator",{"tool":42,"command":43},"bunx","bunx skills add google\u002Fskills --skill bigquery-migration-validator",{"tool":45,"command":46},"pnpm","pnpm dlx skills add google\u002Fskills --skill bigquery-migration-validator","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fbigquery-migration-validator\u002FSKILL.md",true,[50,56,60,64,68,72,76,80,84],{"path":51,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":54,"sourceHtmlUrl":55},"references\u002Fattribution.md","REFERENCE","参考文档",1936,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fbigquery-migration-validator\u002Freferences\u002Fattribution.md",{"path":57,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":58,"sourceHtmlUrl":59},"references\u002Fcli.md",11597,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fbigquery-migration-validator\u002Freferences\u002Fcli.md",{"path":61,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":62,"sourceHtmlUrl":63},"references\u002Fconfiguration.md",10197,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fbigquery-migration-validator\u002Freferences\u002Fconfiguration.md",{"path":65,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":66,"sourceHtmlUrl":67},"references\u002Fconnections.md",5454,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fbigquery-migration-validator\u002Freferences\u002Fconnections.md",{"path":69,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":70,"sourceHtmlUrl":71},"references\u002Fdialect_mapping.md",3508,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fbigquery-migration-validator\u002Freferences\u002Fdialect_mapping.md",{"path":73,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":74,"sourceHtmlUrl":75},"references\u002Finvestigation.md",13998,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fbigquery-migration-validator\u002Freferences\u002Finvestigation.md",{"path":77,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":78,"sourceHtmlUrl":79},"references\u002Fperformance_tuning.md",5714,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fbigquery-migration-validator\u002Freferences\u002Fperformance_tuning.md",{"path":81,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":82,"sourceHtmlUrl":83},"references\u002Fresults.md",17243,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fbigquery-migration-validator\u002Freferences\u002Fresults.md",{"path":85,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":86,"sourceHtmlUrl":87},"references\u002Ftroubleshooting.md",5829,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fbigquery-migration-validator\u002Freferences\u002Ftroubleshooting.md",[89,96,103,110,117,124],{"skillNo":90,"name":91,"nameZh":6,"description":92,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":93,"categories":94,"keywords":95,"publishedAt":19},"01M7DQ9K7JF4T7NJZ2K1MQZ44Q","cloud-operator-share-context","Shares Google Cloud operating context the user or their team already have — environments, projects, services and regions, product preferences, best practices, architecture guidance, runbooks — found in agent instruction files, local memory, skills and workflows, and the repository's GCP docs, with the operator agent's memory after the user reviews the list. Use when the user runs this skill, accepts the cloud-operator or cloud-operator-setup skill's offer to share their existing Google Cloud context, or asks to share, import or save that context or a GCP fact for the operator agent. Don't use for ordinary GCP requests, which go to the cloud-operator skill.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fcloud-operator-share-context\u002FSKILL.md",[],[],{"skillNo":97,"name":98,"nameZh":6,"description":99,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":100,"categories":101,"keywords":102,"publishedAt":19},"01M7DQ9ERFGGQCFAWQW0KRKYBY","cloud-operator-setup","Sets up the operator agent in a Google Cloud project via the autocloud MCP server: picks the operator project, creates or checks the user's operator agent, saves the project, and offers to share the user's existing Google Cloud operating context with the operator agent's memory. Use when the user runs this skill, asks to set up the operator or create their operator agent, or chooses setup when the cloud-operator skill asks. Don't use for ordinary GCP requests or installing gcloud.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fcloud-operator-setup\u002FSKILL.md",[],[],{"skillNo":104,"name":105,"nameZh":6,"description":106,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":107,"categories":108,"keywords":109,"publishedAt":19},"01M7B4RHQCM0DK2XQ7S7ZXVN8D","developer-device-platform-device-run","Runs automated, non-interactive jobs on remote Android devices with Developer Device Platform (DDP) Device Run using `gcloud beta device-run sessions`: native Android executables or shell scripts (for example CLI tools, custom runners, test harnesses or benchmarks) and Android instrumentation tests, on one or many devices, then waits for completion and collects per-device outputs and logs. Use when running a binary or test suite on remote devices without an interactive connection. Don't use it for interactive device reservations, adb tunnels or screen streaming (use developer-device-platform-basics), or for local devices.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fdeveloper-device-platform-device-run\u002FSKILL.md",[],[],{"skillNo":111,"name":112,"nameZh":6,"description":113,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":114,"categories":115,"keywords":116,"publishedAt":19},"01M7B4R9YXMRW7RZP9Q9FMZJ2P","compute-vm-creation","Guides the planning, configuration, and provisioning of standalone Google Compute Engine virtual machines (VMs) and bare metal (-metal) instances using gcloud CLI commands or direct MCP tool execution. Use ONLY when the user wants to create, configure, or provision a new standalone Compute Engine VM or bare metal instance from conversational intents (hardware sizing, region selection, OS image, networking, security defaults). DO NOT read or use for Google Kubernetes Engine (GKE) Autopilot or Standard clusters, Kubernetes node pools or Pods, container orchestration, managing existing VMs, Managed Instance Groups (MIGs), or Instance Templates.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fcompute-vm-creation\u002FSKILL.md",[],[],{"skillNo":118,"name":119,"nameZh":6,"description":120,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":121,"categories":122,"keywords":123,"publishedAt":19},"01M7B4QD02N0YBVT0838Z5K5VC","agent-platform-autorater-customization","Customizes an autorater (evaluation metric) for an AI agent so its grades match the user's own judgment, starting from the agent's real trajectories. Use when a user wants a metric tailored to their agent, wants to turn failing trajectories into rubrics, says an existing autorater grades their agent wrong, or wants an autorater to agree with human labels (autorater hillclimbing, judge calibration). Don't use for quick evals with built-in metrics or for improving the agent itself (use agent-platform-eval-flywheel).","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fagent-platform-autorater-customization\u002FSKILL.md",[],[],{"skillNo":125,"name":126,"nameZh":6,"description":127,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":128,"categories":129,"keywords":130,"publishedAt":19},"01M78JP22K73KK96JH0JNDT5C4","iap-agent-platform-troubleshooting","Diagnoses Identity-Aware Proxy (IAP) denials of outbound requests from Gemini Enterprise Agent Platform agents that run on Agent Runtime and send traffic through Agent Gateway. Use when an agent request fails with \"Egress request is not authorized\", when Cloud Audit Logs show an IAP egressViaIAP decision with granted false, when IAP audit entries are missing while debugging agent egress, or when Access policies, IAP allow policies, policy bindings, or Agent Registry entries might block agent egress. Don't use for Gemini model IAM permission errors (such as aiplatform.endpoints.predict), Agent Platform API errors, or Agent Runtime deployment or 500 errors (use agent-platform-troubleshooting instead). Don't use for setting up, creating, or connecting an Agent Gateway, or for IAP web apps, IAP TCP forwarding, or SSH.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fiap-agent-platform-troubleshooting\u002FSKILL.md",[],[]]