[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M70JSVGGSBNPP15NYZ8S5WE8":3,"skill-same:01M70JSVGGSBNPP15NYZ8S5WE8":72},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":43,"linkOnly":44,"markdownContent":45,"markdownBody":46,"frontMatter":47,"refName":48,"commitSha":49,"skillDirectory":50,"forksCount":51,"repositoryHtmlUrl":52,"repositoryHomepageUrl":53,"ownerHtmlUrl":54,"ownerBlogUrl":55,"ownerDescription":56,"updatedAt":57,"install":58,"resources":71},"01M70JSVGGSBNPP15NYZ8S5WE8","data-quality-frameworks",null,"Implement data quality validation with Great Expectations, dbt tests, and data contracts. Use when building data quality pipelines, implementing validation rules, or establishing data contracts.","shengdabai\u002FTony-Claude-Code-Skills","shengdabai","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F116417419?v=4","01M70F6D2WY5PQV5A92BKYJQEW","Tony Sheng","01M70F6D3GAAAYZ5H4CC1G8VXM",3,"MIT","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002F_collections\u002Fwshobson-agents\u002Fplugins\u002Fdata-engineering\u002Fskills\u002Fdata-quality-frameworks\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000033","数据分析师","PROFESSION","适用职业",[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42],"data quality frameworks","data quality","quality","data","frameworks","contracts","pipelines","data contracts","dbt","validation","with","implement","when","quality validation","Rules","use","tests","implementing","great expectations","2026-11-10 10:12:30",false,"---\nname: data-quality-frameworks\ndescription: Implement data quality validation with Great Expectations, dbt tests, and data contracts. Use when building data quality pipelines, implementing validation rules, or establishing data contracts.\n---\n\n# Data Quality Frameworks\n\nProduction patterns for implementing data quality with Great Expectations, dbt tests, and data contracts to ensure reliable data pipelines.\n\n## When to Use This Skill\n\n- Implementing data quality checks in pipelines\n- Setting up Great Expectations validation\n- Building comprehensive dbt test suites\n- Establishing data contracts between teams\n- Monitoring data quality metrics\n- Automating data validation in CI\u002FCD\n\n## Core Concepts\n\n### 1. Data Quality Dimensions\n\n| Dimension | Description | Example Check |\n|-----------|-------------|---------------|\n| **Completeness** | No missing values | `expect_column_values_to_not_be_null` |\n| **Uniqueness** | No duplicates | `expect_column_values_to_be_unique` |\n| **Validity** | Values in expected range | `expect_column_values_to_be_in_set` |\n| **Accuracy** | Data matches reality | Cross-reference validation |\n| **Consistency** | No contradictions | `expect_column_pair_values_A_to_be_greater_than_B` |\n| **Timeliness** | Data is recent | `expect_column_max_to_be_between` |\n\n### 2. Testing Pyramid for Data\n\n```\n          \u002F\\\n         \u002F  \\     Integration Tests (cross-table)\n        \u002F────\\\n       \u002F      \\   Unit Tests (single column)\n      \u002F────────\\\n     \u002F          \\ Schema Tests (structure)\n    \u002F────────────\\\n```\n\n## Quick Start\n\n### Great Expectations Setup\n\n```bash\n# Install\npip install great_expectations\n\n# Initialize project\ngreat_expectations init\n\n# Create datasource\ngreat_expectations datasource new\n```\n\n```python\n# great_expectations\u002Fcheckpoints\u002Fdaily_validation.yml\nimport great_expectations as gx\n\n# Create context\ncontext = gx.get_context()\n\n# Create expectation suite\nsuite = context.add_expectation_suite(\"orders_suite\")\n\n# Add expectations\nsuite.add_expectation(\n    gx.expectations.ExpectColumnValuesToNotBeNull(column=\"order_id\")\n)\nsuite.add_expectation(\n    gx.expectations.ExpectColumnValuesToBeUnique(column=\"order_id\")\n)\n\n# Validate\nresults = context.run_checkpoint(checkpoint_name=\"daily_orders\")\n```\n\n## Patterns\n\n### Pattern 1: Great Expectations Suite\n\n```python\n# expectations\u002Forders_suite.py\nimport great_expectations as gx\nfrom great_expectations.core import ExpectationSuite\nfrom great_expectations.core.expectation_configuration import ExpectationConfiguration\n\ndef build_orders_suite() -> ExpectationSuite:\n    \"\"\"Build comprehensive orders expectation suite\"\"\"\n\n    suite = ExpectationSuite(expectation_suite_name=\"orders_suite\")\n\n    # Schema expectations\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_table_columns_to_match_set\",\n        kwargs={\n            \"column_set\": [\"order_id\", \"customer_id\", \"amount\", \"status\", \"created_at\"],\n            \"exact_match\": False  # Allow additional columns\n        }\n    ))\n\n    # Primary key\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_values_to_not_be_null\",\n        kwargs={\"column\": \"order_id\"}\n    ))\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_values_to_be_unique\",\n        kwargs={\"column\": \"order_id\"}\n    ))\n\n    # Foreign key\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_values_to_not_be_null\",\n        kwargs={\"column\": \"customer_id\"}\n    ))\n\n    # Categorical values\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_values_to_be_in_set\",\n        kwargs={\n            \"column\": \"status\",\n            \"value_set\": [\"pending\", \"processing\", \"shipped\", \"delivered\", \"cancelled\"]\n        }\n    ))\n\n    # Numeric ranges\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_values_to_be_between\",\n        kwargs={\n            \"column\": \"amount\",\n            \"min_value\": 0,\n            \"max_value\": 100000,\n            \"strict_min\": True  # amount > 0\n        }\n    ))\n\n    # Date validity\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_values_to_be_dateutil_parseable\",\n        kwargs={\"column\": \"created_at\"}\n    ))\n\n    # Freshness - data should be recent\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_max_to_be_between\",\n        kwargs={\n            \"column\": \"created_at\",\n            \"min_value\": {\"$PARAMETER\": \"now - timedelta(days=1)\"},\n            \"max_value\": {\"$PARAMETER\": \"now\"}\n        }\n    ))\n\n    # Row count sanity\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_table_row_count_to_be_between\",\n        kwargs={\n            \"min_value\": 1000,  # Expect at least 1000 rows\n            \"max_value\": 10000000\n        }\n    ))\n\n    # Statistical expectations\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_mean_to_be_between\",\n        kwargs={\n            \"column\": \"amount\",\n            \"min_value\": 50,\n            \"max_value\": 500\n        }\n    ))\n\n    return suite\n```\n\n### Pattern 2: Great Expectations Checkpoint\n\n```yaml\n# great_expectations\u002Fcheckpoints\u002Forders_checkpoint.yml\nname: orders_checkpoint\nconfig_version: 1.0\nclass_name: Checkpoint\nrun_name_template: \"%Y%m%d-%H%M%S-orders-validation\"\n\nvalidations:\n  - batch_request:\n      datasource_name: warehouse\n      data_connector_name: default_inferred_data_connector_name\n      data_asset_name: orders\n      data_connector_query:\n        index: -1  # Latest batch\n    expectation_suite_name: orders_suite\n\naction_list:\n  - name: store_validation_result\n    action:\n      class_name: StoreValidationResultAction\n\n  - name: store_evaluation_parameters\n    action:\n      class_name: StoreEvaluationParametersAction\n\n  - name: update_data_docs\n    action:\n      class_name: UpdateDataDocsAction\n\n  # Slack notification on failure\n  - name: send_slack_notification\n    action:\n      class_name: SlackNotificationAction\n      slack_webhook: ${SLACK_WEBHOOK}\n      notify_on: failure\n      renderer:\n        module_name: great_expectations.render.renderer.slack_renderer\n        class_name: SlackRenderer\n```\n\n```python\n# Run checkpoint\nimport great_expectations as gx\n\ncontext = gx.get_context()\nresult = context.run_checkpoint(checkpoint_name=\"orders_checkpoint\")\n\nif not result.success:\n    failed_expectations = [\n        r for r in result.run_results.values()\n        if not r.success\n    ]\n    raise ValueError(f\"Data quality check failed: {failed_expectations}\")\n```\n\n### Pattern 3: dbt Data Tests\n\n```yaml\n# models\u002Fmarts\u002Fcore\u002F_core__models.yml\nversion: 2\n\nmodels:\n  - name: fct_orders\n    description: Order fact table\n    tests:\n      # Table-level tests\n      - dbt_utils.recency:\n          datepart: day\n          field: created_at\n          interval: 1\n      - dbt_utils.at_least_one\n      - dbt_utils.expression_is_true:\n          expression: \"total_amount >= 0\"\n\n    columns:\n      - name: order_id\n        description: Primary key\n        tests:\n          - unique\n          - not_null\n\n      - name: customer_id\n        description: Foreign key to dim_customers\n        tests:\n          - not_null\n          - relationships:\n              to: ref('dim_customers')\n              field: customer_id\n\n      - name: order_status\n        tests:\n          - accepted_values:\n              values: ['pending', 'processing', 'shipped', 'delivered', 'cancelled']\n\n      - name: total_amount\n        tests:\n          - not_null\n          - dbt_utils.expression_is_true:\n              expression: \">= 0\"\n\n      - name: created_at\n        tests:\n          - not_null\n          - dbt_utils.expression_is_true:\n              expression: \"\u003C= current_timestamp\"\n\n  - name: dim_customers\n    columns:\n      - name: customer_id\n        tests:\n          - unique\n          - not_null\n\n      - name: email\n        tests:\n          - unique\n          - not_null\n          # Custom regex test\n          - dbt_utils.expression_is_true:\n              expression: \"email ~ '^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\\\.[A-Za-z]{2,}$'\"\n```\n\n### Pattern 4: Custom dbt Tests\n\n```sql\n-- tests\u002Fgeneric\u002Ftest_row_count_in_range.sql\n{% test row_count_in_range(model, min_count, max_count) %}\n\nwith row_count as (\n    select count(*) as cnt from {{ model }}\n)\n\nselect cnt\nfrom row_count\nwhere cnt \u003C {{ min_count }} or cnt > {{ max_count }}\n\n{% endtest %}\n\n-- Usage in schema.yml:\n-- tests:\n--   - row_count_in_range:\n--       min_count: 1000\n--       max_count: 10000000\n```\n\n```sql\n-- tests\u002Fgeneric\u002Ftest_sequential_values.sql\n{% test sequential_values(model, column_name, interval=1) %}\n\nwith lagged as (\n    select\n        {{ column_name }},\n        lag({{ column_name }}) over (order by {{ column_name }}) as prev_value\n    from {{ model }}\n)\n\nselect *\nfrom lagged\nwhere {{ column_name }} - prev_value != {{ interval }}\n  and prev_value is not null\n\n{% endtest %}\n```\n\n```sql\n-- tests\u002Fsingular\u002Fassert_orders_customers_match.sql\n-- Singular test: specific business rule\n\nwith orders_customers as (\n    select distinct customer_id from {{ ref('fct_orders') }}\n),\n\ndim_customers as (\n    select customer_id from {{ ref('dim_customers') }}\n),\n\norphaned_orders as (\n    select o.customer_id\n    from orders_customers o\n    left join dim_customers c using (customer_id)\n    where c.customer_id is null\n)\n\nselect * from orphaned_orders\n-- Test passes if this returns 0 rows\n```\n\n### Pattern 5: Data Contracts\n\n```yaml\n# contracts\u002Forders_contract.yaml\napiVersion: datacontract.com\u002Fv1.0.0\nkind: DataContract\nmetadata:\n  name: orders\n  version: 1.0.0\n  owner: data-platform-team\n  contact: data-team@company.com\n\ninfo:\n  title: Orders Data Contract\n  description: Contract for order event data from the ecommerce platform\n  purpose: Analytics, reporting, and ML features\n\nservers:\n  production:\n    type: snowflake\n    account: company.us-east-1\n    database: ANALYTICS\n    schema: CORE\n\nterms:\n  usage: Internal analytics only\n  limitations: PII must not be exposed in downstream marts\n  billing: Charged per query TB scanned\n\nschema:\n  type: object\n  properties:\n    order_id:\n      type: string\n      format: uuid\n      description: Unique order identifier\n      required: true\n      unique: true\n      pii: false\n\n    customer_id:\n      type: string\n      format: uuid\n      description: Customer identifier\n      required: true\n      pii: true\n      piiClassification: indirect\n\n    total_amount:\n      type: number\n      minimum: 0\n      maximum: 100000\n      description: Order total in USD\n\n    created_at:\n      type: string\n      format: date-time\n      description: Order creation timestamp\n      required: true\n\n    status:\n      type: string\n      enum: [pending, processing, shipped, delivered, cancelled]\n      description: Current order status\n\nquality:\n  type: SodaCL\n  specification:\n    checks for orders:\n      - row_count > 0\n      - missing_count(order_id) = 0\n      - duplicate_count(order_id) = 0\n      - invalid_count(status) = 0:\n          valid values: [pending, processing, shipped, delivered, cancelled]\n      - freshness(created_at) \u003C 24h\n\nsla:\n  availability: 99.9%\n  freshness: 1 hour\n  latency: 5 minutes\n```\n\n### Pattern 6: Automated Quality Pipeline\n\n```python\n# quality_pipeline.py\nfrom dataclasses import dataclass\nfrom typing import List, Dict, Any\nimport great_expectations as gx\nfrom datetime import datetime\n\n@dataclass\nclass QualityResult:\n    table: str\n    passed: bool\n    total_expectations: int\n    failed_expectations: int\n    details: List[Dict[str, Any]]\n    timestamp: datetime\n\nclass DataQualityPipeline:\n    \"\"\"Orchestrate data quality checks across tables\"\"\"\n\n    def __init__(self, context: gx.DataContext):\n        self.context = context\n        self.results: List[QualityResult] = []\n\n    def validate_table(self, table: str, suite: str) -> QualityResult:\n        \"\"\"Validate a single table against expectation suite\"\"\"\n\n        checkpoint_config = {\n            \"name\": f\"{table}_validation\",\n            \"config_version\": 1.0,\n            \"class_name\": \"Checkpoint\",\n            \"validations\": [{\n                \"batch_request\": {\n                    \"datasource_name\": \"warehouse\",\n                    \"data_asset_name\": table,\n                },\n                \"expectation_suite_name\": suite,\n            }],\n        }\n\n        result = self.context.run_checkpoint(**checkpoint_config)\n\n        # Parse results\n        validation_result = list(result.run_results.values())[0]\n        results = validation_result.results\n\n        failed = [r for r in results if not r.success]\n\n        return QualityResult(\n            table=table,\n            passed=result.success,\n            total_expectations=len(results),\n            failed_expectations=len(failed),\n            details=[{\n                \"expectation\": r.expectation_config.expectation_type,\n                \"success\": r.success,\n                \"observed_value\": r.result.get(\"observed_value\"),\n            } for r in results],\n            timestamp=datetime.now()\n        )\n\n    def run_all(self, tables: Dict[str, str]) -> Dict[str, QualityResult]:\n        \"\"\"Run validation for all tables\"\"\"\n        results = {}\n\n        for table, suite in tables.items():\n            print(f\"Validating {table}...\")\n            results[table] = self.validate_table(table, suite)\n\n        return results\n\n    def generate_report(self, results: Dict[str, QualityResult]) -> str:\n        \"\"\"Generate quality report\"\"\"\n        report = [\"# Data Quality Report\", f\"Generated: {datetime.now()}\", \"\"]\n\n        total_passed = sum(1 for r in results.values() if r.passed)\n        total_tables = len(results)\n\n        report.append(f\"## Summary: {total_passed}\u002F{total_tables} tables passed\")\n        report.append(\"\")\n\n        for table, result in results.items():\n            status = \"✅\" if result.passed else \"❌\"\n            report.append(f\"### {status} {table}\")\n            report.append(f\"- Expectations: {result.total_expectations}\")\n            report.append(f\"- Failed: {result.failed_expectations}\")\n\n            if not result.passed:\n                report.append(\"- Failed checks:\")\n                for detail in result.details:\n                    if not detail[\"success\"]:\n                        report.append(f\"  - {detail['expectation']}: {detail['observed_value']}\")\n            report.append(\"\")\n\n        return \"\\n\".join(report)\n\n# Usage\ncontext = gx.get_context()\npipeline = DataQualityPipeline(context)\n\ntables_to_validate = {\n    \"orders\": \"orders_suite\",\n    \"customers\": \"customers_suite\",\n    \"products\": \"products_suite\",\n}\n\nresults = pipeline.run_all(tables_to_validate)\nreport = pipeline.generate_report(results)\n\n# Fail pipeline if any table failed\nif not all(r.passed for r in results.values()):\n    print(report)\n    raise ValueError(\"Data quality checks failed!\")\n```\n\n## Best Practices\n\n### Do's\n- **Test early** - Validate source data before transformations\n- **Test incrementally** - Add tests as you find issues\n- **Document expectations** - Clear descriptions for each test\n- **Alert on failures** - Integrate with monitoring\n- **Version contracts** - Track schema changes\n\n### Don'ts\n- **Don't test everything** - Focus on critical columns\n- **Don't ignore warnings** - They often precede failures\n- **Don't skip freshness** - Stale data is bad data\n- **Don't hardcode thresholds** - Use dynamic baselines\n- **Don't test in isolation** - Test relationships too\n\n## Resources\n\n- [Great Expectations Documentation](https:\u002F\u002Fdocs.greatexpectations.io\u002F)\n- [dbt Testing Documentation](https:\u002F\u002Fdocs.getdbt.com\u002Fdocs\u002Fbuild\u002Ftests)\n- [Data Contract Specification](https:\u002F\u002Fdatacontract.com\u002F)\n- [Soda Core](https:\u002F\u002Fdocs.soda.io\u002Fsoda-core\u002Foverview.html)\n","# Data Quality Frameworks\n\nProduction patterns for implementing data quality with Great Expectations, dbt tests, and data contracts to ensure reliable data pipelines.\n\n## When to Use This Skill\n\n- Implementing data quality checks in pipelines\n- Setting up Great Expectations validation\n- Building comprehensive dbt test suites\n- Establishing data contracts between teams\n- Monitoring data quality metrics\n- Automating data validation in CI\u002FCD\n\n## Core Concepts\n\n### 1. Data Quality Dimensions\n\n| Dimension | Description | Example Check |\n|-----------|-------------|---------------|\n| **Completeness** | No missing values | `expect_column_values_to_not_be_null` |\n| **Uniqueness** | No duplicates | `expect_column_values_to_be_unique` |\n| **Validity** | Values in expected range | `expect_column_values_to_be_in_set` |\n| **Accuracy** | Data matches reality | Cross-reference validation |\n| **Consistency** | No contradictions | `expect_column_pair_values_A_to_be_greater_than_B` |\n| **Timeliness** | Data is recent | `expect_column_max_to_be_between` |\n\n### 2. Testing Pyramid for Data\n\n```\n          \u002F\\\n         \u002F  \\     Integration Tests (cross-table)\n        \u002F────\\\n       \u002F      \\   Unit Tests (single column)\n      \u002F────────\\\n     \u002F          \\ Schema Tests (structure)\n    \u002F────────────\\\n```\n\n## Quick Start\n\n### Great Expectations Setup\n\n```bash\n# Install\npip install great_expectations\n\n# Initialize project\ngreat_expectations init\n\n# Create datasource\ngreat_expectations datasource new\n```\n\n```python\n# great_expectations\u002Fcheckpoints\u002Fdaily_validation.yml\nimport great_expectations as gx\n\n# Create context\ncontext = gx.get_context()\n\n# Create expectation suite\nsuite = context.add_expectation_suite(\"orders_suite\")\n\n# Add expectations\nsuite.add_expectation(\n    gx.expectations.ExpectColumnValuesToNotBeNull(column=\"order_id\")\n)\nsuite.add_expectation(\n    gx.expectations.ExpectColumnValuesToBeUnique(column=\"order_id\")\n)\n\n# Validate\nresults = context.run_checkpoint(checkpoint_name=\"daily_orders\")\n```\n\n## Patterns\n\n### Pattern 1: Great Expectations Suite\n\n```python\n# expectations\u002Forders_suite.py\nimport great_expectations as gx\nfrom great_expectations.core import ExpectationSuite\nfrom great_expectations.core.expectation_configuration import ExpectationConfiguration\n\ndef build_orders_suite() -> ExpectationSuite:\n    \"\"\"Build comprehensive orders expectation suite\"\"\"\n\n    suite = ExpectationSuite(expectation_suite_name=\"orders_suite\")\n\n    # Schema expectations\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_table_columns_to_match_set\",\n        kwargs={\n            \"column_set\": [\"order_id\", \"customer_id\", \"amount\", \"status\", \"created_at\"],\n            \"exact_match\": False  # Allow additional columns\n        }\n    ))\n\n    # Primary key\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_values_to_not_be_null\",\n        kwargs={\"column\": \"order_id\"}\n    ))\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_values_to_be_unique\",\n        kwargs={\"column\": \"order_id\"}\n    ))\n\n    # Foreign key\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_values_to_not_be_null\",\n        kwargs={\"column\": \"customer_id\"}\n    ))\n\n    # Categorical values\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_values_to_be_in_set\",\n        kwargs={\n            \"column\": \"status\",\n            \"value_set\": [\"pending\", \"processing\", \"shipped\", \"delivered\", \"cancelled\"]\n        }\n    ))\n\n    # Numeric ranges\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_values_to_be_between\",\n        kwargs={\n            \"column\": \"amount\",\n            \"min_value\": 0,\n            \"max_value\": 100000,\n            \"strict_min\": True  # amount > 0\n        }\n    ))\n\n    # Date validity\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_values_to_be_dateutil_parseable\",\n        kwargs={\"column\": \"created_at\"}\n    ))\n\n    # Freshness - data should be recent\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_max_to_be_between\",\n        kwargs={\n            \"column\": \"created_at\",\n            \"min_value\": {\"$PARAMETER\": \"now - timedelta(days=1)\"},\n            \"max_value\": {\"$PARAMETER\": \"now\"}\n        }\n    ))\n\n    # Row count sanity\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_table_row_count_to_be_between\",\n        kwargs={\n            \"min_value\": 1000,  # Expect at least 1000 rows\n            \"max_value\": 10000000\n        }\n    ))\n\n    # Statistical expectations\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_mean_to_be_between\",\n        kwargs={\n            \"column\": \"amount\",\n            \"min_value\": 50,\n            \"max_value\": 500\n        }\n    ))\n\n    return suite\n```\n\n### Pattern 2: Great Expectations Checkpoint\n\n```yaml\n# great_expectations\u002Fcheckpoints\u002Forders_checkpoint.yml\nname: orders_checkpoint\nconfig_version: 1.0\nclass_name: Checkpoint\nrun_name_template: \"%Y%m%d-%H%M%S-orders-validation\"\n\nvalidations:\n  - batch_request:\n      datasource_name: warehouse\n      data_connector_name: default_inferred_data_connector_name\n      data_asset_name: orders\n      data_connector_query:\n        index: -1  # Latest batch\n    expectation_suite_name: orders_suite\n\naction_list:\n  - name: store_validation_result\n    action:\n      class_name: StoreValidationResultAction\n\n  - name: store_evaluation_parameters\n    action:\n      class_name: StoreEvaluationParametersAction\n\n  - name: update_data_docs\n    action:\n      class_name: UpdateDataDocsAction\n\n  # Slack notification on failure\n  - name: send_slack_notification\n    action:\n      class_name: SlackNotificationAction\n      slack_webhook: ${SLACK_WEBHOOK}\n      notify_on: failure\n      renderer:\n        module_name: great_expectations.render.renderer.slack_renderer\n        class_name: SlackRenderer\n```\n\n```python\n# Run checkpoint\nimport great_expectations as gx\n\ncontext = gx.get_context()\nresult = context.run_checkpoint(checkpoint_name=\"orders_checkpoint\")\n\nif not result.success:\n    failed_expectations = [\n        r for r in result.run_results.values()\n        if not r.success\n    ]\n    raise ValueError(f\"Data quality check failed: {failed_expectations}\")\n```\n\n### Pattern 3: dbt Data Tests\n\n```yaml\n# models\u002Fmarts\u002Fcore\u002F_core__models.yml\nversion: 2\n\nmodels:\n  - name: fct_orders\n    description: Order fact table\n    tests:\n      # Table-level tests\n      - dbt_utils.recency:\n          datepart: day\n          field: created_at\n          interval: 1\n      - dbt_utils.at_least_one\n      - dbt_utils.expression_is_true:\n          expression: \"total_amount >= 0\"\n\n    columns:\n      - name: order_id\n        description: Primary key\n        tests:\n          - unique\n          - not_null\n\n      - name: customer_id\n        description: Foreign key to dim_customers\n        tests:\n          - not_null\n          - relationships:\n              to: ref('dim_customers')\n              field: customer_id\n\n      - name: order_status\n        tests:\n          - accepted_values:\n              values: ['pending', 'processing', 'shipped', 'delivered', 'cancelled']\n\n      - name: total_amount\n        tests:\n          - not_null\n          - dbt_utils.expression_is_true:\n              expression: \">= 0\"\n\n      - name: created_at\n        tests:\n          - not_null\n          - dbt_utils.expression_is_true:\n              expression: \"\u003C= current_timestamp\"\n\n  - name: dim_customers\n    columns:\n      - name: customer_id\n        tests:\n          - unique\n          - not_null\n\n      - name: email\n        tests:\n          - unique\n          - not_null\n          # Custom regex test\n          - dbt_utils.expression_is_true:\n              expression: \"email ~ '^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\\\.[A-Za-z]{2,}$'\"\n```\n\n### Pattern 4: Custom dbt Tests\n\n```sql\n-- tests\u002Fgeneric\u002Ftest_row_count_in_range.sql\n{% test row_count_in_range(model, min_count, max_count) %}\n\nwith row_count as (\n    select count(*) as cnt from {{ model }}\n)\n\nselect cnt\nfrom row_count\nwhere cnt \u003C {{ min_count }} or cnt > {{ max_count }}\n\n{% endtest %}\n\n-- Usage in schema.yml:\n-- tests:\n--   - row_count_in_range:\n--       min_count: 1000\n--       max_count: 10000000\n```\n\n```sql\n-- tests\u002Fgeneric\u002Ftest_sequential_values.sql\n{% test sequential_values(model, column_name, interval=1) %}\n\nwith lagged as (\n    select\n        {{ column_name }},\n        lag({{ column_name }}) over (order by {{ column_name }}) as prev_value\n    from {{ model }}\n)\n\nselect *\nfrom lagged\nwhere {{ column_name }} - prev_value != {{ interval }}\n  and prev_value is not null\n\n{% endtest %}\n```\n\n```sql\n-- tests\u002Fsingular\u002Fassert_orders_customers_match.sql\n-- Singular test: specific business rule\n\nwith orders_customers as (\n    select distinct customer_id from {{ ref('fct_orders') }}\n),\n\ndim_customers as (\n    select customer_id from {{ ref('dim_customers') }}\n),\n\norphaned_orders as (\n    select o.customer_id\n    from orders_customers o\n    left join dim_customers c using (customer_id)\n    where c.customer_id is null\n)\n\nselect * from orphaned_orders\n-- Test passes if this returns 0 rows\n```\n\n### Pattern 5: Data Contracts\n\n```yaml\n# contracts\u002Forders_contract.yaml\napiVersion: datacontract.com\u002Fv1.0.0\nkind: DataContract\nmetadata:\n  name: orders\n  version: 1.0.0\n  owner: data-platform-team\n  contact: data-team@company.com\n\ninfo:\n  title: Orders Data Contract\n  description: Contract for order event data from the ecommerce platform\n  purpose: Analytics, reporting, and ML features\n\nservers:\n  production:\n    type: snowflake\n    account: company.us-east-1\n    database: ANALYTICS\n    schema: CORE\n\nterms:\n  usage: Internal analytics only\n  limitations: PII must not be exposed in downstream marts\n  billing: Charged per query TB scanned\n\nschema:\n  type: object\n  properties:\n    order_id:\n      type: string\n      format: uuid\n      description: Unique order identifier\n      required: true\n      unique: true\n      pii: false\n\n    customer_id:\n      type: string\n      format: uuid\n      description: Customer identifier\n      required: true\n      pii: true\n      piiClassification: indirect\n\n    total_amount:\n      type: number\n      minimum: 0\n      maximum: 100000\n      description: Order total in USD\n\n    created_at:\n      type: string\n      format: date-time\n      description: Order creation timestamp\n      required: true\n\n    status:\n      type: string\n      enum: [pending, processing, shipped, delivered, cancelled]\n      description: Current order status\n\nquality:\n  type: SodaCL\n  specification:\n    checks for orders:\n      - row_count > 0\n      - missing_count(order_id) = 0\n      - duplicate_count(order_id) = 0\n      - invalid_count(status) = 0:\n          valid values: [pending, processing, shipped, delivered, cancelled]\n      - freshness(created_at) \u003C 24h\n\nsla:\n  availability: 99.9%\n  freshness: 1 hour\n  latency: 5 minutes\n```\n\n### Pattern 6: Automated Quality Pipeline\n\n```python\n# quality_pipeline.py\nfrom dataclasses import dataclass\nfrom typing import List, Dict, Any\nimport great_expectations as gx\nfrom datetime import datetime\n\n@dataclass\nclass QualityResult:\n    table: str\n    passed: bool\n    total_expectations: int\n    failed_expectations: int\n    details: List[Dict[str, Any]]\n    timestamp: datetime\n\nclass DataQualityPipeline:\n    \"\"\"Orchestrate data quality checks across tables\"\"\"\n\n    def __init__(self, context: gx.DataContext):\n        self.context = context\n        self.results: List[QualityResult] = []\n\n    def validate_table(self, table: str, suite: str) -> QualityResult:\n        \"\"\"Validate a single table against expectation suite\"\"\"\n\n        checkpoint_config = {\n            \"name\": f\"{table}_validation\",\n            \"config_version\": 1.0,\n            \"class_name\": \"Checkpoint\",\n            \"validations\": [{\n                \"batch_request\": {\n                    \"datasource_name\": \"warehouse\",\n                    \"data_asset_name\": table,\n                },\n                \"expectation_suite_name\": suite,\n            }],\n        }\n\n        result = self.context.run_checkpoint(**checkpoint_config)\n\n        # Parse results\n        validation_result = list(result.run_results.values())[0]\n        results = validation_result.results\n\n        failed = [r for r in results if not r.success]\n\n        return QualityResult(\n            table=table,\n            passed=result.success,\n            total_expectations=len(results),\n            failed_expectations=len(failed),\n            details=[{\n                \"expectation\": r.expectation_config.expectation_type,\n                \"success\": r.success,\n                \"observed_value\": r.result.get(\"observed_value\"),\n            } for r in results],\n            timestamp=datetime.now()\n        )\n\n    def run_all(self, tables: Dict[str, str]) -> Dict[str, QualityResult]:\n        \"\"\"Run validation for all tables\"\"\"\n        results = {}\n\n        for table, suite in tables.items():\n            print(f\"Validating {table}...\")\n            results[table] = self.validate_table(table, suite)\n\n        return results\n\n    def generate_report(self, results: Dict[str, QualityResult]) -> str:\n        \"\"\"Generate quality report\"\"\"\n        report = [\"# Data Quality Report\", f\"Generated: {datetime.now()}\", \"\"]\n\n        total_passed = sum(1 for r in results.values() if r.passed)\n        total_tables = len(results)\n\n        report.append(f\"## Summary: {total_passed}\u002F{total_tables} tables passed\")\n        report.append(\"\")\n\n        for table, result in results.items():\n            status = \"✅\" if result.passed else \"❌\"\n            report.append(f\"### {status} {table}\")\n            report.append(f\"- Expectations: {result.total_expectations}\")\n            report.append(f\"- Failed: {result.failed_expectations}\")\n\n            if not result.passed:\n                report.append(\"- Failed checks:\")\n                for detail in result.details:\n                    if not detail[\"success\"]:\n                        report.append(f\"  - {detail['expectation']}: {detail['observed_value']}\")\n            report.append(\"\")\n\n        return \"\\n\".join(report)\n\n# Usage\ncontext = gx.get_context()\npipeline = DataQualityPipeline(context)\n\ntables_to_validate = {\n    \"orders\": \"orders_suite\",\n    \"customers\": \"customers_suite\",\n    \"products\": \"products_suite\",\n}\n\nresults = pipeline.run_all(tables_to_validate)\nreport = pipeline.generate_report(results)\n\n# Fail pipeline if any table failed\nif not all(r.passed for r in results.values()):\n    print(report)\n    raise ValueError(\"Data quality checks failed!\")\n```\n\n## Best Practices\n\n### Do's\n- **Test early** - Validate source data before transformations\n- **Test incrementally** - Add tests as you find issues\n- **Document expectations** - Clear descriptions for each test\n- **Alert on failures** - Integrate with monitoring\n- **Version contracts** - Track schema changes\n\n### Don'ts\n- **Don't test everything** - Focus on critical columns\n- **Don't ignore warnings** - They often precede failures\n- **Don't skip freshness** - Stale data is bad data\n- **Don't hardcode thresholds** - Use dynamic baselines\n- **Don't test in isolation** - Test relationships too\n\n## Resources\n\n- [Great Expectations Documentation](https:\u002F\u002Fdocs.greatexpectations.io\u002F)\n- [dbt Testing Documentation](https:\u002F\u002Fdocs.getdbt.com\u002Fdocs\u002Fbuild\u002Ftests)\n- [Data Contract Specification](https:\u002F\u002Fdatacontract.com\u002F)\n- [Soda Core](https:\u002F\u002Fdocs.soda.io\u002Fsoda-core\u002Foverview.html)\n",{"name":5,"description":7},"main","7850740bd837adb32f0a109ab2af82482196c6f2","skills\u002F_collections\u002Fwshobson-agents\u002Fplugins\u002Fdata-engineering\u002Fskills\u002Fdata-quality-frameworks",1,"https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills","https:\u002F\u002Fzturnsgo.com","https:\u002F\u002Fgithub.com\u002Fshengdabai","https:\u002F\u002Fzturnsgo.com\u002F","AI × education creator — producer, not consumer. Taught 6,000+ learners; shipping learning tools & daily essays with Claude Code, practicing @xiaolai's way 🥋","2026-11-01 21:28:23",{"source":8,"skillName":5,"commands":59,"prompt":69,"zipAvailable":70},[60,63,66],{"tool":61,"command":62},"npx","npx skills add shengdabai\u002FTony-Claude-Code-Skills --skill data-quality-frameworks",{"tool":64,"command":65},"bunx","bunx skills add shengdabai\u002FTony-Claude-Code-Skills --skill data-quality-frameworks",{"tool":67,"command":68},"pnpm","pnpm dlx skills add shengdabai\u002FTony-Claude-Code-Skills --skill data-quality-frameworks","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002F_collections\u002Fwshobson-agents\u002Fplugins\u002Fdata-engineering\u002Fskills\u002Fdata-quality-frameworks\u002FSKILL.md",true,[],[73,82,90,98,106,114],{"skillNo":74,"name":75,"nameZh":6,"description":76,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":77,"sourceHtmlUrl":78,"categories":79,"keywords":80,"publishedAt":81},"01M73H7XKBHE1ETQDZT94MF9HH","seedance-prompt-en","Write effective prompts for Jimeng Seedance 2.0 multimodal AI video generation. Use when users want to create video prompts using text, images, videos, and audio inputs with the @ reference system. Covers camera movements, effects replication, video extension, editing, music beat-matching, e-commerce ads, short dramas, and educational content.","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fseedance-prompt-en\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fseedance-prompt-en\u002FSKILL.md",[],[],"2026-11-11 15:11:14",{"skillNo":83,"name":84,"nameZh":6,"description":85,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":86,"sourceHtmlUrl":87,"categories":88,"keywords":89,"publishedAt":81},"01M73GRE9998Q1DZWC71N41JWG","onepage-pdf","Convert an HTML page into a single continuous-page PDF (one tall page, no pagination breaks), preserving desktop layout, backgrounds and selectable text. Use when the user asks to turn an HTML file\u002Freport\u002Fproposal into a \"single-page PDF\", \"long PDF\", \"不分页 PDF\", \"单页 PDF\", \"长图式 PDF\", or complains that an HTML-to-PDF export breaks into pages or cuts content. Supports optional string redaction with leak verification.","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fonepage-pdf\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fonepage-pdf\u002FSKILL.md",[],[],{"skillNo":91,"name":92,"nameZh":6,"description":93,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":94,"sourceHtmlUrl":95,"categories":96,"keywords":97,"publishedAt":81},"01M73GQG1KZDADVPV3MHAT1J5A","native-feel-cross-platform-desktop","Use when the user is designing, prototyping, or rewriting a desktop app that must run on multiple OSes (macOS + Windows, optionally Linux) AND feel indistinguishable from a native app to its users — fast launch, native windowing, native input handling, native materials. Trigger words include \"cross-platform desktop\", \"Electron alternative\", \"Tauri vs native\", \"WebView wrapper\", \"near-native performance\", \"Raycast architecture\", \"WebKit\u002FWebView2 quirks\", \"WKWebView\", \"system tray app\", \"global hotkey app\", \"launcher app\". Do NOT trigger this skill for pure web apps, pure mobile apps, or for greenfield projects that have no native-feel requirement.","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fnative-feel-cross-platform-desktop\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fnative-feel-cross-platform-desktop\u002FSKILL.md",[],[],{"skillNo":99,"name":100,"nameZh":6,"description":101,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":102,"sourceHtmlUrl":103,"categories":104,"keywords":105,"publishedAt":81},"01M73G87PDNSMKG2FHWCYER6N8","gstack","Router for the gstack skill suite. (gstack)","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fgstack\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fgstack\u002FSKILL.md",[],[],{"skillNo":107,"name":108,"nameZh":6,"description":109,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":110,"sourceHtmlUrl":111,"categories":112,"keywords":113,"publishedAt":81},"01M73G3EAQWKCM8DB8D0N0QSVG","fireworks-tech-graph","Use when the user wants to create any technical diagram - architecture, data flow, flowchart, sequence, agent\u002Fmemory, or concept map - and export as SVG+PNG. Trigger on: \"画图\" \"帮我画\" \"生成图\" \"做个图\" \"架构图\" \"流程图\" \"可视化一下\" \"出图\" \"generate diagram\" \"draw diagram\" \"visualize\" or any system\u002Fflow description the user wants illustrated.","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Ffireworks-tech-graph\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Ffireworks-tech-graph\u002FSKILL.md",[],[],{"skillNo":115,"name":116,"nameZh":6,"description":117,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":118,"categories":119,"keywords":120,"publishedAt":81},"01M73G086AXAH9DW1H5G5N3CPB","dbs-video-extract","接收抖音、小红书、微信视频号等短视频链接或分享文案，查询作品／账号数据并提取语音文字稿，按作者和标题归档为 Markdown。用户希望完整解析短视频、取得数据或文字稿时使用；运行前检查 TikHub 与轻抖使用凭证，缺少时用新手能理解的方式引导购买和安全配置。","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fdbs-video-extract\u002FSKILL.md",[],[]]