[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M699RG78PC9EE6KBWWG6Z5BB":3,"skill-same:01M699RG78PC9EE6KBWWG6Z5BB":94},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":20,"linkOnly":21,"markdownContent":22,"markdownBody":23,"frontMatter":24,"refName":25,"commitSha":26,"skillDirectory":27,"forksCount":28,"repositoryHtmlUrl":29,"repositoryHomepageUrl":6,"ownerHtmlUrl":30,"ownerBlogUrl":31,"ownerDescription":32,"updatedAt":33,"install":34,"resources":47},"01M699RG78PC9EE6KBWWG6Z5BB","hdinsight-migration",null,"Ports HDInsight Spark and Hive workloads to Fabric, converting HiveContext and SparkContext to SparkSession, WASB and ABFS paths to OneLake shortcuts, Hive DDL to Delta tables in a Lakehouse, and Oozie coordinators and actions to pipeline activities and triggers. Use when the source workload is HDInsight.","microsoft\u002Fskills-for-fabric","microsoft","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F6154722?v=4","01M698VYDXZ6JW5WH3CRV650R8","Microsoft","01M699NAPJKS2WBHHDN86Q5MM6",1353,"MIT","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Fblob\u002Fmain\u002Fskills\u002Fhdinsight-migration\u002FSKILL.md",[],[19],"hdinsight migration","2026-11-10 10:12:30",false,"---\nname: hdinsight-migration\ndescription: \"Ports HDInsight Spark and Hive workloads to Fabric, converting HiveContext and SparkContext to SparkSession, WASB and ABFS paths to OneLake shortcuts, Hive DDL to Delta tables in a Lakehouse, and Oozie coordinators and actions to pipeline activities and triggers. Use when the source workload is HDInsight.\"\n---\n\n> **Telemetry — MANDATORY.** Every `api.fabric.microsoft.com` call must carry\n> `x-ms-fabric-skill: hdinsight-migration` (`az rest`: `--headers \"x-ms-fabric-skill=hdinsight-migration\"`),\n> including every LRO poll, `fabric_lro` and retry. Snippets omit it — add it anyway.\n\n> **CRITICAL NOTES**\n> 1. To find workspace details (including its ID) from a workspace name: list all workspaces, then use JMESPath filtering\n> 2. To find item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace, then use JMESPath filtering\n> 3. HDInsight has no `mssparkutils` or `dbutils` equivalent — `notebookutils` is net-new capability being introduced\n> 4. `HiveContext` and `SQLContext` are legacy Spark 1.x\u002F2.x APIs — Fabric uses Spark 3.x `SparkSession` exclusively\n> 5. `wasb:\u002F\u002F` paths are deprecated and require a Storage Account key or SAS — replace with OneLake shortcuts\n> 6. Fabric cannot read or shortcut `hdfs:\u002F\u002F` directly. Its Pipeline HDFS connector supports Anonymous authentication only; export or bridge Kerberos sources to supported storage first.\n\n# HDInsight → Microsoft Fabric Migration\n\n## Prerequisite Knowledge\n\nRead these companion documents before executing migration tasks:\n\n- [COMMON-CORE.md](..\u002F..\u002Fcommon\u002FCOMMON-CORE.md) — Fabric REST API patterns, authentication, token audiences, item discovery\n- [COMMON-CLI.md](..\u002F..\u002Fcommon\u002FCOMMON-CLI.md) — `az rest`, `az login`, token acquisition, Fabric REST via CLI\n- [SPARK-AUTHORING-CORE.md](..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md) — Notebook deployment, lakehouse creation, Spark job execution\n\nFor notebook and Lakehouse creation, see [spark-cli](..\u002Fspark-cli\u002FSKILL.md).\nFor Fabric Warehouse DDL\u002FDML authoring, see [sqldw-cli](..\u002Fsqldw-cli\u002FSKILL.md).\n\n---\n\n## Table of Contents\n\n| Topic | Reference |\n|---|---|\n| Migration Workload Map | [§ Migration Workload Map](#migration-workload-map) |\n| SparkSession & Context API Changes | [§ SparkSession API Changes](#sparksession--context-api-changes) |\n| WASB \u002F ABFS → OneLake Path Migration | [path-migration.md](resources\u002Fpath-migration.md) |\n| Hive DDL → Delta Lake \u002F Lakehouse Schemas | [hive-to-delta.md](resources\u002Fhive-to-delta.md) |\n| Oozie → Fabric Pipelines | [§ Oozie → Fabric Pipelines](#oozie--fabric-pipelines) |\n| Introducing `notebookutils` | [§ Introducing notebookutils](#introducing-notebookutils) |\n| Before\u002FAfter Code Patterns | [code-patterns.md](resources\u002Fcode-patterns.md) |\n| Spark Configuration Differences | [§ Spark Configuration Differences](#spark-configuration-differences) |\n| Must \u002F Prefer \u002F Avoid | [§ Must \u002F Prefer \u002F Avoid](#must--prefer--avoid) |\n| Authentication & Token Acquisition | [COMMON-CORE.md § Authentication](..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#authentication--token-acquisition) |\n| Lakehouse Management | [SPARK-AUTHORING-CORE.md § Lakehouse Management](..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md#lakehouse-management) |\n\n---\n\n## Migration Workload Map\n\n| HDInsight Component | Fabric Target | Notes |\n|---|---|---|\n| **Spark cluster** (notebooks, scripts) | Fabric Spark (Lakehouse \u002F Notebooks \u002F SJD) | No persistent cluster — Starter Pool or Custom Pool provides on-demand Spark |\n| **Hive \u002F HiveServer2** | **Lakehouse SQL Endpoint** + Lakehouse schemas | Delta Lake replaces Hive metastore; schemas provide namespace equivalent |\n| **HBase** | **Fabric Warehouse** or **Azure Cosmos DB** (separate from Fabric) | HBase has no direct Fabric equivalent — assess workload access patterns |\n| **Oozie workflows** | **Fabric Data Pipelines** | Map Oozie actions to Fabric activities; see [§ Oozie → Fabric Pipelines](#oozie--fabric-pipelines) |\n| **YARN Resource Manager** | **Fabric Spark monitoring** (Spark UI, Monitoring Hub) | No YARN — Fabric manages compute automatically |\n| **Ambari** | **Fabric Monitoring Hub** + **Admin Portal** | Cluster health, capacity, and job monitoring |\n| **WASB \u002F ABFS storage** | **OneLake Shortcuts** → `abfss:\u002F\u002Fworkspace@onelake.dfs.fabric.microsoft.com\u002F` | See [path-migration.md](resources\u002Fpath-migration.md) |\n| **Ranger policies** | **Fabric workspace roles** + **OneLake data access roles** | Map Ranger row\u002Fcolumn filters to Lakehouse row-level security |\n| **Livy REST server** | **Fabric Livy API** | Compatible endpoint — see SPARK-AUTHORING-CORE.md |\n\n---\n\n## SparkSession & Context API Changes\n\nHDInsight Spark clusters often use legacy Spark 1.x \u002F 2.x API styles. Replace all of these with the unified `SparkSession`:\n\n| Legacy HDInsight Pattern | Fabric Spark 3.x Replacement |\n|---|---|\n| `from pyspark import SparkContext; sc = SparkContext()` | Not needed — `sc = spark.sparkContext` (pre-instantiated) |\n| `from pyspark.sql import HiveContext; hc = HiveContext(sc)` | Not needed — `spark` session has Hive-compatible SQL support via Delta schemas |\n| `from pyspark.sql import SQLContext; sqlc = SQLContext(sc)` | Not needed — use `spark.sql(...)` directly |\n| `SparkSession.builder.enableHiveSupport().getOrCreate()` | Not needed in Fabric — `spark` is pre-built and available |\n| `sc.textFile(\"wasb:\u002F\u002Fcontainer@account.blob.core.windows.net\u002Fpath\")` | `spark.read.text(\"abfss:\u002F\u002Fworkspace@onelake.dfs.fabric.microsoft.com\u002Flh.Lakehouse\u002FFiles\u002Fpath\")` |\n| `sqlContext.sql(\"CREATE TABLE ... STORED AS ORC\")` | See [hive-to-delta.md](resources\u002Fhive-to-delta.md) for Delta DDL equivalent |\n\n> In Fabric notebooks, `spark` (SparkSession) and `sc` (SparkContext) are **pre-instantiated** — do not call `SparkContext()` or `SparkSession.builder...getOrCreate()` at the top of migrated notebooks.\n\n---\n\n## Oozie → Fabric Pipelines\n\nMap Oozie workflow actions to Fabric Data Pipeline activities:\n\n| Oozie Action Type | Fabric Pipeline Activity | Notes |\n|---|---|---|\n| `\u003Cspark>` action | **Notebook activity** or **Spark Job Definition activity** | Pass parameters via notebook cell parameters or SJD arguments |\n| `\u003Chive>` action | **Script activity** (SQL) against Lakehouse SQL Endpoint | Convert HiveQL to Spark SQL or Delta SQL |\n| `\u003Cshell>` action | **Azure Function activity** or **Web activity** | Shell scripts must be refactored; no direct shell execution in Fabric Pipelines |\n| `\u003Cjava>` action | **Azure Batch activity** (external) or refactor to PySpark | Java MapReduce jobs must be rewritten |\n| `\u003Csqoop>` action | **Copy Data activity** (Fabric Data Factory connector) | Sqoop import\u002Fexport maps to Fabric Copy Data with JDBC source\u002Fsink |\n| `\u003Ccoordinator>` (time-based schedule) | **Pipeline schedule trigger** | Set recurrence in pipeline trigger; supports cron-like expressions |\n| `\u003Ccoordinator>` (data-triggered) | **Storage Event trigger** | Trigger on OneLake file arrival |\n\n> **Delegate to `spark-cli`** for notebook and SJD creation after mapping pipeline activities.\n\n---\n\n## Introducing `notebookutils`\n\nHDInsight Spark had no built-in utility framework equivalent to `mssparkutils` or `dbutils`. When migrating to Fabric, introduce `notebookutils` for common operations:\n\n| Operation | Old HDInsight Approach | `notebookutils` Equivalent |\n|---|---|---|\n| List files | `dbutils` (N\u002FA) \u002F HDFS CLI | `notebookutils.fs.ls(\"abfss:\u002F\u002F...\")` |\n| Copy file | HDFS API \u002F `shutil` | `notebookutils.fs.cp(src, dest)` |\n| Read secret | Azure Key Vault REST call | `notebookutils.credentials.getSecret(keyVaultUrl, secretName)` |\n| Get notebook context | Not available | `notebookutils.runtime.context` — returns workspace ID, notebook ID, etc. |\n| Run child notebook | Not available | `notebookutils.notebook.run(\"notebook_name\", timeout, {\"param\": \"value\"})` |\n| Exit notebook with value | `sys.exit()` | `notebookutils.notebook.exit(\"value\")` |\n| Mount storage | WASB config in `spark-defaults.conf` | OneLake Shortcut (no runtime mount needed) |\n\n---\n\n## Spark Configuration Differences\n\n| HDInsight Concept | Fabric Spark Equivalent | Migration Action |\n|---|---|---|\n| `spark-defaults.conf` (cluster-wide) | Fabric **Spark Workspace Settings** + **Environment** item | Move config properties to Environment or use `%%configure` in notebooks |\n| `%%configure` magic | `%%configure` magic — **identical** | No change needed |\n| YARN queue \u002F resource allocation | **Fabric Spark pool** node size and autoscale settings | Map queue SLAs to Custom Pool configuration |\n| Ambari service configs (HDFS, YARN tuning) | Not applicable — Fabric manages infrastructure | Remove; focus on application-level Spark configs |\n| HDI Spark version (e.g., Spark 2.4) | Fabric Runtime 1.3 = Spark 3.5 (latest) | Test for deprecated API removals (e.g., `HiveContext`, RDD-style ML) |\n| Conda environment \u002F `bootstrap.sh` | **Fabric Environment** item with custom libraries | Recreate conda\u002Fpip dependencies in a Fabric Environment |\n| `hive-site.xml` (metastore connection) | Not needed — Delta Lake IS the metastore in Fabric | Remove metastore config; use Lakehouse schemas for namespace organization |\n\n---\n\n## Must \u002F Prefer \u002F Avoid\n\n### MUST DO\n- **Replace all `wasb:\u002F\u002F` \u002F `wasbs:\u002F\u002F` paths** with OneLake `abfss:\u002F\u002F` paths or OneLake Shortcuts — `wasb:\u002F\u002F` requires storage account keys which are not the Fabric-preferred auth model\n- **Replace `HiveContext`, `SQLContext`, and standalone `SparkContext()`** — use the pre-instantiated `spark` session in Fabric notebooks\n- **Migrate Hive DDL** (`STORED AS ORC`, `LOCATION`, `TBLPROPERTIES`) to Delta Lake DDL — see [hive-to-delta.md](resources\u002Fhive-to-delta.md)\n- **Introduce `notebookutils`** for file system operations, secret retrieval, and child notebook orchestration where HDInsight used custom scripts or direct API calls\n- **Replace Oozie XML workflows** with Fabric Data Pipelines — see [§ Oozie → Fabric Pipelines](#oozie--fabric-pipelines)\n- **Align library management** to Fabric Environments — remove `bootstrap.sh`, conda envs, and runtime `%pip install` patterns for production workloads\n- **Treat `hdfs:\u002F\u002F` as source-cluster-only** — export or ingest into ADLS Gen2 or OneLake; do not claim that a gateway adds Kerberos support to Fabric's Anonymous-only HDFS connector\n\n### PREFER\n- **OneLake Shortcuts** over copying data — mount existing ADLS Gen2 containers as shortcuts to avoid re-ingestion during migration\n- **Delta Lake** for all tables migrated from Hive ORC\u002FParquet — ACID guarantees, time travel, and schema enforcement improve data quality\n- **Fabric Starter Pool** for initial migration validation — no pool configuration overhead, fast session startup\n- **Lakehouse schemas** (database namespaces) for organizing migrated Hive databases — one schema per Hive database within a single Lakehouse\n- **Medallion architecture** for restructuring migrated data layers during migration — align Bronze\u002FSilver\u002FGold with raw Hive → validated Delta → serving Gold patterns\n\n### AVOID\n- **Do not use `SparkContext()` or `HiveContext()` constructors** in Fabric notebooks — they conflict with the pre-instantiated `spark` session and will raise errors\n- **Do not use `hive-site.xml` or external Hive metastore configuration** — Fabric's Delta Lake-backed Lakehouse IS the metastore\n- **Do not assume YARN queue mappings translate to Fabric pools** — re-design resource allocation based on Fabric Spark pool SLAs\n- **Do not attempt to run Oozie shell actions or Java MapReduce jobs** directly in Fabric — these must be refactored (see [§ Oozie → Fabric Pipelines](#oozie--fabric-pipelines))\n- **Do not use `%sh` magic for file system operations** in production notebooks — use `notebookutils.fs.*` for portability and OneLake token-based auth\n\n---\n\n## Examples\n\nSee [code-patterns.md](resources\u002Fcode-patterns.md) for full before\u002Fafter examples. Key quick references:\n\n**Legacy context → Fabric pre-instantiated session**\n\n```python\n# HDInsight (remove entirely)\nfrom pyspark.sql import HiveContext\nhc = HiveContext(sc)\n\n# Fabric — use pre-instantiated spark directly\ndf = spark.sql(\"SELECT * FROM sales.fact_orders\")\n```\n\n**WASB path → OneLake path (after shortcut creation)**\n\n```python\n# HDInsight\ndf = spark.read.parquet(\"wasb:\u002F\u002Fraw@myaccount.blob.core.windows.net\u002Forders\u002F\")\n\n# Fabric\ndf = spark.read.parquet(\"Files\u002Fraw\u002Forders\u002F\")\n```\n\n**Hive DDL → Delta DDL**\n\n```sql\n-- HDInsight\nCREATE TABLE sales_db.fact_orders (...) STORED AS ORC LOCATION 'wasb:\u002F\u002F...';\n\n-- Fabric\nCREATE SCHEMA IF NOT EXISTS sales_db;\nCREATE TABLE sales_db.fact_orders (...) USING DELTA;\n```\n","> **Telemetry — MANDATORY.** Every `api.fabric.microsoft.com` call must carry\n> `x-ms-fabric-skill: hdinsight-migration` (`az rest`: `--headers \"x-ms-fabric-skill=hdinsight-migration\"`),\n> including every LRO poll, `fabric_lro` and retry. Snippets omit it — add it anyway.\n\n> **CRITICAL NOTES**\n> 1. To find workspace details (including its ID) from a workspace name: list all workspaces, then use JMESPath filtering\n> 2. To find item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace, then use JMESPath filtering\n> 3. HDInsight has no `mssparkutils` or `dbutils` equivalent — `notebookutils` is net-new capability being introduced\n> 4. `HiveContext` and `SQLContext` are legacy Spark 1.x\u002F2.x APIs — Fabric uses Spark 3.x `SparkSession` exclusively\n> 5. `wasb:\u002F\u002F` paths are deprecated and require a Storage Account key or SAS — replace with OneLake shortcuts\n> 6. Fabric cannot read or shortcut `hdfs:\u002F\u002F` directly. Its Pipeline HDFS connector supports Anonymous authentication only; export or bridge Kerberos sources to supported storage first.\n\n# HDInsight → Microsoft Fabric Migration\n\n## Prerequisite Knowledge\n\nRead these companion documents before executing migration tasks:\n\n- [COMMON-CORE.md](..\u002F..\u002Fcommon\u002FCOMMON-CORE.md) — Fabric REST API patterns, authentication, token audiences, item discovery\n- [COMMON-CLI.md](..\u002F..\u002Fcommon\u002FCOMMON-CLI.md) — `az rest`, `az login`, token acquisition, Fabric REST via CLI\n- [SPARK-AUTHORING-CORE.md](..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md) — Notebook deployment, lakehouse creation, Spark job execution\n\nFor notebook and Lakehouse creation, see [spark-cli](..\u002Fspark-cli\u002FSKILL.md).\nFor Fabric Warehouse DDL\u002FDML authoring, see [sqldw-cli](..\u002Fsqldw-cli\u002FSKILL.md).\n\n---\n\n## Table of Contents\n\n| Topic | Reference |\n|---|---|\n| Migration Workload Map | [§ Migration Workload Map](#migration-workload-map) |\n| SparkSession & Context API Changes | [§ SparkSession API Changes](#sparksession--context-api-changes) |\n| WASB \u002F ABFS → OneLake Path Migration | [path-migration.md](resources\u002Fpath-migration.md) |\n| Hive DDL → Delta Lake \u002F Lakehouse Schemas | [hive-to-delta.md](resources\u002Fhive-to-delta.md) |\n| Oozie → Fabric Pipelines | [§ Oozie → Fabric Pipelines](#oozie--fabric-pipelines) |\n| Introducing `notebookutils` | [§ Introducing notebookutils](#introducing-notebookutils) |\n| Before\u002FAfter Code Patterns | [code-patterns.md](resources\u002Fcode-patterns.md) |\n| Spark Configuration Differences | [§ Spark Configuration Differences](#spark-configuration-differences) |\n| Must \u002F Prefer \u002F Avoid | [§ Must \u002F Prefer \u002F Avoid](#must--prefer--avoid) |\n| Authentication & Token Acquisition | [COMMON-CORE.md § Authentication](..\u002F..\u002Fcommon\u002FCOMMON-CORE.md#authentication--token-acquisition) |\n| Lakehouse Management | [SPARK-AUTHORING-CORE.md § Lakehouse Management](..\u002F..\u002Fcommon\u002FSPARK-AUTHORING-CORE.md#lakehouse-management) |\n\n---\n\n## Migration Workload Map\n\n| HDInsight Component | Fabric Target | Notes |\n|---|---|---|\n| **Spark cluster** (notebooks, scripts) | Fabric Spark (Lakehouse \u002F Notebooks \u002F SJD) | No persistent cluster — Starter Pool or Custom Pool provides on-demand Spark |\n| **Hive \u002F HiveServer2** | **Lakehouse SQL Endpoint** + Lakehouse schemas | Delta Lake replaces Hive metastore; schemas provide namespace equivalent |\n| **HBase** | **Fabric Warehouse** or **Azure Cosmos DB** (separate from Fabric) | HBase has no direct Fabric equivalent — assess workload access patterns |\n| **Oozie workflows** | **Fabric Data Pipelines** | Map Oozie actions to Fabric activities; see [§ Oozie → Fabric Pipelines](#oozie--fabric-pipelines) |\n| **YARN Resource Manager** | **Fabric Spark monitoring** (Spark UI, Monitoring Hub) | No YARN — Fabric manages compute automatically |\n| **Ambari** | **Fabric Monitoring Hub** + **Admin Portal** | Cluster health, capacity, and job monitoring |\n| **WASB \u002F ABFS storage** | **OneLake Shortcuts** → `abfss:\u002F\u002Fworkspace@onelake.dfs.fabric.microsoft.com\u002F` | See [path-migration.md](resources\u002Fpath-migration.md) |\n| **Ranger policies** | **Fabric workspace roles** + **OneLake data access roles** | Map Ranger row\u002Fcolumn filters to Lakehouse row-level security |\n| **Livy REST server** | **Fabric Livy API** | Compatible endpoint — see SPARK-AUTHORING-CORE.md |\n\n---\n\n## SparkSession & Context API Changes\n\nHDInsight Spark clusters often use legacy Spark 1.x \u002F 2.x API styles. Replace all of these with the unified `SparkSession`:\n\n| Legacy HDInsight Pattern | Fabric Spark 3.x Replacement |\n|---|---|\n| `from pyspark import SparkContext; sc = SparkContext()` | Not needed — `sc = spark.sparkContext` (pre-instantiated) |\n| `from pyspark.sql import HiveContext; hc = HiveContext(sc)` | Not needed — `spark` session has Hive-compatible SQL support via Delta schemas |\n| `from pyspark.sql import SQLContext; sqlc = SQLContext(sc)` | Not needed — use `spark.sql(...)` directly |\n| `SparkSession.builder.enableHiveSupport().getOrCreate()` | Not needed in Fabric — `spark` is pre-built and available |\n| `sc.textFile(\"wasb:\u002F\u002Fcontainer@account.blob.core.windows.net\u002Fpath\")` | `spark.read.text(\"abfss:\u002F\u002Fworkspace@onelake.dfs.fabric.microsoft.com\u002Flh.Lakehouse\u002FFiles\u002Fpath\")` |\n| `sqlContext.sql(\"CREATE TABLE ... STORED AS ORC\")` | See [hive-to-delta.md](resources\u002Fhive-to-delta.md) for Delta DDL equivalent |\n\n> In Fabric notebooks, `spark` (SparkSession) and `sc` (SparkContext) are **pre-instantiated** — do not call `SparkContext()` or `SparkSession.builder...getOrCreate()` at the top of migrated notebooks.\n\n---\n\n## Oozie → Fabric Pipelines\n\nMap Oozie workflow actions to Fabric Data Pipeline activities:\n\n| Oozie Action Type | Fabric Pipeline Activity | Notes |\n|---|---|---|\n| `\u003Cspark>` action | **Notebook activity** or **Spark Job Definition activity** | Pass parameters via notebook cell parameters or SJD arguments |\n| `\u003Chive>` action | **Script activity** (SQL) against Lakehouse SQL Endpoint | Convert HiveQL to Spark SQL or Delta SQL |\n| `\u003Cshell>` action | **Azure Function activity** or **Web activity** | Shell scripts must be refactored; no direct shell execution in Fabric Pipelines |\n| `\u003Cjava>` action | **Azure Batch activity** (external) or refactor to PySpark | Java MapReduce jobs must be rewritten |\n| `\u003Csqoop>` action | **Copy Data activity** (Fabric Data Factory connector) | Sqoop import\u002Fexport maps to Fabric Copy Data with JDBC source\u002Fsink |\n| `\u003Ccoordinator>` (time-based schedule) | **Pipeline schedule trigger** | Set recurrence in pipeline trigger; supports cron-like expressions |\n| `\u003Ccoordinator>` (data-triggered) | **Storage Event trigger** | Trigger on OneLake file arrival |\n\n> **Delegate to `spark-cli`** for notebook and SJD creation after mapping pipeline activities.\n\n---\n\n## Introducing `notebookutils`\n\nHDInsight Spark had no built-in utility framework equivalent to `mssparkutils` or `dbutils`. When migrating to Fabric, introduce `notebookutils` for common operations:\n\n| Operation | Old HDInsight Approach | `notebookutils` Equivalent |\n|---|---|---|\n| List files | `dbutils` (N\u002FA) \u002F HDFS CLI | `notebookutils.fs.ls(\"abfss:\u002F\u002F...\")` |\n| Copy file | HDFS API \u002F `shutil` | `notebookutils.fs.cp(src, dest)` |\n| Read secret | Azure Key Vault REST call | `notebookutils.credentials.getSecret(keyVaultUrl, secretName)` |\n| Get notebook context | Not available | `notebookutils.runtime.context` — returns workspace ID, notebook ID, etc. |\n| Run child notebook | Not available | `notebookutils.notebook.run(\"notebook_name\", timeout, {\"param\": \"value\"})` |\n| Exit notebook with value | `sys.exit()` | `notebookutils.notebook.exit(\"value\")` |\n| Mount storage | WASB config in `spark-defaults.conf` | OneLake Shortcut (no runtime mount needed) |\n\n---\n\n## Spark Configuration Differences\n\n| HDInsight Concept | Fabric Spark Equivalent | Migration Action |\n|---|---|---|\n| `spark-defaults.conf` (cluster-wide) | Fabric **Spark Workspace Settings** + **Environment** item | Move config properties to Environment or use `%%configure` in notebooks |\n| `%%configure` magic | `%%configure` magic — **identical** | No change needed |\n| YARN queue \u002F resource allocation | **Fabric Spark pool** node size and autoscale settings | Map queue SLAs to Custom Pool configuration |\n| Ambari service configs (HDFS, YARN tuning) | Not applicable — Fabric manages infrastructure | Remove; focus on application-level Spark configs |\n| HDI Spark version (e.g., Spark 2.4) | Fabric Runtime 1.3 = Spark 3.5 (latest) | Test for deprecated API removals (e.g., `HiveContext`, RDD-style ML) |\n| Conda environment \u002F `bootstrap.sh` | **Fabric Environment** item with custom libraries | Recreate conda\u002Fpip dependencies in a Fabric Environment |\n| `hive-site.xml` (metastore connection) | Not needed — Delta Lake IS the metastore in Fabric | Remove metastore config; use Lakehouse schemas for namespace organization |\n\n---\n\n## Must \u002F Prefer \u002F Avoid\n\n### MUST DO\n- **Replace all `wasb:\u002F\u002F` \u002F `wasbs:\u002F\u002F` paths** with OneLake `abfss:\u002F\u002F` paths or OneLake Shortcuts — `wasb:\u002F\u002F` requires storage account keys which are not the Fabric-preferred auth model\n- **Replace `HiveContext`, `SQLContext`, and standalone `SparkContext()`** — use the pre-instantiated `spark` session in Fabric notebooks\n- **Migrate Hive DDL** (`STORED AS ORC`, `LOCATION`, `TBLPROPERTIES`) to Delta Lake DDL — see [hive-to-delta.md](resources\u002Fhive-to-delta.md)\n- **Introduce `notebookutils`** for file system operations, secret retrieval, and child notebook orchestration where HDInsight used custom scripts or direct API calls\n- **Replace Oozie XML workflows** with Fabric Data Pipelines — see [§ Oozie → Fabric Pipelines](#oozie--fabric-pipelines)\n- **Align library management** to Fabric Environments — remove `bootstrap.sh`, conda envs, and runtime `%pip install` patterns for production workloads\n- **Treat `hdfs:\u002F\u002F` as source-cluster-only** — export or ingest into ADLS Gen2 or OneLake; do not claim that a gateway adds Kerberos support to Fabric's Anonymous-only HDFS connector\n\n### PREFER\n- **OneLake Shortcuts** over copying data — mount existing ADLS Gen2 containers as shortcuts to avoid re-ingestion during migration\n- **Delta Lake** for all tables migrated from Hive ORC\u002FParquet — ACID guarantees, time travel, and schema enforcement improve data quality\n- **Fabric Starter Pool** for initial migration validation — no pool configuration overhead, fast session startup\n- **Lakehouse schemas** (database namespaces) for organizing migrated Hive databases — one schema per Hive database within a single Lakehouse\n- **Medallion architecture** for restructuring migrated data layers during migration — align Bronze\u002FSilver\u002FGold with raw Hive → validated Delta → serving Gold patterns\n\n### AVOID\n- **Do not use `SparkContext()` or `HiveContext()` constructors** in Fabric notebooks — they conflict with the pre-instantiated `spark` session and will raise errors\n- **Do not use `hive-site.xml` or external Hive metastore configuration** — Fabric's Delta Lake-backed Lakehouse IS the metastore\n- **Do not assume YARN queue mappings translate to Fabric pools** — re-design resource allocation based on Fabric Spark pool SLAs\n- **Do not attempt to run Oozie shell actions or Java MapReduce jobs** directly in Fabric — these must be refactored (see [§ Oozie → Fabric Pipelines](#oozie--fabric-pipelines))\n- **Do not use `%sh` magic for file system operations** in production notebooks — use `notebookutils.fs.*` for portability and OneLake token-based auth\n\n---\n\n## Examples\n\nSee [code-patterns.md](resources\u002Fcode-patterns.md) for full before\u002Fafter examples. Key quick references:\n\n**Legacy context → Fabric pre-instantiated session**\n\n```python\n# HDInsight (remove entirely)\nfrom pyspark.sql import HiveContext\nhc = HiveContext(sc)\n\n# Fabric — use pre-instantiated spark directly\ndf = spark.sql(\"SELECT * FROM sales.fact_orders\")\n```\n\n**WASB path → OneLake path (after shortcut creation)**\n\n```python\n# HDInsight\ndf = spark.read.parquet(\"wasb:\u002F\u002Fraw@myaccount.blob.core.windows.net\u002Forders\u002F\")\n\n# Fabric\ndf = spark.read.parquet(\"Files\u002Fraw\u002Forders\u002F\")\n```\n\n**Hive DDL → Delta DDL**\n\n```sql\n-- HDInsight\nCREATE TABLE sales_db.fact_orders (...) STORED AS ORC LOCATION 'wasb:\u002F\u002F...';\n\n-- Fabric\nCREATE SCHEMA IF NOT EXISTS sales_db;\nCREATE TABLE sales_db.fact_orders (...) USING DELTA;\n```\n",{"name":5,"description":7},"main","4c3986d2e5b498ce2d6e6025533b94b5e5bb2ccd","skills\u002Fhdinsight-migration",395,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric","https:\u002F\u002Fgithub.com\u002Fmicrosoft","https:\u002F\u002Fopensource.microsoft.com","Open source projects and samples from Microsoft","2026-11-12 20:10:24",{"source":8,"skillName":5,"commands":35,"prompt":45,"zipAvailable":46},[36,39,42],{"tool":37,"command":38},"npx","npx skills add microsoft\u002Fskills-for-fabric --skill hdinsight-migration",{"tool":40,"command":41},"bunx","bunx skills add microsoft\u002Fskills-for-fabric --skill hdinsight-migration",{"tool":43,"command":44},"pnpm","pnpm dlx skills add microsoft\u002Fskills-for-fabric --skill hdinsight-migration","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Fblob\u002Fmain\u002Fskills\u002Fhdinsight-migration\u002FSKILL.md",true,[48,54,58,62,68,74,78,82,86,90],{"path":49,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":52,"sourceHtmlUrl":53},"anthropic-best-practices.md","REFERENCE","参考文档",45825,"https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002F_collections\u002Fdot_config\u002Fopencode\u002Fsuperpowers\u002Fskills\u002Fwriting-skills\u002Fanthropic-best-practices.md",{"path":55,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":56,"sourceHtmlUrl":57},"apm.yml",459,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Fblob\u002Fmain\u002Fskills\u002Fhdinsight-migration\u002Fapm.yml",{"path":59,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":60,"sourceHtmlUrl":61},"examples\u002FCLAUDE_MD_TESTING.md",5423,"https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002F_collections\u002Fdot_config\u002Fopencode\u002Fsuperpowers\u002Fskills\u002Fwriting-skills\u002Fexamples\u002FCLAUDE_MD_TESTING.md",{"path":63,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":66,"sourceHtmlUrl":67},"executable_render-graphs.js","SCRIPT","脚本",4857,"https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002F_collections\u002Fdot_config\u002Fopencode\u002Fsuperpowers\u002Fskills\u002Fwriting-skills\u002Fexecutable_render-graphs.js",{"path":69,"resourceType":70,"resourceTypeDesc":71,"sizeBytes":72,"sourceHtmlUrl":73},"graphviz-conventions.dot","OTHER","其他",5970,"https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002F_collections\u002Fdot_config\u002Fopencode\u002Fsuperpowers\u002Fskills\u002Fwriting-skills\u002Fgraphviz-conventions.dot",{"path":75,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":76,"sourceHtmlUrl":77},"persuasion-principles.md",5908,"https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002F_collections\u002Fdot_config\u002Fopencode\u002Fsuperpowers\u002Fskills\u002Fwriting-skills\u002Fpersuasion-principles.md",{"path":79,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":80,"sourceHtmlUrl":81},"resources\u002Fcode-patterns.md",5557,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Fblob\u002Fmain\u002Fskills\u002Fhdinsight-migration\u002Fresources\u002Fcode-patterns.md",{"path":83,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":84,"sourceHtmlUrl":85},"resources\u002Fhive-to-delta.md",6680,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Fblob\u002Fmain\u002Fskills\u002Fhdinsight-migration\u002Fresources\u002Fhive-to-delta.md",{"path":87,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":88,"sourceHtmlUrl":89},"resources\u002Fpath-migration.md",6881,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Fblob\u002Fmain\u002Fskills\u002Fhdinsight-migration\u002Fresources\u002Fpath-migration.md",{"path":91,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":92,"sourceHtmlUrl":93},"testing-skills-with-subagents.md",12557,"https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002F_collections\u002Fdot_config\u002Fopencode\u002Fsuperpowers\u002Fskills\u002Fwriting-skills\u002Ftesting-skills-with-subagents.md",[95,103,108,134,163,168],{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":102},"01M78JNNY5VN2MZ7TSV040K3FJ","fabric-policies-cli","Author, review and manage Microsoft Fabric governance policies and PolicySet items. Invoke before clarifying underspecified lockdown requests or missing intent, identities or scope. Discover current contracts and roles, map scenarios to rules, and execute one approved API lifecycle. Not catalog audits or KQL policies. Triggers: Fabric policy, PolicySet, governance rule, latest Fabric policies, policy contract review.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Fblob\u002Fmain\u002Fskills\u002Ffabric-policies-cli\u002FSKILL.md",[],[],"2026-11-15 14:19:17",{"skillNo":104,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":105,"categories":106,"keywords":107,"publishedAt":102},"01M78JNBFP3GAR4MSKEMHY2HDE","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Fblob\u002Fmain\u002Fplugins\u002Ffabric-skills\u002Fskills\u002Ffabric-policies-cli\u002FSKILL.md",[],[],{"skillNo":109,"name":110,"nameZh":6,"description":111,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":112,"categories":113,"keywords":114,"publishedAt":20},"01M6PHZNVB6K7DCCJH4YZHKC4F","ontology-cli","Manages Microsoft Fabric Ontology items: authors legacy V1 entity\u002Frelationship definitions and answers questions for new-experience (V2) Ontologies through ask_ontology. Use for Ontology item operations or grounded questions; V2 authoring\u002Fupgrades and V1 queries are unsupported. Power BI data questions use fabriciq, report authoring powerbi-report-cli, and model\u002FDAX changes semantic-model-authoring.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Fblob\u002Fmain\u002Fskills\u002Fontology-cli\u002FSKILL.md",[],[115,116,117,118,9,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133],"cli","ontology","ontology cli","ask","operations","bi","microsoft fabric","changes","queries","power bi","relationship","fabriciq","v1","Legacy","use","upgrades","for","model","power",{"skillNo":135,"name":136,"nameZh":6,"description":137,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":138,"categories":139,"keywords":145,"publishedAt":20},"01M6PHZ91CSTP67DM6JN8TYEFV","databasehub-cli","Analyzes Database Hub inventory, health, and security posture through Entra-authenticated Fabric APIs. Use for estate counts; SQL\u002FPostgreSQL CPU, storage, and memory; Cosmos DB availability\u002FRU; authentication\u002Fauditing\u002FCMK warnings; and Fabric SQL inventory. Fabric SQL tuning routes to sqldb-cli; Warehouse\u002FLakehouse\u002FMirrored Database SQL to sqldw-cli. Triggers: data estate performance, database estate health, database security posture.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Fblob\u002Fmain\u002Fskills\u002Fdatabasehub-cli\u002FSKILL.md",[140],{"categoryNo":141,"categoryName":142,"taxonomyType":143,"taxonomyTypeDesc":144},"01JBSEEDCAT000000000000010","后端与接口","SCENARIO","使用场景",[115,146,147,148,149,150,151,152,153,131,129,154,155,156,157,158,159,160,161,162],"databasehub cli","database","CPU","database sql","security posture","postgresql","health","database security","triggers","cosmos","authentication","inventory","performance","db","memory","availability","warehouse",{"skillNo":164,"name":110,"nameZh":6,"description":111,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":165,"categories":166,"keywords":167,"publishedAt":20},"01M6PHYVDVBD0VQN4FFAMN428Z","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Fblob\u002Fmain\u002Fplugins\u002Ffabric-skills\u002Fskills\u002Fontology-cli\u002FSKILL.md",[],[115,116,117,131,132,133,126,130,122,9,119,120,121,123,124,125,118,127,128,129],{"skillNo":169,"name":136,"nameZh":6,"description":137,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":170,"categories":171,"keywords":173,"publishedAt":20},"01M6PHYDZ17VY69E3GS71KFW09","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills-for-fabric\u002Fblob\u002Fmain\u002Fplugins\u002Ffabric-skills\u002Fskills\u002Fdatabasehub-cli\u002FSKILL.md",[172],{"categoryNo":141,"categoryName":142,"taxonomyType":143,"taxonomyTypeDesc":144},[115,146,162,161,149,159,158,157,148,150,151,152,153,147,129,154,155,160,131,156]]