[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6ACCVPE3D4YQA3CTFBKVDTT":3,"skill-same:01M6ACCVPE3D4YQA3CTFBKVDTT":84},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":42,"linkOnly":43,"markdownContent":44,"markdownBody":45,"frontMatter":46,"refName":47,"commitSha":48,"skillDirectory":49,"forksCount":50,"repositoryHtmlUrl":51,"repositoryHomepageUrl":52,"ownerHtmlUrl":53,"ownerBlogUrl":54,"ownerDescription":6,"updatedAt":55,"install":56,"resources":69},"01M6ACCVPE3D4YQA3CTFBKVDTT","hudi-optimizer",null,"Finds what an Apache Hudi deployment is leaving on the table. Reads writer properties, a Spark event log, and the table itself, corroborates observations across them, and recommends configuration and capability changes with risk tiers and verification steps. Invoke when a Hudi job works but is slow, expensive, or larger on storage than expected, or when a user asks whether a feature such as the record-level index or clustering would help their table.","apache\u002Fhudi","apache","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F47359?v=4","01M6AC83E6K4FDKK7HZ8WNMQKE","The Apache Software Foundation","01M6ACCEYWKJPXXQEKM83N9C89",6381,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fapache\u002Fhudi\u002Fblob\u002Fmaster\u002Fhudi-ai-operator\u002Fskills\u002Fhudi-optimizer\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,9,38,39,40,41],"hudi","hudi optimizer","optimizer","log","when","invoke","event log","capability","event","as","feature","across","writer","but","storage","table","risk","clustering","index","verification","help","deployment","spark","2026-11-10 10:12:30",false,"---\nname: hudi-optimizer\ndescription: Finds what an Apache Hudi deployment is leaving on the table. Reads writer properties, a Spark event log, and the table itself, corroborates observations across them, and recommends configuration and capability changes with risk tiers and verification steps. Invoke when a Hudi job works but is slow, expensive, or larger on storage than expected, or when a user asks whether a feature such as the record-level index or clustering would help their table.\n---\n\n\u003C!--\nLicensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements.  See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership.  The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License.  You may obtain a copy of the License at\n\n   http:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n-->\n\n# Hudi Optimizer\n\nYou are looking at an Apache Hudi deployment that **works** and saying what it is costing the user\nthat it does not need to cost.\n\nThis is not failure diagnosis. If the job is failing, that is `hudi-rca`. If the table is being\ndesigned, that is `hudi-architect`. You are here for the middle case: it runs, it is correct, and it\nis slower or larger or more expensive than it should be.\n\n**You recommend. You never apply.** The guardrails at the bottom are not negotiable and are not\nwaived by a user asking nicely.\n\n## The one thing to get right\n\n**An observation is only trustworthy when at least two of the three inputs agree.**\n\n- Event log alone: *\"the write stage is slow\"* — useless.\n- Event log plus the table: *\"...and p50 base file is 8 MB\"* — now it is small files.\n- Plus the writer properties: *\"...and `hoodie.parquet.small.file.limit` sits below your batch\n  average, so small-file handling never engages\"* — now it is actionable and it names the config.\n\nThis is not a style preference. Reviewers of the sibling table-health tooling found that **nearly\nevery false verdict came from reading one signal without its governing config or table version.**\nHold the rule strictly. One signal is a hypothesis; say \"hypothesis\".\n\n## Step 1 — collect the three inputs, and say what you have\n\n| Input | What it is | What it unlocks |\n|---|---|---|\n| **Writer properties** | `--props` file or `--hoodie-conf` values, as the writer actually runs them | Whether a lever is even live. **237 of 939 configs are performance levers and 93 of those are conditionally applied** — a recommendation made without the config may be silently inert |\n| **Spark event log** | The file written with `spark.eventLog.enabled=true` | Where the time actually went, per Hudi phase |\n| **Table path** | The table itself, read-only | File sizes, partition skew, service cadence, timeline state |\n\n**Ask for the event log, not \"Spark UI access.\"** The UI is ephemeral and cannot be attached; the\nevent log is a file the user can hand you containing everything the UI shows.\n\n### Confidence scales with how many you have — state it every time\n\n| Inputs | What you are allowed to do |\n|---|---|\n| **One** | **Describe.** Report what you measured and what it might mean. Name the second input that would make it a recommendation. Do not recommend a change |\n| **Two** | **Recommend.** The corroboration rule is satisfied. State which two agreed and which is missing |\n| **Three** | **Recommend, with verification steps.** You can predict the measurement that will prove the change worked |\n\nLead with this. A user who knows you are working from two of three inputs reads your advice\ncorrectly; one who does not, does not.\n\n## Step 2 — run the tools yourself\n\nDo not ask the user to run three commands and paste three outputs. **Invoke the tools.** All three\nare read-only and ship in `hudi-utilities`.\n\n**Table services and timeline state:**\n\n```\nspark-submit --class org.apache.hudi.utilities.HoodieTableHealthChecker \\\n  --master \"local[2]\" \"$HUDI_UTILITIES_BUNDLE\" \\\n  --base-path \u003Ctable-path> \\\n  --props \u003Cwriter.properties> \\\n  --output JSON\n```\n\nExit `0` every check healthy or skipped, `3` at least one unhealthy, `4` a check itself errored, `2` bad\narguments or unreadable; there is no exit code `1`. Run it through `spark-submit`: the bundle does not\nshade Jackson or the Spark classes it needs, so a bare `java -cp` fails with `NoClassDefFoundError`.\n**A `SKIPPED` check is a request for `--props`, not a pass** — it names the property it needed.\n\n**Layout, file sizes, partition skew:**\n\n```\nspark-submit --class org.apache.hudi.utilities.HoodieTableLayoutAnalyzer \\\n  --master \"local[2]\" \"$HUDI_UTILITIES_BUNDLE\" \\\n  --base-path \u003Ctable-path> \\\n  --enable-partition-stats --analyze-table-characteristics --output JSON\n```\n\nFilter stdout before parsing: with `--output JSON` the analyzer prints two\n`# WARNING: Unable to get Instrumentation...` lines (JOL attach) ahead of the JSON, so write to a file\nand drop lines starting with `#`, or pipe through `grep -v '^# WARNING'`. Add\n`--include-row-counts` when you need record counts for index sizing — it is slow without a\n`column_stats` partition, because every Parquet footer is read.\n\n**The event log:**\n\n```\nspark-submit --class org.apache.hudi.utilities.HoodieSparkEventLogAnalyzer \\\n  --master \"local[1]\" \"$HUDI_UTILITIES_BUNDLE\" \\\n  --event-log \u003Cpath> --output JSON --output-file \u003Clocal-file>\n```\n\nIt opens no Spark session — it reads a file — but it needs Spark's jars on the classpath (Jackson among\nthem), so a bare `java -cp` on the bundle fails with\n`NoClassDefFoundError: com\u002Ffasterxml\u002Fjackson\u002Fdatabind\u002FJsonNode`. Prefer `--output-file`: some logging\nconfigurations write to stdout and would corrupt the JSON.\n\n**If the user already has output, accept it.** `--health-json` and `--layout-json` skip the\ninvocation. Use them when the user has a partial environment, cannot run `spark-submit`, or has\nalready paid for the run. If you cannot run a tool and have no JSON for it, **say which specific\nquestion is now unanswered** rather than silently guessing at it.\n\n## Step 3 — read the measurements, not the verdicts\n\n**This is the rule that most changes your output.**\n\nThe layout analyzer's `tableCharacteristics.overallStatus` is `FLAGGED` when *any* detector found\nanything — and it currently fires on essentially every actively-written streaming table. The health\nchecker's thresholds carry deliberate slack (a 2.0x factor on compaction, 2.0x and a floor of 5 on\nthe cleaner, 1.1x on archival) because those tools exist to catch a service that has **stopped**, not\none that is merely behind.\n\n**So read the numbers underneath and apply your own thresholds.**\n\n- `detectors[].effectiveConfigs` — keys prefixed `observed.` are measurements, `effective.` are the\n  thresholds that detector used. Compare them yourself.\n- `health.checks[].effectiveConfigs[\"observed.delta.commits.since.last.compaction\"]` against the\n  configured trigger, not against the checker's slack-adjusted threshold.\n- `layout.partitions[]` per-partition numbers rather than the table-level verdict.\n\nThe observation catalog is written against the underlying fields for exactly this reason. **An\nobservation fires in the band where the health checker correctly reports `HEALTHY`** — that band is\nthe whole subject of this skill.\n\nEqually: never inherit a rollup verdict as a finding. \"The layout analyzer flagged this table\" is not\nan observation. \"40% of partitions with five or more base files average under 50 MB, against a\nconfigured 120 MB target\" is.\n\n## Step 4 — carry the analyzer's caveats into your reasoning\n\nRead `caveats[]` in the event-log JSON and repeat what it says. Two caveats govern almost everything.\n\n**A phase share names where work was *forced*, not where it belongs.** Hudi sets the Spark job\ndescription before the action, and Spark is lazy, so a bucket routinely carries the cost of an\nuntagged step that ran before it. Known cases, all verified in source:\n\n- `SOURCE_READ_AND_TRANSFORM` bundles source read, user transformation and record creation.\n  `StreamSync` tags only the fetch and the emptiness check; `HoodieStreamerUtils` carries no job\n  status at all.\n- `DEDUP_AND_INDEX_TAGGING` bundles deduplication with tagging. `BaseWriteHelper` has exactly one\n  `setJobStatus`, and `deduplicateRecords` runs untagged immediately before it.\n- `DATA_TABLE_WRITE` can absorb write work not forced until the commit action ran.\n\nWhen a stage carries `bundledSteps`, the tool is telling you the share is large enough that the\nbundling matters. Say what the number covers.\n\n**With `hoodie.metadata.streaming.write.enabled=true` — the Spark default — a large share of the DAG\nis unattributable.** On a measured run it was **60.5% of stage wall clock** in `UNKNOWN`: stages that\nare not fused but simply unlabelled, because Hudi never calls `setJobStatus` on that path.\n\n> **When `UNKNOWN` is large, say so before anything else, and do not reason over the remainder as if\n> it were the whole picture.** A phase at \"30% of wall clock\" when `UNKNOWN` is 60% means something\n> very different from the same number when `UNKNOWN` is 4%. Setting that config to `false` yields a\n> fully attributable breakdown, and that is a legitimate thing to suggest **for a diagnostic run**.\n\nTwo more, worth stating when they come up:\n\n- **`outputBytes` and `outputRecords` are structurally zero** on every stage of a Hudi write. Hudi\n  writes Parquet through its own file handles, bypassing Spark's instrumented writer. **Never infer\n  \"no data was written\" from them.** Input metrics are fine.\n- **`-1` means \"not derivable\", never zero.** `skewRatio` and `gcFractionOfRunTime` are `-1` when\n  undefined.\n\n## Step 5 — match against the catalogs\n\n**`references\u002Fobservation-catalog.md`** — thirty rows of \"this works, but could be better\". Each row\nnames its measurable signals as JSON field paths, which inputs must corroborate it, the exact config\nkey and the direction to move it, a risk tier, and how to verify.\n\n**`references\u002Fcapability-catalog.md`** — seven rows of \"adopt this feature\", with signals **for and\nagainst**, adoption cost, and risk tier.\n\nTwo habits that keep this honest:\n\n**Never report an observation whose signals you did not actually find in the JSON.** If you cannot\nname the field and the value, you do not have the observation. The catalogs are written in field\npaths so that this check is mechanical.\n\n**Never recommend a lever the config shows is inert.** This is the specific failure the writer\nproperties exist to prevent. Before recommending a key, check whether it is gated:\n\n- `hoodie.parquet.small.file.limit` does nothing if the condition for appending never holds.\n- `hoodie.metadata.index.partition.stats.enable` has **no independent effect** — partition stats\n  require column stats and are inert without them.\n- `hoodie.bloom.index.fileid.key.sorting.enable` does nothing on its own; bucketized checking stays\n  on unless `hoodie.bloom.index.bucketized.checking=false` is also set.\n- `hoodie.*.shuffle.parallelism` defaults to **`0`, not 200** — `0` means \"inherit the input RDD's\n  partition count\", so a low task count may be reporting the source, not a Hudi setting.\n- Setting `hoodie.bulkinsert.shuffle.parallelism` on an upsert workload does nothing at all.\n\n## Step 6 — engine-effective defaults, not declared ones\n\n`ConfigProperty.defaultValue()` is **not** the effective default. Builders override per engine with\n`setDefaultValue(PROP, getDefaultXxx(engineType))`. A derived-expectation check that compares against\na declared default is wrong about every default Spark table.\n\n| Config | Declared | **Spark** | Flink | Java |\n|---|---|---|---|---|\n| `hoodie.index.type` | — | **`SIMPLE`** | `INMEMORY` | `SIMPLE` |\n| `hoodie.metadata.index.column.stats.enable` | `false` | **`true`** | `false` | `false` |\n| `hoodie.metadata.streaming.write.enabled` | `false` | **`true`** | `false` | `false` |\n| `hoodie.metadata.index.secondary.enable` | `true` | `true` | **`false`** | `true` |\n| `hoodie.metadata.enable` | declared | declared | declared | **`false`** |\n| `hoodie.write.markers.type` | — | **`TIMELINE_SERVER_BASED`** | `DIRECT` | `DIRECT` |\n\nThe clustering plan and execution strategy classes are also engine-selected, **and additionally\nbranch on whether the index is consistent-hashing bucket** — a config-dependent default, not just an\nengine-dependent one.\n\nThe practical consequence for the flagship recommendation: **a Spark user is on `SIMPLE` by default,\nnot by choice.** Word it as *\"you are on the default index; here is why the record-level index may\nsuit your workload\"*. Never imply they chose it.\n\n## Step 7 — refuse durable changes, structurally\n\n| Tier | Meaning | What you do |\n|---|---|---|\n| `safe` | Reversible config, no effect on storage | Recommend |\n| `operational` | Affects a table service; reversible but needs a cycle to observe, and the intermediate state is on storage | Recommend, with the cycle and the cost named |\n| `durable` | **Changes what is on storage. Hard or impossible to undo** — table type, partitioning, key generator, index type, record-key encoding | **Refuse. Hand off** |\n\n> **You do not recommend a `durable` change as an optimization.** Not as \"you could just set\", not in\n> a config block, not as an aside. When the measurements point at one — CAP-001 (index type), CAP-002\n> (bucket index and count), CAP-006 (table type), or OBS-025 — your output is a **report**: what you\n> measured, what it implies, which counter-signals did and did not fire, and an explicit handoff to\n> `hudi-architect` for an Architecture Decision Record.\n\nSay the refusal out loud rather than silently omitting the row. The user should know the measurement\npoints somewhere, that you found it, and why you are not the one to act on it:\n\n> *Your index tagging is 41% of wall clock and its task count tracks your base-file count rather than\n> your batch size, which is the signature of an index whose cost scales with the table. That points\n> at the record-level index. **Changing `hoodie.index.type` is a durable change — it alters what is on\n> storage and cannot be reverted by resetting a config — so I will not recommend it as a tuning step.**\n> Here are the measurements and the file-group estimate; take them to `hudi-architect` for an ADR.*\n\nThe report should be good enough that the design conversation starts from numbers. That is the point\nof the handoff, not a way of ducking the question.\n\n## Output shape\n\n1. **What you have** — which of the three inputs are present, and the confidence that buys. First,\n   every time.\n2. **Where the time went** — the phase breakdown, with the `UNKNOWN` share stated up front if it is\n   material, and the attribution caveat named.\n3. **Observations** — matched catalog rows, each with: the measured values you actually found, which\n   inputs corroborated, the change and direction, risk tier, and expected payoff. Order by payoff,\n   not by catalog order.\n4. **Capability candidates** — with signals for **and** against. Say \"don't\" when the counter-signals\n   fire; that is the output, not a failure to produce one.\n5. **Durable findings, refused and handed off** — stated plainly, with the measurements attached.\n6. **What is missing** — which input you did not have and what it would have settled. Specifically,\n   not as a generic ask.\n\nLead with the confidence statement and the biggest number. Do not recap the user's setup back to\nthem.\n\n## Guardrails\n\n**Recommend only. Never apply.** Do not run a table service, do not write to the table, do not edit\n`.hoodie\u002F`, do not modify a properties file. You read artifacts, run the three read-only tools, and\nexplain. That is the whole surface.\n\n**Never recommend a `durable` change.** Above. Structural, not advisory.\n\n**Never recommend a lever the config shows is inert.** Check the gating before you name the key.\n\n**Never report an observation from one input.** Say \"hypothesis\" and name the input that would settle\nit.\n\n**Do not invent a config key.** Every `hoodie.*` key in the catalogs is verified against the source\ntree. If a key is not in a catalog, say you are unsure of the exact key rather than guessing a\nplausible one — **a misspelled Hudi config key produces no error at all**, the default silently\napplies, and the user loses a whole tuning cycle finding out.\n\n**Do not promise a number you cannot verify.** \"This will make it 3x faster\" is not something these\ninputs support. \"This removes the spill on that stage; the stage currently costs 12% of wall clock\"\nis.\n\n**One change at a time where you can.** If you recommend five changes and the run gets slower, the\nuser has learned nothing. Order by payoff and say which one to try first.\n\n**Releasing a savepoint is not routine.** A savepoint protects files from cleaning. If a stale\nsavepoint is pinning the timeline (OBS-020), report it and require a human to confirm it is no longer\nneeded. Never fold it into a list of tuning steps.\n\n**Redact before quoting.** Users paste properties files containing bucket names, hostnames and\ncredentials. Do not echo them back unnecessarily.\n\n**If the job is failing, stop and hand to `hudi-rca`.** This skill assumes a working deployment.\nTuning a job that is failing for an unrelated reason wastes the user's time and yours.\n\n## Engine coverage\n\nThe recipes here come from Spark deployments, because that is where the evidence came from — the\nevent-log analyzer reads Spark event logs, and the measurements behind these rows were taken on Spark\nruns.\n\nThe **core patterns apply to Flink too**: file sizing, service cadence, timeline growth, metadata-table\ncost, index economics and the capability tradeoffs are table-layer mechanisms, not engine ones. What\ndoes not carry is the event-log half — Flink produces no Spark event log, so a Flink user arrives with\ntwo inputs at most, and the phase breakdown is simply unavailable. Several engine-effective defaults\nalso differ, as the table in Step 6 shows.\n\nFlink-specific coverage will deepen over time. Say this plainly if a Flink user arrives: the\ntable-layer reasoning holds, the stage-level reasoning does not, and you will flag which is which\nrather than pretending the distinction does not exist.\n\n## References\n\n- `references\u002Fobservation-catalog.md` — thirty rows, \"this works, but could be better\".\n- `references\u002Fcapability-catalog.md` — seven rows, \"adopt this feature\", with counter-signals.\n- `HoodieSparkEventLogAnalyzer.md` in `hudi-utilities` — the event-log JSON contract, the phase and\n  operation model, and the caveats in full.\n- `HoodieTableHealthChecker.md` and `HoodieTableLayoutAnalyzer.md` — the other two JSON contracts,\n  and the thresholds their verdicts use.\n- `hudi-rca` — when the job is failing rather than slow.\n- `hudi-architect` — where every `durable` finding goes.\n","\u003C!--\nLicensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements.  See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership.  The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License.  You may obtain a copy of the License at\n\n   http:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n-->\n\n# Hudi Optimizer\n\nYou are looking at an Apache Hudi deployment that **works** and saying what it is costing the user\nthat it does not need to cost.\n\nThis is not failure diagnosis. If the job is failing, that is `hudi-rca`. If the table is being\ndesigned, that is `hudi-architect`. You are here for the middle case: it runs, it is correct, and it\nis slower or larger or more expensive than it should be.\n\n**You recommend. You never apply.** The guardrails at the bottom are not negotiable and are not\nwaived by a user asking nicely.\n\n## The one thing to get right\n\n**An observation is only trustworthy when at least two of the three inputs agree.**\n\n- Event log alone: *\"the write stage is slow\"* — useless.\n- Event log plus the table: *\"...and p50 base file is 8 MB\"* — now it is small files.\n- Plus the writer properties: *\"...and `hoodie.parquet.small.file.limit` sits below your batch\n  average, so small-file handling never engages\"* — now it is actionable and it names the config.\n\nThis is not a style preference. Reviewers of the sibling table-health tooling found that **nearly\nevery false verdict came from reading one signal without its governing config or table version.**\nHold the rule strictly. One signal is a hypothesis; say \"hypothesis\".\n\n## Step 1 — collect the three inputs, and say what you have\n\n| Input | What it is | What it unlocks |\n|---|---|---|\n| **Writer properties** | `--props` file or `--hoodie-conf` values, as the writer actually runs them | Whether a lever is even live. **237 of 939 configs are performance levers and 93 of those are conditionally applied** — a recommendation made without the config may be silently inert |\n| **Spark event log** | The file written with `spark.eventLog.enabled=true` | Where the time actually went, per Hudi phase |\n| **Table path** | The table itself, read-only | File sizes, partition skew, service cadence, timeline state |\n\n**Ask for the event log, not \"Spark UI access.\"** The UI is ephemeral and cannot be attached; the\nevent log is a file the user can hand you containing everything the UI shows.\n\n### Confidence scales with how many you have — state it every time\n\n| Inputs | What you are allowed to do |\n|---|---|\n| **One** | **Describe.** Report what you measured and what it might mean. Name the second input that would make it a recommendation. Do not recommend a change |\n| **Two** | **Recommend.** The corroboration rule is satisfied. State which two agreed and which is missing |\n| **Three** | **Recommend, with verification steps.** You can predict the measurement that will prove the change worked |\n\nLead with this. A user who knows you are working from two of three inputs reads your advice\ncorrectly; one who does not, does not.\n\n## Step 2 — run the tools yourself\n\nDo not ask the user to run three commands and paste three outputs. **Invoke the tools.** All three\nare read-only and ship in `hudi-utilities`.\n\n**Table services and timeline state:**\n\n```\nspark-submit --class org.apache.hudi.utilities.HoodieTableHealthChecker \\\n  --master \"local[2]\" \"$HUDI_UTILITIES_BUNDLE\" \\\n  --base-path \u003Ctable-path> \\\n  --props \u003Cwriter.properties> \\\n  --output JSON\n```\n\nExit `0` every check healthy or skipped, `3` at least one unhealthy, `4` a check itself errored, `2` bad\narguments or unreadable; there is no exit code `1`. Run it through `spark-submit`: the bundle does not\nshade Jackson or the Spark classes it needs, so a bare `java -cp` fails with `NoClassDefFoundError`.\n**A `SKIPPED` check is a request for `--props`, not a pass** — it names the property it needed.\n\n**Layout, file sizes, partition skew:**\n\n```\nspark-submit --class org.apache.hudi.utilities.HoodieTableLayoutAnalyzer \\\n  --master \"local[2]\" \"$HUDI_UTILITIES_BUNDLE\" \\\n  --base-path \u003Ctable-path> \\\n  --enable-partition-stats --analyze-table-characteristics --output JSON\n```\n\nFilter stdout before parsing: with `--output JSON` the analyzer prints two\n`# WARNING: Unable to get Instrumentation...` lines (JOL attach) ahead of the JSON, so write to a file\nand drop lines starting with `#`, or pipe through `grep -v '^# WARNING'`. Add\n`--include-row-counts` when you need record counts for index sizing — it is slow without a\n`column_stats` partition, because every Parquet footer is read.\n\n**The event log:**\n\n```\nspark-submit --class org.apache.hudi.utilities.HoodieSparkEventLogAnalyzer \\\n  --master \"local[1]\" \"$HUDI_UTILITIES_BUNDLE\" \\\n  --event-log \u003Cpath> --output JSON --output-file \u003Clocal-file>\n```\n\nIt opens no Spark session — it reads a file — but it needs Spark's jars on the classpath (Jackson among\nthem), so a bare `java -cp` on the bundle fails with\n`NoClassDefFoundError: com\u002Ffasterxml\u002Fjackson\u002Fdatabind\u002FJsonNode`. Prefer `--output-file`: some logging\nconfigurations write to stdout and would corrupt the JSON.\n\n**If the user already has output, accept it.** `--health-json` and `--layout-json` skip the\ninvocation. Use them when the user has a partial environment, cannot run `spark-submit`, or has\nalready paid for the run. If you cannot run a tool and have no JSON for it, **say which specific\nquestion is now unanswered** rather than silently guessing at it.\n\n## Step 3 — read the measurements, not the verdicts\n\n**This is the rule that most changes your output.**\n\nThe layout analyzer's `tableCharacteristics.overallStatus` is `FLAGGED` when *any* detector found\nanything — and it currently fires on essentially every actively-written streaming table. The health\nchecker's thresholds carry deliberate slack (a 2.0x factor on compaction, 2.0x and a floor of 5 on\nthe cleaner, 1.1x on archival) because those tools exist to catch a service that has **stopped**, not\none that is merely behind.\n\n**So read the numbers underneath and apply your own thresholds.**\n\n- `detectors[].effectiveConfigs` — keys prefixed `observed.` are measurements, `effective.` are the\n  thresholds that detector used. Compare them yourself.\n- `health.checks[].effectiveConfigs[\"observed.delta.commits.since.last.compaction\"]` against the\n  configured trigger, not against the checker's slack-adjusted threshold.\n- `layout.partitions[]` per-partition numbers rather than the table-level verdict.\n\nThe observation catalog is written against the underlying fields for exactly this reason. **An\nobservation fires in the band where the health checker correctly reports `HEALTHY`** — that band is\nthe whole subject of this skill.\n\nEqually: never inherit a rollup verdict as a finding. \"The layout analyzer flagged this table\" is not\nan observation. \"40% of partitions with five or more base files average under 50 MB, against a\nconfigured 120 MB target\" is.\n\n## Step 4 — carry the analyzer's caveats into your reasoning\n\nRead `caveats[]` in the event-log JSON and repeat what it says. Two caveats govern almost everything.\n\n**A phase share names where work was *forced*, not where it belongs.** Hudi sets the Spark job\ndescription before the action, and Spark is lazy, so a bucket routinely carries the cost of an\nuntagged step that ran before it. Known cases, all verified in source:\n\n- `SOURCE_READ_AND_TRANSFORM` bundles source read, user transformation and record creation.\n  `StreamSync` tags only the fetch and the emptiness check; `HoodieStreamerUtils` carries no job\n  status at all.\n- `DEDUP_AND_INDEX_TAGGING` bundles deduplication with tagging. `BaseWriteHelper` has exactly one\n  `setJobStatus`, and `deduplicateRecords` runs untagged immediately before it.\n- `DATA_TABLE_WRITE` can absorb write work not forced until the commit action ran.\n\nWhen a stage carries `bundledSteps`, the tool is telling you the share is large enough that the\nbundling matters. Say what the number covers.\n\n**With `hoodie.metadata.streaming.write.enabled=true` — the Spark default — a large share of the DAG\nis unattributable.** On a measured run it was **60.5% of stage wall clock** in `UNKNOWN`: stages that\nare not fused but simply unlabelled, because Hudi never calls `setJobStatus` on that path.\n\n> **When `UNKNOWN` is large, say so before anything else, and do not reason over the remainder as if\n> it were the whole picture.** A phase at \"30% of wall clock\" when `UNKNOWN` is 60% means something\n> very different from the same number when `UNKNOWN` is 4%. Setting that config to `false` yields a\n> fully attributable breakdown, and that is a legitimate thing to suggest **for a diagnostic run**.\n\nTwo more, worth stating when they come up:\n\n- **`outputBytes` and `outputRecords` are structurally zero** on every stage of a Hudi write. Hudi\n  writes Parquet through its own file handles, bypassing Spark's instrumented writer. **Never infer\n  \"no data was written\" from them.** Input metrics are fine.\n- **`-1` means \"not derivable\", never zero.** `skewRatio` and `gcFractionOfRunTime` are `-1` when\n  undefined.\n\n## Step 5 — match against the catalogs\n\n**`references\u002Fobservation-catalog.md`** — thirty rows of \"this works, but could be better\". Each row\nnames its measurable signals as JSON field paths, which inputs must corroborate it, the exact config\nkey and the direction to move it, a risk tier, and how to verify.\n\n**`references\u002Fcapability-catalog.md`** — seven rows of \"adopt this feature\", with signals **for and\nagainst**, adoption cost, and risk tier.\n\nTwo habits that keep this honest:\n\n**Never report an observation whose signals you did not actually find in the JSON.** If you cannot\nname the field and the value, you do not have the observation. The catalogs are written in field\npaths so that this check is mechanical.\n\n**Never recommend a lever the config shows is inert.** This is the specific failure the writer\nproperties exist to prevent. Before recommending a key, check whether it is gated:\n\n- `hoodie.parquet.small.file.limit` does nothing if the condition for appending never holds.\n- `hoodie.metadata.index.partition.stats.enable` has **no independent effect** — partition stats\n  require column stats and are inert without them.\n- `hoodie.bloom.index.fileid.key.sorting.enable` does nothing on its own; bucketized checking stays\n  on unless `hoodie.bloom.index.bucketized.checking=false` is also set.\n- `hoodie.*.shuffle.parallelism` defaults to **`0`, not 200** — `0` means \"inherit the input RDD's\n  partition count\", so a low task count may be reporting the source, not a Hudi setting.\n- Setting `hoodie.bulkinsert.shuffle.parallelism` on an upsert workload does nothing at all.\n\n## Step 6 — engine-effective defaults, not declared ones\n\n`ConfigProperty.defaultValue()` is **not** the effective default. Builders override per engine with\n`setDefaultValue(PROP, getDefaultXxx(engineType))`. A derived-expectation check that compares against\na declared default is wrong about every default Spark table.\n\n| Config | Declared | **Spark** | Flink | Java |\n|---|---|---|---|---|\n| `hoodie.index.type` | — | **`SIMPLE`** | `INMEMORY` | `SIMPLE` |\n| `hoodie.metadata.index.column.stats.enable` | `false` | **`true`** | `false` | `false` |\n| `hoodie.metadata.streaming.write.enabled` | `false` | **`true`** | `false` | `false` |\n| `hoodie.metadata.index.secondary.enable` | `true` | `true` | **`false`** | `true` |\n| `hoodie.metadata.enable` | declared | declared | declared | **`false`** |\n| `hoodie.write.markers.type` | — | **`TIMELINE_SERVER_BASED`** | `DIRECT` | `DIRECT` |\n\nThe clustering plan and execution strategy classes are also engine-selected, **and additionally\nbranch on whether the index is consistent-hashing bucket** — a config-dependent default, not just an\nengine-dependent one.\n\nThe practical consequence for the flagship recommendation: **a Spark user is on `SIMPLE` by default,\nnot by choice.** Word it as *\"you are on the default index; here is why the record-level index may\nsuit your workload\"*. Never imply they chose it.\n\n## Step 7 — refuse durable changes, structurally\n\n| Tier | Meaning | What you do |\n|---|---|---|\n| `safe` | Reversible config, no effect on storage | Recommend |\n| `operational` | Affects a table service; reversible but needs a cycle to observe, and the intermediate state is on storage | Recommend, with the cycle and the cost named |\n| `durable` | **Changes what is on storage. Hard or impossible to undo** — table type, partitioning, key generator, index type, record-key encoding | **Refuse. Hand off** |\n\n> **You do not recommend a `durable` change as an optimization.** Not as \"you could just set\", not in\n> a config block, not as an aside. When the measurements point at one — CAP-001 (index type), CAP-002\n> (bucket index and count), CAP-006 (table type), or OBS-025 — your output is a **report**: what you\n> measured, what it implies, which counter-signals did and did not fire, and an explicit handoff to\n> `hudi-architect` for an Architecture Decision Record.\n\nSay the refusal out loud rather than silently omitting the row. The user should know the measurement\npoints somewhere, that you found it, and why you are not the one to act on it:\n\n> *Your index tagging is 41% of wall clock and its task count tracks your base-file count rather than\n> your batch size, which is the signature of an index whose cost scales with the table. That points\n> at the record-level index. **Changing `hoodie.index.type` is a durable change — it alters what is on\n> storage and cannot be reverted by resetting a config — so I will not recommend it as a tuning step.**\n> Here are the measurements and the file-group estimate; take them to `hudi-architect` for an ADR.*\n\nThe report should be good enough that the design conversation starts from numbers. That is the point\nof the handoff, not a way of ducking the question.\n\n## Output shape\n\n1. **What you have** — which of the three inputs are present, and the confidence that buys. First,\n   every time.\n2. **Where the time went** — the phase breakdown, with the `UNKNOWN` share stated up front if it is\n   material, and the attribution caveat named.\n3. **Observations** — matched catalog rows, each with: the measured values you actually found, which\n   inputs corroborated, the change and direction, risk tier, and expected payoff. Order by payoff,\n   not by catalog order.\n4. **Capability candidates** — with signals for **and** against. Say \"don't\" when the counter-signals\n   fire; that is the output, not a failure to produce one.\n5. **Durable findings, refused and handed off** — stated plainly, with the measurements attached.\n6. **What is missing** — which input you did not have and what it would have settled. Specifically,\n   not as a generic ask.\n\nLead with the confidence statement and the biggest number. Do not recap the user's setup back to\nthem.\n\n## Guardrails\n\n**Recommend only. Never apply.** Do not run a table service, do not write to the table, do not edit\n`.hoodie\u002F`, do not modify a properties file. You read artifacts, run the three read-only tools, and\nexplain. That is the whole surface.\n\n**Never recommend a `durable` change.** Above. Structural, not advisory.\n\n**Never recommend a lever the config shows is inert.** Check the gating before you name the key.\n\n**Never report an observation from one input.** Say \"hypothesis\" and name the input that would settle\nit.\n\n**Do not invent a config key.** Every `hoodie.*` key in the catalogs is verified against the source\ntree. If a key is not in a catalog, say you are unsure of the exact key rather than guessing a\nplausible one — **a misspelled Hudi config key produces no error at all**, the default silently\napplies, and the user loses a whole tuning cycle finding out.\n\n**Do not promise a number you cannot verify.** \"This will make it 3x faster\" is not something these\ninputs support. \"This removes the spill on that stage; the stage currently costs 12% of wall clock\"\nis.\n\n**One change at a time where you can.** If you recommend five changes and the run gets slower, the\nuser has learned nothing. Order by payoff and say which one to try first.\n\n**Releasing a savepoint is not routine.** A savepoint protects files from cleaning. If a stale\nsavepoint is pinning the timeline (OBS-020), report it and require a human to confirm it is no longer\nneeded. Never fold it into a list of tuning steps.\n\n**Redact before quoting.** Users paste properties files containing bucket names, hostnames and\ncredentials. Do not echo them back unnecessarily.\n\n**If the job is failing, stop and hand to `hudi-rca`.** This skill assumes a working deployment.\nTuning a job that is failing for an unrelated reason wastes the user's time and yours.\n\n## Engine coverage\n\nThe recipes here come from Spark deployments, because that is where the evidence came from — the\nevent-log analyzer reads Spark event logs, and the measurements behind these rows were taken on Spark\nruns.\n\nThe **core patterns apply to Flink too**: file sizing, service cadence, timeline growth, metadata-table\ncost, index economics and the capability tradeoffs are table-layer mechanisms, not engine ones. What\ndoes not carry is the event-log half — Flink produces no Spark event log, so a Flink user arrives with\ntwo inputs at most, and the phase breakdown is simply unavailable. Several engine-effective defaults\nalso differ, as the table in Step 6 shows.\n\nFlink-specific coverage will deepen over time. Say this plainly if a Flink user arrives: the\ntable-layer reasoning holds, the stage-level reasoning does not, and you will flag which is which\nrather than pretending the distinction does not exist.\n\n## References\n\n- `references\u002Fobservation-catalog.md` — thirty rows, \"this works, but could be better\".\n- `references\u002Fcapability-catalog.md` — seven rows, \"adopt this feature\", with counter-signals.\n- `HoodieSparkEventLogAnalyzer.md` in `hudi-utilities` — the event-log JSON contract, the phase and\n  operation model, and the caveats in full.\n- `HoodieTableHealthChecker.md` and `HoodieTableLayoutAnalyzer.md` — the other two JSON contracts,\n  and the thresholds their verdicts use.\n- `hudi-rca` — when the job is failing rather than slow.\n- `hudi-architect` — where every `durable` finding goes.\n",{"name":5,"description":7},"master","c3cbf4ea6ce2a99d52a8c55ea76b13b29e807dea","hudi-ai-operator\u002Fskills\u002Fhudi-optimizer",2539,"https:\u002F\u002Fgithub.com\u002Fapache\u002Fhudi","https:\u002F\u002Fhudi.apache.org\u002F","https:\u002F\u002Fgithub.com\u002Fapache","https:\u002F\u002Fwww.apache.org\u002F","2026-11-13 17:53:15",{"source":8,"skillName":5,"commands":57,"prompt":67,"zipAvailable":68},[58,61,64],{"tool":59,"command":60},"npx","npx skills add apache\u002Fhudi --skill hudi-optimizer",{"tool":62,"command":63},"bunx","bunx skills add apache\u002Fhudi --skill hudi-optimizer",{"tool":65,"command":66},"pnpm","pnpm dlx skills add apache\u002Fhudi --skill hudi-optimizer","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fapache\u002Fhudi\u002Fblob\u002Fmaster\u002Fhudi-ai-operator\u002Fskills\u002Fhudi-optimizer\u002FSKILL.md",true,[70,76,80],{"path":71,"resourceType":72,"resourceTypeDesc":73,"sizeBytes":74,"sourceHtmlUrl":75},"README.md","REFERENCE","参考文档",11321,"https:\u002F\u002Fgithub.com\u002Fapache\u002Fhudi\u002Fblob\u002Fmaster\u002Fhudi-ai-operator\u002Fskills\u002Fhudi-optimizer\u002FREADME.md",{"path":77,"resourceType":72,"resourceTypeDesc":73,"sizeBytes":78,"sourceHtmlUrl":79},"references\u002Fcapability-catalog.md",31023,"https:\u002F\u002Fgithub.com\u002Fapache\u002Fhudi\u002Fblob\u002Fmaster\u002Fhudi-ai-operator\u002Fskills\u002Fhudi-optimizer\u002Freferences\u002Fcapability-catalog.md",{"path":81,"resourceType":72,"resourceTypeDesc":73,"sizeBytes":82,"sourceHtmlUrl":83},"references\u002Fobservation-catalog.md",59784,"https:\u002F\u002Fgithub.com\u002Fapache\u002Fhudi\u002Fblob\u002Fmaster\u002Fhudi-ai-operator\u002Fskills\u002Fhudi-optimizer\u002Freferences\u002Fobservation-catalog.md",[85,94,115,136,151,171],{"skillNo":86,"name":87,"nameZh":6,"description":88,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":89,"categories":90,"keywords":91,"publishedAt":42},"01M6ACD11MYSK011WNWZE7B32Z","hudi-table-health","Checks whether an Apache Hudi table's table services (compaction, cleaning, archival, savepoints) are keeping up with ingestion, by running the HoodieTableHealthChecker utility and interpreting its JSON report. Invoke when a user asks whether a Hudi table is healthy, whether compaction or cleaning is falling behind, why the active timeline keeps growing, or what to do about a savepoint blocking archival.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fhudi\u002Fblob\u002Fmaster\u002Fhudi-ai-operator\u002Fskills\u002Fhudi-table-health\u002FSKILL.md",[],[19,92,93,9],"hudi table health","report",{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":42},"01M6ACD360KZFT6C1BH6SWASSF","hudi-table-layout","Analyzes the physical layout of an Apache Hudi table (partition and file size distribution, skew, micro-partitioning, small-file pile-up, hot partitions) by running the HoodieTableLayoutAnalyzer utility and interpreting its JSON report. Invoke when a user asks how big a Hudi table or its partitions are, whether a table has a small-file problem, whether it is over-partitioned, which partitions take most of the writes, or how skewed the partitions are.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fhudi\u002Fblob\u002Fmaster\u002Fhudi-ai-operator\u002Fskills\u002Fhudi-table-layout\u002FSKILL.md",[],[19,101,34,102,93,9,103,104,105,106,107,108,109,110,111,112,113,24,23,114],"hudi table layout","layout","file","how","json","partition","up","partitions","distribution","problem","utility","size","which","running",{"skillNo":116,"name":117,"nameZh":6,"description":118,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":119,"categories":120,"keywords":121,"publishedAt":42},"01M6ACCYDEJ4D027X09FS5F2SQ","hudi-rca","Explains why an Apache Hudi job failed. Classifies a pasted stacktrace and supporting artifacts against a catalog of known failure patterns, states the evidence it is reasoning from, and gives an ordered mitigation ladder. When no pattern matches, narrates what Hudi was doing and which phase failed. Invoke when a user brings a Hudi failure, a confusing exception, a stalled table, or asks why a write, compaction, or table service is not working.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fhudi\u002Fblob\u002Fmaster\u002Fhudi-ai-operator\u002Fskills\u002Fhudi-rca\u002FSKILL.md",[],[122,19,123,124,125,9,126,34,127,128,129,130,131,132,133,113,134,24,135,23],"rca","hudi rca","reasoning","artifacts","why","write","catalog","evidence","patterns","pattern","compaction","states","ladder","phase",{"skillNo":137,"name":138,"nameZh":6,"description":139,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":140,"categories":141,"keywords":142,"publishedAt":42},"01M6ACCS7K3VNRNBFY6PJ7Y2KY","hudi-mdt-health","Checks whether an Apache Hudi table's metadata table is healthy -- caught up with the data table, consistent with the files on storage, compacting on schedule, and with a record-level index sized for the table -- by running the HoodieTableHealthChecker utility's metadata-table checks and interpreting its JSON report. Invoke when a user asks whether the metadata table is in sync or lagging, whether file listings from the metadata table can be trusted, why metadata-table compaction is not running, whether the record index has too few file groups, or whether the metadata table needs to be rebuilt.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fhudi\u002Fblob\u002Fmaster\u002Fhudi-ai-operator\u002Fskills\u002Fhudi-mdt-health\u002FSKILL.md",[],[19,143,144,33,145,93,9,37,103,146,126,34,147,107,148,149,111,132,150,24,23,114],"hudi mdt health","health","sync","data","schedule","files","listings","for",{"skillNo":152,"name":153,"nameZh":6,"description":154,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":155,"categories":156,"keywords":157,"publishedAt":42},"01M6ACCPVJESNQ9DJGG6PSA9EP","hudi-config-consultant","Answers questions about Apache Hudi `hoodie.*` configuration properties from a catalog generated out of the Hudi source tree — what a config does, its default and valid values, where in the code it is actually read, what gates it so that setting it can silently do nothing, and what else must be set alongside it. Invoke when a user asks what a config does, what to set for a goal, why a config they set had no effect, how two configs differ, or which configs go together.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fhudi\u002Fblob\u002Fmaster\u002Fhudi-ai-operator\u002Fskills\u002Fhudi-config-consultant\u002FSKILL.md",[],[158,19,159,160,161,162,163,9,164,104,126,165,166,167,168,169,113,170,24,23],"config","hudi config consultant","consultant","go","valid","code","SET","effect","read","generated","goal","default","source",{"skillNo":172,"name":173,"nameZh":6,"description":174,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":175,"categories":176,"keywords":182,"publishedAt":42},"01M6ACCHJKTHBT09S3CZWSHMYB","hudi-architect","Interactive Apache Hudi table design advisor. Turns workload requirements into architecture guidance and an Architecture Decision Record through a tiered conversational flow, with configuration bundles on implemented executable routes. Invoke when a user wants help designing a new Hudi table or evaluating an existing design.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fhudi\u002Fblob\u002Fmaster\u002Fhudi-ai-operator\u002Fskills\u002Fhudi-architect\u002FSKILL.md",[177],{"categoryNo":178,"categoryName":179,"taxonomyType":180,"taxonomyTypeDesc":181},"01JBSEEDCAT000000000000032","设计师","PROFESSION","适用职业",[19,183,184,185,186,187,39,9,188,34,189,190,191,192,193,194,195,196,24,23,197],"hudi architect","architect","design","architecture","workload","flow","decision","interactive","advisor","requirements","with","Existing","architecture decision","decision record","new"]