[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6ACM3GXZHMNHNE4DHDBBXVK":3,"skill-same:01M6ACM3GXZHMNHNE4DHDBBXVK":85},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":34,"publishedAt":43,"linkOnly":44,"markdownContent":45,"markdownBody":46,"frontMatter":47,"refName":48,"commitSha":49,"skillDirectory":50,"forksCount":51,"repositoryHtmlUrl":52,"repositoryHomepageUrl":53,"ownerHtmlUrl":54,"ownerBlogUrl":55,"ownerDescription":6,"updatedAt":56,"install":57,"resources":70},"01M6ACM3GXZHMNHNE4DHDBBXVK","datafusion-python",null,"Use when the user is writing datafusion-python (Apache DataFusion Python bindings) DataFrame or SQL code. Covers imports, data loading, DataFrame operations, expression building, SQL-to-DataFrame mappings, idiomatic patterns, and common pitfalls.","apache\u002Fdatafusion-python","apache","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F47359?v=4","01M6AC83E6K4FDKK7HZ8WNMQKE","The Apache Software Foundation","01M6ACKST0XJWJESE0ZMQG0W2M",613,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-python\u002Fblob\u002Fmain\u002Fskills\u002Fdatafusion_python\u002FSKILL.md",[18,23,26,31],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000010","后端与接口","SCENARIO","使用场景",{"categoryNo":24,"categoryName":25,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000018","写作与文案",{"categoryNo":27,"categoryName":28,"taxonomyType":29,"taxonomyTypeDesc":30},"01JBSEEDCAT000000000000033","数据分析师","PROFESSION","适用职业",{"categoryNo":32,"categoryName":33,"taxonomyType":29,"taxonomyTypeDesc":30},"01JBSEEDCAT000000000000036","内容创作者",[35,36,37,38,39,40,41,9,42],"python","datafusion","datafusion python","writing","sql","code","data loading","dataframe","2026-11-10 10:12:30",false,"---\nname: datafusion-python\ndescription: Use when the user is writing datafusion-python (Apache DataFusion Python bindings) DataFrame or SQL code. Covers imports, data loading, DataFrame operations, expression building, SQL-to-DataFrame mappings, idiomatic patterns, and common pitfalls.\n---\n\n# DataFusion Python DataFrame API Guide\n\n## What Is DataFusion?\n\nDataFusion is an **in-process query engine** built on Apache Arrow. It is not a\ndatabase -- there is no server, no connection string, and no external\ndependencies. You create a `SessionContext`, point it at data (Parquet, CSV,\nJSON, Arrow IPC, Pandas, Polars, or raw Python dicts\u002Flists), and run queries\nusing either SQL or the DataFrame API described below.\n\nAll data flows through **Apache Arrow**. The canonical Python implementation is\nPyArrow (`pyarrow.RecordBatch` \u002F `pyarrow.Table`), but any library that\nconforms to the [Arrow C Data Interface](https:\u002F\u002Farrow.apache.org\u002Fdocs\u002Fformat\u002FCDataInterface.html)\ncan interoperate with DataFusion.\n\n## Core Abstractions\n\n| Abstraction | Role | Key import |\n|---|---|---|\n| `SessionContext` | Entry point. Loads data, runs SQL, produces DataFrames. | `from datafusion import SessionContext` |\n| `DataFrame` | Lazy query builder. Each method returns a new DataFrame. | Returned by context methods |\n| `Expr` | Expression tree node (column ref, literal, function call, ...). | `from datafusion import col, lit` |\n| `functions` | 290+ built-in scalar, aggregate, and window functions. | `from datafusion import functions as F` |\n| `functions.spark` | PySpark-compatible function surface (parameter names match `pyspark.sql.functions`). | `from datafusion.functions import spark` |\n\n## Import Conventions\n\n```python\nfrom datafusion import SessionContext, col, lit\nfrom datafusion import functions as F\nfrom datafusion.functions import spark   # only when porting pyspark code\n```\n\n## Data Loading\n\n```python\nctx = SessionContext()\n\n# From files\ndf = ctx.read_parquet(\"path\u002Fto\u002Fdata.parquet\")\ndf = ctx.read_csv(\"path\u002Fto\u002Fdata.csv\")\ndf = ctx.read_json(\"path\u002Fto\u002Fdata.json\")\n\n# From Python objects\ndf = ctx.from_pydict({\"a\": [1, 2, 3], \"b\": [\"x\", \"y\", \"z\"]})\ndf = ctx.from_pylist([{\"a\": 1, \"b\": \"x\"}, {\"a\": 2, \"b\": \"y\"}])\ndf = ctx.from_pandas(pandas_df)\ndf = ctx.from_polars(polars_df)\ndf = ctx.from_arrow(arrow_table)\ndf = ctx.read_batch(record_batch)               # one pa.RecordBatch, no named table\ndf = ctx.read_batches([batch1, batch2])         # several pa.RecordBatch\n\n# From SQL\ndf = ctx.sql(\"SELECT a, b FROM my_table WHERE a > 1\")\n```\n\nTo make a DataFrame queryable by name in SQL, register it first:\n\n```python\nctx.register_parquet(\"my_table\", \"path\u002Fto\u002Fdata.parquet\")\nctx.register_csv(\"my_table\", \"path\u002Fto\u002Fdata.csv\")\n```\n\n## DataFrame Operations Quick Reference\n\nEvery method returns a **new** DataFrame (immutable\u002Flazy). Chain them fluently.\n\n### Projection\n\n```python\ndf.select(\"a\", \"b\")                         # preferred: plain names as strings\ndf.select(col(\"a\"), (col(\"b\") + 1).alias(\"b_plus_1\"))  # use col()\u002FExpr only when you need an expression\n\ndf.with_column(\"new_col\", col(\"a\") + lit(10))  # add one column\ndf.with_columns(\n    col(\"a\").alias(\"x\"),\n    y=col(\"b\") + lit(1),                    # named keyword form\n)\n\ndf.drop(\"unwanted_col\")\ndf.with_column_renamed(\"old_name\", \"new_name\")\n```\n\nWhen a column is referenced by name alone, pass the name as a string rather\nthan wrapping it in `col()`. Reach for `col()` only when the projection needs\narithmetic, aliasing, casting, or another expression operation.\n\n**Case sensitivity**: both `select(\"Name\")` and `col(\"Name\")` lowercase the\nidentifier. For a column whose real name has uppercase letters, embed double\nquotes inside the string: `select('\"MyCol\"')` or `col('\"MyCol\"')`. Without the\ninner quotes the lookup will fail with `No field named mycol`.\n\n### Filtering\n\n```python\ndf.filter(col(\"a\") > 10)\ndf.filter(col(\"a\") > 10, col(\"b\") == \"x\")   # multiple = AND\ndf.filter(\"a > 10\")                          # SQL expression string\n```\n\nRaw Python values on the right-hand side of a comparison are auto-wrapped\ninto literals by the `Expr` operators, so prefer `col(\"a\") > 10` over\n`col(\"a\") > lit(10)`. See the Comparisons section and pitfall #2 for the\nfull rule.\n\n### Aggregation\n\n```python\n# GROUP BY a, compute sum(b) and count(*)\ndf.aggregate([\"a\"], [F.sum(col(\"b\")), F.count(col(\"a\"))])\n\n# HAVING equivalent: use the filter keyword on the aggregate function\ndf.aggregate(\n    [\"region\"],\n    [F.sum(col(\"sales\"), filter=col(\"sales\") > 1000).alias(\"large_sales\")],\n)\n```\n\nAs with `select()`, group keys can be passed as plain name strings. Reach for\n`col(...)` only when the grouping expression needs arithmetic, aliasing,\ncasting, or another expression operation.\n\nMost aggregate functions accept an optional `filter` keyword argument. When\nprovided, only rows where the filter expression is true contribute to the\naggregate.\n\n### Sorting\n\n```python\ndf.sort(\"a\")                                 # ascending (plain name, preferred)\ndf.sort(col(\"a\"))                            # ascending via col()\ndf.sort(col(\"a\").sort(ascending=False))      # descending\ndf.sort(col(\"a\").sort(nulls_first=False))    # override null placement\n\ndf.sort_by(\"a\", \"b\")                         # ascending-only shortcut\n```\n\nAs with `select()` and `aggregate()`, bare column references can be passed as\nplain name strings. A plain expression passed to `sort()` is already treated\nas ascending, so reach for `col(...).sort(...)` only when you need to override\na default (descending order or null placement). Writing\n`col(\"a\").sort(ascending=True)` is redundant.\n\nFor ascending-only sorts with no null-placement override, `df.sort_by(...)` is\na shorter alias for `df.sort(...)`.\n\n### Joining\n\n```python\n# Equi-join on shared column name\ndf1.join(df2, on=\"key\")\ndf1.join(df2, on=\"key\", how=\"left\")\n\n# Different column names\ndf1.join(df2, left_on=\"id\", right_on=\"fk_id\", how=\"inner\")\n\n# Expression-based join (supports inequality predicates)\ndf1.join_on(df2, col(\"a\") == col(\"b\"), how=\"inner\")\n\n# Semi join: keep rows from left where a match exists in right (like EXISTS)\ndf1.join(df2, on=\"key\", how=\"semi\")\n\n# Anti join: keep rows from left where NO match exists in right (like NOT EXISTS)\ndf1.join(df2, on=\"key\", how=\"anti\")\n```\n\nJoin types: `\"inner\"`, `\"left\"`, `\"right\"`, `\"full\"`, `\"semi\"`, `\"anti\"`.\n\nInner is the default `how`. Prefer `df1.join(df2, on=\"key\")` over\n`df1.join(df2, on=\"key\", how=\"inner\")` — drop `how=` unless you need a\nnon-inner join type.\n\nWhen the two sides' join columns have different native names, use\n`left_on=`\u002F`right_on=` with the original names rather than aliasing one side\nto match the other — see pitfall #7.\n\n### Window Functions\n\n```python\nfrom datafusion import WindowFrame\n\n# Row number partitioned by group, ordered by value\ndf.window(\n    F.row_number(\n        partition_by=[col(\"group\")],\n        order_by=[col(\"value\")],\n    ).alias(\"rn\")\n)\n\n# Using a Window object for reuse\nfrom datafusion.expr import Window\n\nwin = Window(\n    partition_by=[col(\"group\")],\n    order_by=[col(\"value\").sort(ascending=True)],\n)\ndf.select(\n    col(\"group\"),\n    col(\"value\"),\n    F.sum(col(\"value\")).over(win).alias(\"running_total\"),\n)\n\n# With explicit frame bounds\nwin = Window(\n    partition_by=[col(\"group\")],\n    order_by=[col(\"value\").sort(ascending=True)],\n    window_frame=WindowFrame(\"rows\", 0, None),  # current row to unbounded following\n)\n```\n\n### Set Operations\n\n```python\ndf1.union(df2)                          # UNION ALL (by position)\ndf1.union(df2, distinct=True)           # UNION DISTINCT\ndf1.union_by_name(df2)                  # match columns by name, not position\ndf1.intersect(df2)                      # INTERSECT ALL\ndf1.intersect(df2, distinct=True)       # INTERSECT (distinct)\ndf1.except_all(df2)                     # EXCEPT ALL\ndf1.except_all(df2, distinct=True)      # EXCEPT (distinct)\n```\n\n### Limit and Offset\n\n```python\ndf.limit(10)            # first 10 rows\ndf.limit(10, offset=20) # skip 20, then take 10\n```\n\n### Deduplication\n\n```python\ndf.distinct()           # remove duplicate rows\ndf.distinct_on(         # keep first row per group (like DISTINCT ON in Postgres)\n    [col(\"a\")],                     # uniqueness columns\n    [col(\"a\"), col(\"b\")],           # output columns\n    [col(\"b\").sort(ascending=True)], # which row to keep\n)\n```\n\n## Executing and Collecting Results\n\nDataFrames are lazy until you collect.\n\n```python\ndf.show()                               # print formatted table to stdout\nbatches = df.collect()                  # list[pa.RecordBatch]\narr = df.collect_column(\"col_name\")     # pa.Array | pa.ChunkedArray (single column)\ntable = df.to_arrow_table()             # pa.Table\npandas_df = df.to_pandas()              # pd.DataFrame\npolars_df = df.to_polars()              # pl.DataFrame\npy_dict = df.to_pydict()                # dict[str, list]\npy_list = df.to_pylist()                # list[dict]\ncount = df.count()                      # int\ndf = df.cache()                         # materialize in memory, return DataFrame\n```\n\n### Date and Timestamp Type Conversion\n\nThe Python type returned by `to_pydict()` \u002F `to_pylist()` depends on the Arrow\ncolumn type, and the mapping is inherited from PyArrow:\n\n| Arrow type | Python type returned |\n|---|---|\n| `timestamp(s)` \u002F `(ms)` \u002F `(us)` | `datetime.datetime` |\n| `timestamp(ns)` | `pandas.Timestamp` |\n| `date32` \u002F `date64` | `datetime.date` |\n| `duration(s)` \u002F `(ms)` \u002F `(us)` | `datetime.timedelta` |\n| `duration(ns)` | `pandas.Timedelta` |\n\nThe nanosecond-precision fallback to pandas types is the main surprise:\npandas is not a hard dependency of `datafusion`, but PyArrow reaches for it\nwhen `datetime.datetime` \u002F `datetime.timedelta` would lose precision (stdlib\ntypes only go to microseconds). If you need plain stdlib types, cast to a\ncoarser unit before collecting, e.g.\n`df.select(col(\"ts\").cast(pa.timestamp(\"us\")))`.\n\n`df.to_pandas()` has its own footgun for dates: pandas has no pure-date dtype,\nso a `date32`\u002F`date64` column comes back as an `object` column of\n`datetime.date` values rather than `datetime64[ns]`. If downstream code\nexpects a datetime column, cast on the DataFusion side first:\n`col(\"ship_date\").cast(pa.timestamp(\"ns\"))`.\n\n### Streaming Results\n\nPrefer streaming over `collect()` when the result is too large to materialize\nin memory, when you want to start processing before the query finishes, or\nwhen you may break out of the loop early. `execute_stream()` pulls one\n`RecordBatch` at a time from the execution plan rather than buffering the\nwhole result up front.\n\n```python\n# Single-partition stream; batch is a datafusion.RecordBatch\nstream = df.execute_stream()\nfor batch in stream:\n    process(batch.to_pyarrow())         # convert to pa.RecordBatch if needed\n\n# DataFrame is iterable directly (delegates to execute_stream)\nfor batch in df:\n    process(batch.to_pyarrow())\n\n# One stream per partition, for parallel consumption\nfor stream in df.execute_stream_partitioned():\n    for batch in stream:\n        process(batch.to_pyarrow())\n```\n\nAsync iteration is also supported via `async for batch in df: ...` (or\n`df.execute_stream()`), which is useful when batches are interleaved with\nother I\u002FO.\n\n### Caching Intermediate Results\n\n`df.cache()` materializes a DataFrame as an in-memory table and returns a new\nDataFrame backed by it. Reach for it when the same intermediate result feeds\nmultiple downstream queries — without `cache()`, each branch re-executes the\nfull upstream plan (re-reading files, recomputing filters\u002Faggregates).\n\n```python\nbase = (\n    ctx.read_parquet(\"orders.parquet\")\n    .filter(col(\"status\") == \"shipped\")\n    .cache()                                # materialize once, reuse below\n)\nby_region = base.aggregate([\"region\"], [F.sum(col(\"amount\")).alias(\"total\")])\nby_customer = base.aggregate([\"customer\"], [F.sum(col(\"amount\")).alias(\"total\")])\n```\n\nSkip `cache()` for single-use DataFrames — the lazy plan is already optimal.\n\nThe cached table is owned by the DataFrame returned from `cache()` (and any\nDataFrames chained from it). To free the memory, drop every reference — let\nthem go out of scope, or `del base; del by_region; del by_customer`.\n\n### Writing Results\n\n```python\ndf.write_parquet(\"output.parquet\")\ndf.write_csv(\"output.csv\")\ndf.write_json(\"output.json\")\n```\n\nYou can also pass a directory path (e.g., `\"output\u002F\"`) to write a multi-file\npartitioned output.\n\n## Expression Building\n\n### Column References and Literals\n\n```python\ncol(\"column_name\")              # reference a column\nlit(42)                          # integer literal\nlit(\"hello\")                     # string literal\nlit(3.14)                        # float literal\nlit(pa.scalar(value))            # PyArrow scalar (preserves Arrow type)\n```\n\n`lit()` accepts PyArrow scalars directly -- prefer this over converting Arrow\ndata to Python and back when working with values extracted from query results.\n\n### Arithmetic\n\n```python\ncol(\"price\") * col(\"quantity\")            # multiplication\ncol(\"a\") + lit(1)                          # addition\ncol(\"a\") - col(\"b\")                        # subtraction\ncol(\"a\") \u002F lit(2)                          # division\ncol(\"a\") % lit(3)                          # modulo\n```\n\n### Date Arithmetic\n\n`Date32` and `Date64` columns both require `Interval` types for arithmetic,\nnot `Duration`. Use PyArrow's `month_day_nano_interval` type, which takes a\n`(months, days, nanos)` tuple:\n\n```python\nimport pyarrow as pa\n\n# Subtract 90 days from a date column\ncol(\"ship_date\") - lit(pa.scalar((0, 90, 0), type=pa.month_day_nano_interval()))\n\n# Subtract 3 months\ncol(\"ship_date\") - lit(pa.scalar((3, 0, 0), type=pa.month_day_nano_interval()))\n```\n\n**Important**: `lit(datetime.timedelta(days=90))` creates a `Duration(µs)`\nliteral, which is **not** compatible with `Date32`\u002F`Date64` arithmetic\n(`Duration(ms)` and `Duration(ns)` are rejected too). Always use\n`pa.month_day_nano_interval()` for date operations.\n\n**Timestamps behave differently**: `Timestamp` columns *do* accept `Duration`,\nso `col(\"ts\") - lit(datetime.timedelta(days=1))` works. The interval-only\nrule applies specifically to date columns.\n\n### Comparisons\n\n```python\ncol(\"a\") > 10\ncol(\"a\") >= 10\ncol(\"a\") \u003C 10\ncol(\"a\") \u003C= 10\ncol(\"a\") == \"x\"\ncol(\"a\") != \"x\"\ncol(\"a\") == None                           # same as col(\"a\").is_null()\ncol(\"a\") != None                           # same as col(\"a\").is_not_null()\n```\n\nComparison operators auto-wrap the right-hand Python value into a literal,\nso writing `col(\"a\") > lit(10)` is redundant. Drop the `lit()` in\ncomparisons. Reach for `lit()` only when auto-wrapping does not apply — see\npitfall #2.\n\n### Boolean Logic\n\n**Important**: Python's `and`, `or`, `not` keywords do NOT work with Expr\nobjects. You must use the bitwise operators:\n\n```python\n(col(\"a\") > 1) & (col(\"b\") \u003C 10)   # AND\n(col(\"a\") > 1) | (col(\"b\") \u003C 10)   # OR\n~(col(\"a\") > 1)                    # NOT\n```\n\nAlways wrap each comparison in parentheses when combining with `&`, `|`, `~`\nbecause Python's operator precedence for bitwise operators is different from\nlogical operators.\n\n### Null Handling\n\n```python\ncol(\"a\").is_null()\ncol(\"a\").is_not_null()\ncol(\"a\").fill_null(lit(0))          # replace NULL with a value (single expression)\nF.coalesce(col(\"a\"), col(\"b\"))     # first non-null value\nF.nullif(col(\"a\"), lit(0))         # return NULL if a == 0\n```\n\nTo fill nulls across the whole DataFrame (optionally limited to a subset of\ncolumns), use the DataFrame-level method:\n\n```python\ndf.fill_null(0)                     # every column\ndf.fill_null(0, subset=[\"a\", \"b\"])  # only these columns\n```\n\n### CASE \u002F WHEN\n\n```python\n# Simple CASE (matching on a single expression)\nstatus_label = (\n    F.case(col(\"status\"))\n    .when(lit(\"A\"), lit(\"Active\"))\n    .when(lit(\"I\"), lit(\"Inactive\"))\n    .otherwise(lit(\"Unknown\"))\n)\n\n# Searched CASE (each branch has its own predicate)\nseverity = (\n    F.when(col(\"value\") > 100, lit(\"high\"))\n    .when(col(\"value\") > 50, lit(\"medium\"))\n    .otherwise(lit(\"low\"))\n)\n```\n\n### Casting\n\n```python\nimport pyarrow as pa\n\ncol(\"a\").cast(pa.float64())\ncol(\"a\").cast(pa.utf8())\ncol(\"a\").cast(pa.date32())\n\ncol(\"a\").try_cast(pa.int32())        # like cast(), but yields NULL on failure instead of erroring\n```\n\nTo cast several columns at once at the DataFrame level, pass a mapping to\n`df.cast(...)`:\n\n```python\ndf.cast({\"a\": pa.float64(), \"b\": pa.int32()})\n```\n\n### Aliasing\n\n```python\n(col(\"a\") + col(\"b\")).alias(\"total\")\n```\n\n### BETWEEN and IN\n\n```python\ncol(\"a\").between(1, 10)                                 # 1 \u003C= a \u003C= 10 (bounds auto-wrap)\nF.in_list(col(\"a\"), [lit(1), lit(2), lit(3)])           # a IN (1, 2, 3)\nF.in_list(col(\"a\"), [lit(1), lit(2)], negated=True)     # a NOT IN (1, 2)\n```\n\n### Struct and Array Access\n\n```python\ncol(\"struct_col\")[\"field_name\"]     # access struct field\ncol(\"array_col\")[0]                  # access array element (0-indexed)\ncol(\"array_col\")[1:3]                # array slice (0-indexed)\n```\n\n### Lambda Functions\n\nSome array functions take a lambda function that runs once per element. Pass a\nPython `lambda` directly — its parameter names become the lambda parameters and\nits return value becomes the body:\n\n```python\nF.array_transform(col(\"a\"), lambda v: v * 2)    # map: [1,2,3] -> [2,4,6]\nF.array_filter(col(\"a\"), lambda v: v > 2)        # filter: [1,2,3] -> [3]\nF.array_any_match(col(\"a\"), lambda v: v > 3)     # predicate: any element > 3\n```\n\nFor explicit parameter names, build the lambda by hand:\n\n```python\nF.array_transform(col(\"a\"), F.lambda_([\"v\"], F.lambda_var(\"v\") * lit(2)))\n```\n\n## SQL-to-DataFrame Reference\n\n| SQL | DataFrame API |\n|---|---|\n| `SELECT a, b` | `df.select(\"a\", \"b\")` |\n| `SELECT a, b + 1 AS c` | `df.select(col(\"a\"), (col(\"b\") + lit(1)).alias(\"c\"))` |\n| `SELECT *, a + 1 AS c` | `df.with_column(\"c\", col(\"a\") + lit(1))` |\n| `WHERE a > 10` | `df.filter(col(\"a\") > 10)` |\n| `GROUP BY a` with `SUM(b)` | `df.aggregate([\"a\"], [F.sum(col(\"b\"))])` |\n| `SUM(b) FILTER (WHERE b > 100)` | `F.sum(col(\"b\"), filter=col(\"b\") > 100)` |\n| `ORDER BY a DESC` | `df.sort(col(\"a\").sort(ascending=False))` |\n| `LIMIT 10 OFFSET 5` | `df.limit(10, offset=5)` |\n| `DISTINCT` | `df.distinct()` |\n| `a INNER JOIN b ON a.id = b.id` | `a.join(b, on=\"id\")` |\n| `a LEFT JOIN b ON a.id = b.fk` | `a.join(b, left_on=\"id\", right_on=\"fk\", how=\"left\")` |\n| `WHERE EXISTS (SELECT ...)` | `a.join(b, on=\"key\", how=\"semi\")` |\n| `WHERE NOT EXISTS (SELECT ...)` | `a.join(b, on=\"key\", how=\"anti\")` |\n| `UNION ALL` | `df1.union(df2)` |\n| `UNION` (distinct) | `df1.union(df2, distinct=True)` |\n| `INTERSECT ALL` | `df1.intersect(df2)` |\n| `INTERSECT` (distinct) | `df1.intersect(df2, distinct=True)` |\n| `EXCEPT ALL` | `df1.except_all(df2)` |\n| `EXCEPT` (distinct) | `df1.except_all(df2, distinct=True)` |\n| `CASE x WHEN 1 THEN 'a' END` | `F.case(col(\"x\")).when(lit(1), lit(\"a\")).end()` |\n| `CASE WHEN x > 1 THEN 'a' END` | `F.when(col(\"x\") > 1, lit(\"a\")).end()` |\n| `x IN (1, 2, 3)` | `F.in_list(col(\"x\"), [lit(1), lit(2), lit(3)])` |\n| `x BETWEEN 1 AND 10` | `col(\"x\").between(1, 10)` |\n| `CAST(x AS DOUBLE)` | `col(\"x\").cast(pa.float64())` |\n| `ROW_NUMBER() OVER (...)` | `F.row_number(partition_by=[...], order_by=[...])` |\n| `SUM(x) OVER (...)` | `F.sum(col(\"x\")).over(window)` |\n| `x IS NULL` | `col(\"x\").is_null()` |\n| `COALESCE(a, b)` | `F.coalesce(col(\"a\"), col(\"b\"))` |\n\n## Common Pitfalls\n\n1. **Boolean operators**: Use `&`, `|`, `~` -- not Python's `and`, `or`, `not`.\n   Always parenthesize: `(col(\"a\") > 1) & (col(\"b\") \u003C 2)`.\n\n2. **Wrapping scalars with `lit()`**: Prefer raw Python values on the\n   right-hand side of comparisons — `col(\"a\") > 10`, `col(\"name\") == \"Alice\"`\n   — because the Expr comparison operators auto-wrap them. Writing\n   `col(\"a\") > lit(10)` is redundant. Reserve `lit()` for places where\n   auto-wrapping does *not* apply:\n   - standalone scalars passed into function calls:\n     `F.coalesce(col(\"a\"), lit(0))`, not `F.coalesce(col(\"a\"), 0)`\n   - arithmetic between two literals with no column involved:\n     `lit(1) - col(\"discount\")` is fine, but `lit(1) - lit(2)` needs both\n   - values that must carry a specific Arrow type, via `lit(pa.scalar(...))`\n   - `.when(...)`, `.otherwise(...)`, `F.nullif(...)`, `F.in_list(...)`\n     and similar method\u002Ffunction arguments (note: `.between(...)`\n     auto-wraps its bounds, so `col(\"a\").between(1, 10)` needs no `lit()`)\n\n3. **Column name quoting**: Column names are normalized to lowercase by default\n   in both `select(\"...\")` and `col(\"...\")`. To reference a column with\n   uppercase letters, use double quotes inside the string:\n   `select('\"MyColumn\"')` or `col('\"MyColumn\"')`.\n\n4. **DataFrames are immutable**: Every method returns a **new** DataFrame. You\n   must capture the return value:\n   ```python\n   df = df.filter(col(\"a\") > 1)   # correct\n   df.filter(col(\"a\") > 1)         # WRONG -- result is discarded\n   ```\n\n5. **Window frame defaults**: When using `order_by` in a window, the default\n   frame is `RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`. For a full\n   partition frame, set `window_frame=WindowFrame(\"rows\", None, None)`.\n\n6. **Arithmetic on aggregates belongs in a later `select`, not inside\n   `aggregate`** *(applies to datafusion-python 53 and earlier; fixed in 54)*:\n   Each item in the aggregate list must be a single aggregate\n   call (optionally aliased). Combining aggregates with arithmetic inside\n   `aggregate(...)` fails with `Internal error: Invalid aggregate expression`.\n   Alias the aggregates, then compute the combination downstream:\n   ```python\n   # WRONG -- arithmetic wraps two aggregates\n   df.aggregate([], [(lit(100) * F.sum(col(\"a\")) \u002F F.sum(col(\"b\"))).alias(\"ratio\")])\n\n   # CORRECT -- aggregate first, then combine\n   (df.aggregate([], [F.sum(col(\"a\")).alias(\"num\"), F.sum(col(\"b\")).alias(\"den\")])\n      .select((lit(100) * col(\"num\") \u002F col(\"den\")).alias(\"ratio\")))\n   ```\n\n7. **Don't alias a join column to match the other side**: When equi-joining\n   with `on=\"key\"`, renaming the join column on one side via `.alias(\"key\")`\n   in a fresh projection creates a schema where one side's `key` is\n   qualified (`?table?.key`) and the other is unqualified. The join then\n   fails with `Schema contains qualified field name ... and unqualified\n   field name ... which would be ambiguous`. Use `left_on=`\u002F`right_on=` with\n   the native names, or use `join_on(...)` with an explicit equality.\n   ```python\n   # WRONG -- alias on one side produces ambiguous schema after join\n   failed = orders.select(col(\"o_orderkey\").alias(\"l_orderkey\"))\n   li.join(failed, on=\"l_orderkey\")   # ambiguous l_orderkey error\n\n   # CORRECT -- keep native names, use left_on\u002Fright_on\n   failed = orders.select(\"o_orderkey\")\n   li.join(failed, left_on=\"l_orderkey\", right_on=\"o_orderkey\")\n\n   # ALSO CORRECT -- explicit predicate via join_on\n   # (note: join_on keeps both key columns in the output, unlike on=\"key\")\n   li.join_on(failed, col(\"l_orderkey\") == col(\"o_orderkey\"))\n   ```\n   When the same column name exists on both sides, `DataFrame.col(name)`\n   (and `DataFrame.column(name)`) returns a column reference qualified to\n   that DataFrame, which disambiguates the predicate explicitly:\n   ```python\n   li.join_on(failed, li.col(\"l_orderkey\") == failed.col(\"o_orderkey\"))\n   ```\n\n## Idiomatic Patterns\n\n### Fluent Chaining\n\n```python\nresult = (\n    ctx.read_parquet(\"data.parquet\")\n    .filter(col(\"year\") >= 2020)\n    .select(col(\"region\"), col(\"sales\"))\n    .aggregate([\"region\"], [F.sum(col(\"sales\")).alias(\"total\")])\n    .sort(col(\"total\").sort(ascending=False))\n    .limit(10)\n)\nresult.show()\n```\n\n### Using Variables as CTEs\n\nInstead of SQL CTEs (`WITH ... AS`), assign intermediate DataFrames to\nvariables:\n\n```python\nbase = ctx.read_parquet(\"orders.parquet\").filter(col(\"status\") == \"shipped\")\nby_region = base.aggregate([\"region\"], [F.sum(col(\"amount\")).alias(\"total\")])\ntop_regions = by_region.filter(col(\"total\") > 10000)\n```\n\n### Reusing Expressions as Variables\n\nJust like DataFrames, expressions (`Expr`) can be stored in variables and used\nanywhere an `Expr` is expected. This is useful for building up complex\nexpressions or reusing a computed value across multiple operations:\n\n```python\n# Build an expression and reuse it\ndisc_price = col(\"price\") * (lit(1) - col(\"discount\"))\ndf = df.select(\n    col(\"id\"),\n    disc_price.alias(\"disc_price\"),\n    (disc_price * (lit(1) + col(\"tax\"))).alias(\"total\"),\n)\n\n# Use a collected scalar as an expression\nmax_val = result_df.collect_column(\"max_price\")[0]   # PyArrow scalar\ncutoff = lit(max_val) - lit(pa.scalar((0, 90, 0), type=pa.month_day_nano_interval()))\ndf = df.filter(col(\"ship_date\") \u003C= cutoff)           # cutoff is already an Expr\n```\n\n**Important**: Do not wrap an `Expr` in `lit()`. `lit()` is for converting\nPython\u002FPyArrow values into expressions. If a value is already an `Expr`, use it\ndirectly.\n\n### Window Functions for Scalar Subqueries\n\nWhere SQL uses a correlated scalar subquery, the idiomatic DataFrame approach\nis a window function:\n\n```sql\n-- SQL scalar subquery\nSELECT *, (SELECT SUM(b) FROM t WHERE t.group = s.group) AS group_total FROM s\n```\n\n```python\n# DataFrame: window function\nwin = Window(partition_by=[col(\"group\")])\ndf = df.with_column(\"group_total\", F.sum(col(\"b\")).over(win))\n```\n\n### Semi\u002FAnti Joins for EXISTS \u002F NOT EXISTS\n\n```sql\n-- SQL: WHERE EXISTS (SELECT 1 FROM other WHERE other.key = main.key)\n-- DataFrame:\nresult = main.join(other, on=\"key\", how=\"semi\")\n\n-- SQL: WHERE NOT EXISTS (SELECT 1 FROM other WHERE other.key = main.key)\n-- DataFrame:\nresult = main.join(other, on=\"key\", how=\"anti\")\n```\n\n### Computed Columns\n\n```python\n# Add computed columns while keeping all originals\ndf = df.with_column(\"full_name\", F.concat(col(\"first\"), lit(\" \"), col(\"last\")))\ndf = df.with_column(\"discounted\", col(\"price\") * lit(0.9))\n```\n\n## Available Functions (Categorized)\n\nThe `functions` module (imported as `F`) provides 290+ functions. Key categories:\n\n**Aggregate**: `sum`, `avg`, `min`, `max`, `count`, `count_star`, `median`,\n`stddev`, `stddev_pop`, `var_samp`, `var_pop`, `corr`, `covar`, `approx_distinct`,\n`approx_median`, `approx_percentile_cont`, `array_agg`, `string_agg`,\n`first_value`, `last_value`, `bit_and`, `bit_or`, `bit_xor`, `bool_and`,\n`bool_or`, `grouping`, `regr_*` (9 regression functions)\n\n**Window**: `row_number`, `rank`, `dense_rank`, `percent_rank`, `cume_dist`,\n`ntile`, `lag`, `lead`, `first_value`, `last_value`, `nth_value`\n\n**String**: `length`, `lower`, `upper`, `trim`, `ltrim`, `rtrim`, `lpad`,\n`rpad`, `starts_with`, `ends_with`, `contains`, `substr`, `substring`,\n`replace`, `reverse`, `repeat`, `split_part`, `concat`, `concat_ws`,\n`initcap`, `ascii`, `chr`, `left`, `right`, `strpos`, `translate`, `overlay`,\n`levenshtein`\n\n`F.substr(str, start)` returns the tail of the string from `start` onward;\n`F.substr(str, start, length=n)` returns `n` characters from `start`, like\nSQL's `SUBSTRING(str FROM start FOR length)`. `start` and `length` accept\nnative ints. `F.substring(str, start, length)` is the same with `length`\nrequired. For a fixed-length prefix, `F.left(col(\"s\"), lit(n))` is cleanest.\n\n*The `length` argument to `substr` requires datafusion-python 55 or newer.\nOn earlier versions a third argument raises `TypeError: substr() takes 2\npositional arguments but 3 were given`; use `F.substring` there.*\n\n```python\nF.substr(col(\"c_phone\"), 1, length=2)         # first 2 characters\nF.substring(col(\"c_phone\"), lit(1), lit(2))   # same, on any version\nF.left(col(\"c_phone\"), lit(2))                # prefix shortcut\n```\n\n**Math**: `abs`, `ceil`, `floor`, `round`, `trunc`, `sqrt`, `cbrt`, `exp`,\n`ln`, `log`, `log2`, `log10`, `pow`, `signum`, `pi`, `random`, `factorial`,\n`gcd`, `lcm`, `greatest`, `least`, sin\u002Fcos\u002Ftan and inverse\u002Fhyperbolic variants\n\n**Date\u002FTime**: `now`, `today`, `current_date`, `current_time`,\n`current_timestamp`, `date_part`, `date_trunc`, `date_bin`, `extract`,\n`to_timestamp`, `to_timestamp_millis`, `to_timestamp_micros`,\n`to_timestamp_nanos`, `to_timestamp_seconds`, `to_unixtime`, `from_unixtime`,\n`make_date`, `make_time`, `to_date`, `to_time`, `to_local_time`, `date_format`\n\n**Conditional**: `case`, `when`, `coalesce`, `nullif`, `ifnull`, `nvl`, `nvl2`\n\n**Array\u002FList**: `array`, `make_array`, `array_agg`, `array_length`,\n`array_element`, `array_slice`, `array_append`, `array_prepend`,\n`array_concat`, `array_contains`, `array_has`, `array_has_all`, `array_has_any`, `array_position`,\n`array_remove`, `array_distinct`, `array_sort`, `array_reverse`, `flatten`,\n`array_to_string`, `array_intersect`, `array_union`, `array_except`,\n`generate_series`\n(Most `array_*` functions also have `list_*` aliases.)\n\n**Struct\u002FMap**: `struct`, `named_struct`, `get_field`, `make_map`, `map_keys`,\n`map_values`, `map_entries`, `map_extract`\n\n**Regex**: `regexp_like`, `regexp_match`, `regexp_replace`, `regexp_count`,\n`regexp_instr`\n\n**Hash**: `md5`, `sha224`, `sha256`, `sha384`, `sha512`, `digest`\n\n**Type**: `arrow_typeof`, `arrow_cast`, `arrow_try_cast`, `arrow_field`,\n`arrow_metadata`, `cast_to_type`, `with_metadata`\n\nNote: ``cast_to_type(value, type_ref, *, try_cast=False)`` is the single\nPython entry point for both upstream ``cast_to_type`` and ``try_cast_to_type``;\npass ``try_cast=True`` for the variant that returns NULL on failure.\n\n**Other**: `in_list`, `order_by`, `alias`, `col`, `encode`, `decode`,\n`to_hex`, `to_char`, `uuid`, `version`, `bit_length`, `octet_length`\n\n### Spark-Compatible Functions\n\nA separate `datafusion.functions.spark` namespace mirrors the\n`pyspark.sql.functions` API for callers porting code from PySpark.\n\n```python\nfrom datafusion.functions import spark\n```\n\nUse it for DataFrame work; for SQL, register the Spark UDFs first:\n\n```python\nctx = SessionContext()\nctx.enable_spark_functions()                  # makes Spark UDFs visible to SQL\nctx.sql(\"SELECT sha2('hello', 256)\").show()\n```\n\nCoverage spans aggregate, array, bitmap, bitwise, datetime, hash, JSON,\nmap, math, string, URL, and conditional categories. The authoritative\nlist of what is currently exposed is the `__all__` in\n`python\u002Fdatafusion\u002Ffunctions\u002Fspark.py`:\n\n```bash\npython -c \"from datafusion.functions import spark; print(sorted(spark.__all__))\"\n```\n\nWhen you need to know whether a specific pyspark function is available,\ncheck `__all__` rather than this skill — the list there moves with the\ncode; any enumeration here would drift.\n\n**Semantic divergences vs the default namespace.** Functions that exist in\nboth `functions` and `functions.spark` may behave differently:\n\n| Function | Default `functions` | `functions.spark` |\n|---|---|---|\n| `concat` | NULL inputs treated as empty | NULL inputs propagate to NULL |\n| `round` | HALF_EVEN (banker's) | HALF_UP |\n| `trunc` | Numeric truncation | Date truncation |\n\nPick the namespace whose semantics match your intent — both stay imported\nside by side; `enable_spark_functions()` only affects SQL.\n\n**Parameter names match pyspark exactly.** The spark namespace uses\npyspark parameter names (`col`, `str`, `numBits`, `partToExtract`, ...) so\nyou can paste pyspark code and keep keyword arguments working. The default\nnamespace keeps DataFusion's parameter names.\n","# DataFusion Python DataFrame API Guide\n\n## What Is DataFusion?\n\nDataFusion is an **in-process query engine** built on Apache Arrow. It is not a\ndatabase -- there is no server, no connection string, and no external\ndependencies. You create a `SessionContext`, point it at data (Parquet, CSV,\nJSON, Arrow IPC, Pandas, Polars, or raw Python dicts\u002Flists), and run queries\nusing either SQL or the DataFrame API described below.\n\nAll data flows through **Apache Arrow**. The canonical Python implementation is\nPyArrow (`pyarrow.RecordBatch` \u002F `pyarrow.Table`), but any library that\nconforms to the [Arrow C Data Interface](https:\u002F\u002Farrow.apache.org\u002Fdocs\u002Fformat\u002FCDataInterface.html)\ncan interoperate with DataFusion.\n\n## Core Abstractions\n\n| Abstraction | Role | Key import |\n|---|---|---|\n| `SessionContext` | Entry point. Loads data, runs SQL, produces DataFrames. | `from datafusion import SessionContext` |\n| `DataFrame` | Lazy query builder. Each method returns a new DataFrame. | Returned by context methods |\n| `Expr` | Expression tree node (column ref, literal, function call, ...). | `from datafusion import col, lit` |\n| `functions` | 290+ built-in scalar, aggregate, and window functions. | `from datafusion import functions as F` |\n| `functions.spark` | PySpark-compatible function surface (parameter names match `pyspark.sql.functions`). | `from datafusion.functions import spark` |\n\n## Import Conventions\n\n```python\nfrom datafusion import SessionContext, col, lit\nfrom datafusion import functions as F\nfrom datafusion.functions import spark   # only when porting pyspark code\n```\n\n## Data Loading\n\n```python\nctx = SessionContext()\n\n# From files\ndf = ctx.read_parquet(\"path\u002Fto\u002Fdata.parquet\")\ndf = ctx.read_csv(\"path\u002Fto\u002Fdata.csv\")\ndf = ctx.read_json(\"path\u002Fto\u002Fdata.json\")\n\n# From Python objects\ndf = ctx.from_pydict({\"a\": [1, 2, 3], \"b\": [\"x\", \"y\", \"z\"]})\ndf = ctx.from_pylist([{\"a\": 1, \"b\": \"x\"}, {\"a\": 2, \"b\": \"y\"}])\ndf = ctx.from_pandas(pandas_df)\ndf = ctx.from_polars(polars_df)\ndf = ctx.from_arrow(arrow_table)\ndf = ctx.read_batch(record_batch)               # one pa.RecordBatch, no named table\ndf = ctx.read_batches([batch1, batch2])         # several pa.RecordBatch\n\n# From SQL\ndf = ctx.sql(\"SELECT a, b FROM my_table WHERE a > 1\")\n```\n\nTo make a DataFrame queryable by name in SQL, register it first:\n\n```python\nctx.register_parquet(\"my_table\", \"path\u002Fto\u002Fdata.parquet\")\nctx.register_csv(\"my_table\", \"path\u002Fto\u002Fdata.csv\")\n```\n\n## DataFrame Operations Quick Reference\n\nEvery method returns a **new** DataFrame (immutable\u002Flazy). Chain them fluently.\n\n### Projection\n\n```python\ndf.select(\"a\", \"b\")                         # preferred: plain names as strings\ndf.select(col(\"a\"), (col(\"b\") + 1).alias(\"b_plus_1\"))  # use col()\u002FExpr only when you need an expression\n\ndf.with_column(\"new_col\", col(\"a\") + lit(10))  # add one column\ndf.with_columns(\n    col(\"a\").alias(\"x\"),\n    y=col(\"b\") + lit(1),                    # named keyword form\n)\n\ndf.drop(\"unwanted_col\")\ndf.with_column_renamed(\"old_name\", \"new_name\")\n```\n\nWhen a column is referenced by name alone, pass the name as a string rather\nthan wrapping it in `col()`. Reach for `col()` only when the projection needs\narithmetic, aliasing, casting, or another expression operation.\n\n**Case sensitivity**: both `select(\"Name\")` and `col(\"Name\")` lowercase the\nidentifier. For a column whose real name has uppercase letters, embed double\nquotes inside the string: `select('\"MyCol\"')` or `col('\"MyCol\"')`. Without the\ninner quotes the lookup will fail with `No field named mycol`.\n\n### Filtering\n\n```python\ndf.filter(col(\"a\") > 10)\ndf.filter(col(\"a\") > 10, col(\"b\") == \"x\")   # multiple = AND\ndf.filter(\"a > 10\")                          # SQL expression string\n```\n\nRaw Python values on the right-hand side of a comparison are auto-wrapped\ninto literals by the `Expr` operators, so prefer `col(\"a\") > 10` over\n`col(\"a\") > lit(10)`. See the Comparisons section and pitfall #2 for the\nfull rule.\n\n### Aggregation\n\n```python\n# GROUP BY a, compute sum(b) and count(*)\ndf.aggregate([\"a\"], [F.sum(col(\"b\")), F.count(col(\"a\"))])\n\n# HAVING equivalent: use the filter keyword on the aggregate function\ndf.aggregate(\n    [\"region\"],\n    [F.sum(col(\"sales\"), filter=col(\"sales\") > 1000).alias(\"large_sales\")],\n)\n```\n\nAs with `select()`, group keys can be passed as plain name strings. Reach for\n`col(...)` only when the grouping expression needs arithmetic, aliasing,\ncasting, or another expression operation.\n\nMost aggregate functions accept an optional `filter` keyword argument. When\nprovided, only rows where the filter expression is true contribute to the\naggregate.\n\n### Sorting\n\n```python\ndf.sort(\"a\")                                 # ascending (plain name, preferred)\ndf.sort(col(\"a\"))                            # ascending via col()\ndf.sort(col(\"a\").sort(ascending=False))      # descending\ndf.sort(col(\"a\").sort(nulls_first=False))    # override null placement\n\ndf.sort_by(\"a\", \"b\")                         # ascending-only shortcut\n```\n\nAs with `select()` and `aggregate()`, bare column references can be passed as\nplain name strings. A plain expression passed to `sort()` is already treated\nas ascending, so reach for `col(...).sort(...)` only when you need to override\na default (descending order or null placement). Writing\n`col(\"a\").sort(ascending=True)` is redundant.\n\nFor ascending-only sorts with no null-placement override, `df.sort_by(...)` is\na shorter alias for `df.sort(...)`.\n\n### Joining\n\n```python\n# Equi-join on shared column name\ndf1.join(df2, on=\"key\")\ndf1.join(df2, on=\"key\", how=\"left\")\n\n# Different column names\ndf1.join(df2, left_on=\"id\", right_on=\"fk_id\", how=\"inner\")\n\n# Expression-based join (supports inequality predicates)\ndf1.join_on(df2, col(\"a\") == col(\"b\"), how=\"inner\")\n\n# Semi join: keep rows from left where a match exists in right (like EXISTS)\ndf1.join(df2, on=\"key\", how=\"semi\")\n\n# Anti join: keep rows from left where NO match exists in right (like NOT EXISTS)\ndf1.join(df2, on=\"key\", how=\"anti\")\n```\n\nJoin types: `\"inner\"`, `\"left\"`, `\"right\"`, `\"full\"`, `\"semi\"`, `\"anti\"`.\n\nInner is the default `how`. Prefer `df1.join(df2, on=\"key\")` over\n`df1.join(df2, on=\"key\", how=\"inner\")` — drop `how=` unless you need a\nnon-inner join type.\n\nWhen the two sides' join columns have different native names, use\n`left_on=`\u002F`right_on=` with the original names rather than aliasing one side\nto match the other — see pitfall #7.\n\n### Window Functions\n\n```python\nfrom datafusion import WindowFrame\n\n# Row number partitioned by group, ordered by value\ndf.window(\n    F.row_number(\n        partition_by=[col(\"group\")],\n        order_by=[col(\"value\")],\n    ).alias(\"rn\")\n)\n\n# Using a Window object for reuse\nfrom datafusion.expr import Window\n\nwin = Window(\n    partition_by=[col(\"group\")],\n    order_by=[col(\"value\").sort(ascending=True)],\n)\ndf.select(\n    col(\"group\"),\n    col(\"value\"),\n    F.sum(col(\"value\")).over(win).alias(\"running_total\"),\n)\n\n# With explicit frame bounds\nwin = Window(\n    partition_by=[col(\"group\")],\n    order_by=[col(\"value\").sort(ascending=True)],\n    window_frame=WindowFrame(\"rows\", 0, None),  # current row to unbounded following\n)\n```\n\n### Set Operations\n\n```python\ndf1.union(df2)                          # UNION ALL (by position)\ndf1.union(df2, distinct=True)           # UNION DISTINCT\ndf1.union_by_name(df2)                  # match columns by name, not position\ndf1.intersect(df2)                      # INTERSECT ALL\ndf1.intersect(df2, distinct=True)       # INTERSECT (distinct)\ndf1.except_all(df2)                     # EXCEPT ALL\ndf1.except_all(df2, distinct=True)      # EXCEPT (distinct)\n```\n\n### Limit and Offset\n\n```python\ndf.limit(10)            # first 10 rows\ndf.limit(10, offset=20) # skip 20, then take 10\n```\n\n### Deduplication\n\n```python\ndf.distinct()           # remove duplicate rows\ndf.distinct_on(         # keep first row per group (like DISTINCT ON in Postgres)\n    [col(\"a\")],                     # uniqueness columns\n    [col(\"a\"), col(\"b\")],           # output columns\n    [col(\"b\").sort(ascending=True)], # which row to keep\n)\n```\n\n## Executing and Collecting Results\n\nDataFrames are lazy until you collect.\n\n```python\ndf.show()                               # print formatted table to stdout\nbatches = df.collect()                  # list[pa.RecordBatch]\narr = df.collect_column(\"col_name\")     # pa.Array | pa.ChunkedArray (single column)\ntable = df.to_arrow_table()             # pa.Table\npandas_df = df.to_pandas()              # pd.DataFrame\npolars_df = df.to_polars()              # pl.DataFrame\npy_dict = df.to_pydict()                # dict[str, list]\npy_list = df.to_pylist()                # list[dict]\ncount = df.count()                      # int\ndf = df.cache()                         # materialize in memory, return DataFrame\n```\n\n### Date and Timestamp Type Conversion\n\nThe Python type returned by `to_pydict()` \u002F `to_pylist()` depends on the Arrow\ncolumn type, and the mapping is inherited from PyArrow:\n\n| Arrow type | Python type returned |\n|---|---|\n| `timestamp(s)` \u002F `(ms)` \u002F `(us)` | `datetime.datetime` |\n| `timestamp(ns)` | `pandas.Timestamp` |\n| `date32` \u002F `date64` | `datetime.date` |\n| `duration(s)` \u002F `(ms)` \u002F `(us)` | `datetime.timedelta` |\n| `duration(ns)` | `pandas.Timedelta` |\n\nThe nanosecond-precision fallback to pandas types is the main surprise:\npandas is not a hard dependency of `datafusion`, but PyArrow reaches for it\nwhen `datetime.datetime` \u002F `datetime.timedelta` would lose precision (stdlib\ntypes only go to microseconds). If you need plain stdlib types, cast to a\ncoarser unit before collecting, e.g.\n`df.select(col(\"ts\").cast(pa.timestamp(\"us\")))`.\n\n`df.to_pandas()` has its own footgun for dates: pandas has no pure-date dtype,\nso a `date32`\u002F`date64` column comes back as an `object` column of\n`datetime.date` values rather than `datetime64[ns]`. If downstream code\nexpects a datetime column, cast on the DataFusion side first:\n`col(\"ship_date\").cast(pa.timestamp(\"ns\"))`.\n\n### Streaming Results\n\nPrefer streaming over `collect()` when the result is too large to materialize\nin memory, when you want to start processing before the query finishes, or\nwhen you may break out of the loop early. `execute_stream()` pulls one\n`RecordBatch` at a time from the execution plan rather than buffering the\nwhole result up front.\n\n```python\n# Single-partition stream; batch is a datafusion.RecordBatch\nstream = df.execute_stream()\nfor batch in stream:\n    process(batch.to_pyarrow())         # convert to pa.RecordBatch if needed\n\n# DataFrame is iterable directly (delegates to execute_stream)\nfor batch in df:\n    process(batch.to_pyarrow())\n\n# One stream per partition, for parallel consumption\nfor stream in df.execute_stream_partitioned():\n    for batch in stream:\n        process(batch.to_pyarrow())\n```\n\nAsync iteration is also supported via `async for batch in df: ...` (or\n`df.execute_stream()`), which is useful when batches are interleaved with\nother I\u002FO.\n\n### Caching Intermediate Results\n\n`df.cache()` materializes a DataFrame as an in-memory table and returns a new\nDataFrame backed by it. Reach for it when the same intermediate result feeds\nmultiple downstream queries — without `cache()`, each branch re-executes the\nfull upstream plan (re-reading files, recomputing filters\u002Faggregates).\n\n```python\nbase = (\n    ctx.read_parquet(\"orders.parquet\")\n    .filter(col(\"status\") == \"shipped\")\n    .cache()                                # materialize once, reuse below\n)\nby_region = base.aggregate([\"region\"], [F.sum(col(\"amount\")).alias(\"total\")])\nby_customer = base.aggregate([\"customer\"], [F.sum(col(\"amount\")).alias(\"total\")])\n```\n\nSkip `cache()` for single-use DataFrames — the lazy plan is already optimal.\n\nThe cached table is owned by the DataFrame returned from `cache()` (and any\nDataFrames chained from it). To free the memory, drop every reference — let\nthem go out of scope, or `del base; del by_region; del by_customer`.\n\n### Writing Results\n\n```python\ndf.write_parquet(\"output.parquet\")\ndf.write_csv(\"output.csv\")\ndf.write_json(\"output.json\")\n```\n\nYou can also pass a directory path (e.g., `\"output\u002F\"`) to write a multi-file\npartitioned output.\n\n## Expression Building\n\n### Column References and Literals\n\n```python\ncol(\"column_name\")              # reference a column\nlit(42)                          # integer literal\nlit(\"hello\")                     # string literal\nlit(3.14)                        # float literal\nlit(pa.scalar(value))            # PyArrow scalar (preserves Arrow type)\n```\n\n`lit()` accepts PyArrow scalars directly -- prefer this over converting Arrow\ndata to Python and back when working with values extracted from query results.\n\n### Arithmetic\n\n```python\ncol(\"price\") * col(\"quantity\")            # multiplication\ncol(\"a\") + lit(1)                          # addition\ncol(\"a\") - col(\"b\")                        # subtraction\ncol(\"a\") \u002F lit(2)                          # division\ncol(\"a\") % lit(3)                          # modulo\n```\n\n### Date Arithmetic\n\n`Date32` and `Date64` columns both require `Interval` types for arithmetic,\nnot `Duration`. Use PyArrow's `month_day_nano_interval` type, which takes a\n`(months, days, nanos)` tuple:\n\n```python\nimport pyarrow as pa\n\n# Subtract 90 days from a date column\ncol(\"ship_date\") - lit(pa.scalar((0, 90, 0), type=pa.month_day_nano_interval()))\n\n# Subtract 3 months\ncol(\"ship_date\") - lit(pa.scalar((3, 0, 0), type=pa.month_day_nano_interval()))\n```\n\n**Important**: `lit(datetime.timedelta(days=90))` creates a `Duration(µs)`\nliteral, which is **not** compatible with `Date32`\u002F`Date64` arithmetic\n(`Duration(ms)` and `Duration(ns)` are rejected too). Always use\n`pa.month_day_nano_interval()` for date operations.\n\n**Timestamps behave differently**: `Timestamp` columns *do* accept `Duration`,\nso `col(\"ts\") - lit(datetime.timedelta(days=1))` works. The interval-only\nrule applies specifically to date columns.\n\n### Comparisons\n\n```python\ncol(\"a\") > 10\ncol(\"a\") >= 10\ncol(\"a\") \u003C 10\ncol(\"a\") \u003C= 10\ncol(\"a\") == \"x\"\ncol(\"a\") != \"x\"\ncol(\"a\") == None                           # same as col(\"a\").is_null()\ncol(\"a\") != None                           # same as col(\"a\").is_not_null()\n```\n\nComparison operators auto-wrap the right-hand Python value into a literal,\nso writing `col(\"a\") > lit(10)` is redundant. Drop the `lit()` in\ncomparisons. Reach for `lit()` only when auto-wrapping does not apply — see\npitfall #2.\n\n### Boolean Logic\n\n**Important**: Python's `and`, `or`, `not` keywords do NOT work with Expr\nobjects. You must use the bitwise operators:\n\n```python\n(col(\"a\") > 1) & (col(\"b\") \u003C 10)   # AND\n(col(\"a\") > 1) | (col(\"b\") \u003C 10)   # OR\n~(col(\"a\") > 1)                    # NOT\n```\n\nAlways wrap each comparison in parentheses when combining with `&`, `|`, `~`\nbecause Python's operator precedence for bitwise operators is different from\nlogical operators.\n\n### Null Handling\n\n```python\ncol(\"a\").is_null()\ncol(\"a\").is_not_null()\ncol(\"a\").fill_null(lit(0))          # replace NULL with a value (single expression)\nF.coalesce(col(\"a\"), col(\"b\"))     # first non-null value\nF.nullif(col(\"a\"), lit(0))         # return NULL if a == 0\n```\n\nTo fill nulls across the whole DataFrame (optionally limited to a subset of\ncolumns), use the DataFrame-level method:\n\n```python\ndf.fill_null(0)                     # every column\ndf.fill_null(0, subset=[\"a\", \"b\"])  # only these columns\n```\n\n### CASE \u002F WHEN\n\n```python\n# Simple CASE (matching on a single expression)\nstatus_label = (\n    F.case(col(\"status\"))\n    .when(lit(\"A\"), lit(\"Active\"))\n    .when(lit(\"I\"), lit(\"Inactive\"))\n    .otherwise(lit(\"Unknown\"))\n)\n\n# Searched CASE (each branch has its own predicate)\nseverity = (\n    F.when(col(\"value\") > 100, lit(\"high\"))\n    .when(col(\"value\") > 50, lit(\"medium\"))\n    .otherwise(lit(\"low\"))\n)\n```\n\n### Casting\n\n```python\nimport pyarrow as pa\n\ncol(\"a\").cast(pa.float64())\ncol(\"a\").cast(pa.utf8())\ncol(\"a\").cast(pa.date32())\n\ncol(\"a\").try_cast(pa.int32())        # like cast(), but yields NULL on failure instead of erroring\n```\n\nTo cast several columns at once at the DataFrame level, pass a mapping to\n`df.cast(...)`:\n\n```python\ndf.cast({\"a\": pa.float64(), \"b\": pa.int32()})\n```\n\n### Aliasing\n\n```python\n(col(\"a\") + col(\"b\")).alias(\"total\")\n```\n\n### BETWEEN and IN\n\n```python\ncol(\"a\").between(1, 10)                                 # 1 \u003C= a \u003C= 10 (bounds auto-wrap)\nF.in_list(col(\"a\"), [lit(1), lit(2), lit(3)])           # a IN (1, 2, 3)\nF.in_list(col(\"a\"), [lit(1), lit(2)], negated=True)     # a NOT IN (1, 2)\n```\n\n### Struct and Array Access\n\n```python\ncol(\"struct_col\")[\"field_name\"]     # access struct field\ncol(\"array_col\")[0]                  # access array element (0-indexed)\ncol(\"array_col\")[1:3]                # array slice (0-indexed)\n```\n\n### Lambda Functions\n\nSome array functions take a lambda function that runs once per element. Pass a\nPython `lambda` directly — its parameter names become the lambda parameters and\nits return value becomes the body:\n\n```python\nF.array_transform(col(\"a\"), lambda v: v * 2)    # map: [1,2,3] -> [2,4,6]\nF.array_filter(col(\"a\"), lambda v: v > 2)        # filter: [1,2,3] -> [3]\nF.array_any_match(col(\"a\"), lambda v: v > 3)     # predicate: any element > 3\n```\n\nFor explicit parameter names, build the lambda by hand:\n\n```python\nF.array_transform(col(\"a\"), F.lambda_([\"v\"], F.lambda_var(\"v\") * lit(2)))\n```\n\n## SQL-to-DataFrame Reference\n\n| SQL | DataFrame API |\n|---|---|\n| `SELECT a, b` | `df.select(\"a\", \"b\")` |\n| `SELECT a, b + 1 AS c` | `df.select(col(\"a\"), (col(\"b\") + lit(1)).alias(\"c\"))` |\n| `SELECT *, a + 1 AS c` | `df.with_column(\"c\", col(\"a\") + lit(1))` |\n| `WHERE a > 10` | `df.filter(col(\"a\") > 10)` |\n| `GROUP BY a` with `SUM(b)` | `df.aggregate([\"a\"], [F.sum(col(\"b\"))])` |\n| `SUM(b) FILTER (WHERE b > 100)` | `F.sum(col(\"b\"), filter=col(\"b\") > 100)` |\n| `ORDER BY a DESC` | `df.sort(col(\"a\").sort(ascending=False))` |\n| `LIMIT 10 OFFSET 5` | `df.limit(10, offset=5)` |\n| `DISTINCT` | `df.distinct()` |\n| `a INNER JOIN b ON a.id = b.id` | `a.join(b, on=\"id\")` |\n| `a LEFT JOIN b ON a.id = b.fk` | `a.join(b, left_on=\"id\", right_on=\"fk\", how=\"left\")` |\n| `WHERE EXISTS (SELECT ...)` | `a.join(b, on=\"key\", how=\"semi\")` |\n| `WHERE NOT EXISTS (SELECT ...)` | `a.join(b, on=\"key\", how=\"anti\")` |\n| `UNION ALL` | `df1.union(df2)` |\n| `UNION` (distinct) | `df1.union(df2, distinct=True)` |\n| `INTERSECT ALL` | `df1.intersect(df2)` |\n| `INTERSECT` (distinct) | `df1.intersect(df2, distinct=True)` |\n| `EXCEPT ALL` | `df1.except_all(df2)` |\n| `EXCEPT` (distinct) | `df1.except_all(df2, distinct=True)` |\n| `CASE x WHEN 1 THEN 'a' END` | `F.case(col(\"x\")).when(lit(1), lit(\"a\")).end()` |\n| `CASE WHEN x > 1 THEN 'a' END` | `F.when(col(\"x\") > 1, lit(\"a\")).end()` |\n| `x IN (1, 2, 3)` | `F.in_list(col(\"x\"), [lit(1), lit(2), lit(3)])` |\n| `x BETWEEN 1 AND 10` | `col(\"x\").between(1, 10)` |\n| `CAST(x AS DOUBLE)` | `col(\"x\").cast(pa.float64())` |\n| `ROW_NUMBER() OVER (...)` | `F.row_number(partition_by=[...], order_by=[...])` |\n| `SUM(x) OVER (...)` | `F.sum(col(\"x\")).over(window)` |\n| `x IS NULL` | `col(\"x\").is_null()` |\n| `COALESCE(a, b)` | `F.coalesce(col(\"a\"), col(\"b\"))` |\n\n## Common Pitfalls\n\n1. **Boolean operators**: Use `&`, `|`, `~` -- not Python's `and`, `or`, `not`.\n   Always parenthesize: `(col(\"a\") > 1) & (col(\"b\") \u003C 2)`.\n\n2. **Wrapping scalars with `lit()`**: Prefer raw Python values on the\n   right-hand side of comparisons — `col(\"a\") > 10`, `col(\"name\") == \"Alice\"`\n   — because the Expr comparison operators auto-wrap them. Writing\n   `col(\"a\") > lit(10)` is redundant. Reserve `lit()` for places where\n   auto-wrapping does *not* apply:\n   - standalone scalars passed into function calls:\n     `F.coalesce(col(\"a\"), lit(0))`, not `F.coalesce(col(\"a\"), 0)`\n   - arithmetic between two literals with no column involved:\n     `lit(1) - col(\"discount\")` is fine, but `lit(1) - lit(2)` needs both\n   - values that must carry a specific Arrow type, via `lit(pa.scalar(...))`\n   - `.when(...)`, `.otherwise(...)`, `F.nullif(...)`, `F.in_list(...)`\n     and similar method\u002Ffunction arguments (note: `.between(...)`\n     auto-wraps its bounds, so `col(\"a\").between(1, 10)` needs no `lit()`)\n\n3. **Column name quoting**: Column names are normalized to lowercase by default\n   in both `select(\"...\")` and `col(\"...\")`. To reference a column with\n   uppercase letters, use double quotes inside the string:\n   `select('\"MyColumn\"')` or `col('\"MyColumn\"')`.\n\n4. **DataFrames are immutable**: Every method returns a **new** DataFrame. You\n   must capture the return value:\n   ```python\n   df = df.filter(col(\"a\") > 1)   # correct\n   df.filter(col(\"a\") > 1)         # WRONG -- result is discarded\n   ```\n\n5. **Window frame defaults**: When using `order_by` in a window, the default\n   frame is `RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`. For a full\n   partition frame, set `window_frame=WindowFrame(\"rows\", None, None)`.\n\n6. **Arithmetic on aggregates belongs in a later `select`, not inside\n   `aggregate`** *(applies to datafusion-python 53 and earlier; fixed in 54)*:\n   Each item in the aggregate list must be a single aggregate\n   call (optionally aliased). Combining aggregates with arithmetic inside\n   `aggregate(...)` fails with `Internal error: Invalid aggregate expression`.\n   Alias the aggregates, then compute the combination downstream:\n   ```python\n   # WRONG -- arithmetic wraps two aggregates\n   df.aggregate([], [(lit(100) * F.sum(col(\"a\")) \u002F F.sum(col(\"b\"))).alias(\"ratio\")])\n\n   # CORRECT -- aggregate first, then combine\n   (df.aggregate([], [F.sum(col(\"a\")).alias(\"num\"), F.sum(col(\"b\")).alias(\"den\")])\n      .select((lit(100) * col(\"num\") \u002F col(\"den\")).alias(\"ratio\")))\n   ```\n\n7. **Don't alias a join column to match the other side**: When equi-joining\n   with `on=\"key\"`, renaming the join column on one side via `.alias(\"key\")`\n   in a fresh projection creates a schema where one side's `key` is\n   qualified (`?table?.key`) and the other is unqualified. The join then\n   fails with `Schema contains qualified field name ... and unqualified\n   field name ... which would be ambiguous`. Use `left_on=`\u002F`right_on=` with\n   the native names, or use `join_on(...)` with an explicit equality.\n   ```python\n   # WRONG -- alias on one side produces ambiguous schema after join\n   failed = orders.select(col(\"o_orderkey\").alias(\"l_orderkey\"))\n   li.join(failed, on=\"l_orderkey\")   # ambiguous l_orderkey error\n\n   # CORRECT -- keep native names, use left_on\u002Fright_on\n   failed = orders.select(\"o_orderkey\")\n   li.join(failed, left_on=\"l_orderkey\", right_on=\"o_orderkey\")\n\n   # ALSO CORRECT -- explicit predicate via join_on\n   # (note: join_on keeps both key columns in the output, unlike on=\"key\")\n   li.join_on(failed, col(\"l_orderkey\") == col(\"o_orderkey\"))\n   ```\n   When the same column name exists on both sides, `DataFrame.col(name)`\n   (and `DataFrame.column(name)`) returns a column reference qualified to\n   that DataFrame, which disambiguates the predicate explicitly:\n   ```python\n   li.join_on(failed, li.col(\"l_orderkey\") == failed.col(\"o_orderkey\"))\n   ```\n\n## Idiomatic Patterns\n\n### Fluent Chaining\n\n```python\nresult = (\n    ctx.read_parquet(\"data.parquet\")\n    .filter(col(\"year\") >= 2020)\n    .select(col(\"region\"), col(\"sales\"))\n    .aggregate([\"region\"], [F.sum(col(\"sales\")).alias(\"total\")])\n    .sort(col(\"total\").sort(ascending=False))\n    .limit(10)\n)\nresult.show()\n```\n\n### Using Variables as CTEs\n\nInstead of SQL CTEs (`WITH ... AS`), assign intermediate DataFrames to\nvariables:\n\n```python\nbase = ctx.read_parquet(\"orders.parquet\").filter(col(\"status\") == \"shipped\")\nby_region = base.aggregate([\"region\"], [F.sum(col(\"amount\")).alias(\"total\")])\ntop_regions = by_region.filter(col(\"total\") > 10000)\n```\n\n### Reusing Expressions as Variables\n\nJust like DataFrames, expressions (`Expr`) can be stored in variables and used\nanywhere an `Expr` is expected. This is useful for building up complex\nexpressions or reusing a computed value across multiple operations:\n\n```python\n# Build an expression and reuse it\ndisc_price = col(\"price\") * (lit(1) - col(\"discount\"))\ndf = df.select(\n    col(\"id\"),\n    disc_price.alias(\"disc_price\"),\n    (disc_price * (lit(1) + col(\"tax\"))).alias(\"total\"),\n)\n\n# Use a collected scalar as an expression\nmax_val = result_df.collect_column(\"max_price\")[0]   # PyArrow scalar\ncutoff = lit(max_val) - lit(pa.scalar((0, 90, 0), type=pa.month_day_nano_interval()))\ndf = df.filter(col(\"ship_date\") \u003C= cutoff)           # cutoff is already an Expr\n```\n\n**Important**: Do not wrap an `Expr` in `lit()`. `lit()` is for converting\nPython\u002FPyArrow values into expressions. If a value is already an `Expr`, use it\ndirectly.\n\n### Window Functions for Scalar Subqueries\n\nWhere SQL uses a correlated scalar subquery, the idiomatic DataFrame approach\nis a window function:\n\n```sql\n-- SQL scalar subquery\nSELECT *, (SELECT SUM(b) FROM t WHERE t.group = s.group) AS group_total FROM s\n```\n\n```python\n# DataFrame: window function\nwin = Window(partition_by=[col(\"group\")])\ndf = df.with_column(\"group_total\", F.sum(col(\"b\")).over(win))\n```\n\n### Semi\u002FAnti Joins for EXISTS \u002F NOT EXISTS\n\n```sql\n-- SQL: WHERE EXISTS (SELECT 1 FROM other WHERE other.key = main.key)\n-- DataFrame:\nresult = main.join(other, on=\"key\", how=\"semi\")\n\n-- SQL: WHERE NOT EXISTS (SELECT 1 FROM other WHERE other.key = main.key)\n-- DataFrame:\nresult = main.join(other, on=\"key\", how=\"anti\")\n```\n\n### Computed Columns\n\n```python\n# Add computed columns while keeping all originals\ndf = df.with_column(\"full_name\", F.concat(col(\"first\"), lit(\" \"), col(\"last\")))\ndf = df.with_column(\"discounted\", col(\"price\") * lit(0.9))\n```\n\n## Available Functions (Categorized)\n\nThe `functions` module (imported as `F`) provides 290+ functions. Key categories:\n\n**Aggregate**: `sum`, `avg`, `min`, `max`, `count`, `count_star`, `median`,\n`stddev`, `stddev_pop`, `var_samp`, `var_pop`, `corr`, `covar`, `approx_distinct`,\n`approx_median`, `approx_percentile_cont`, `array_agg`, `string_agg`,\n`first_value`, `last_value`, `bit_and`, `bit_or`, `bit_xor`, `bool_and`,\n`bool_or`, `grouping`, `regr_*` (9 regression functions)\n\n**Window**: `row_number`, `rank`, `dense_rank`, `percent_rank`, `cume_dist`,\n`ntile`, `lag`, `lead`, `first_value`, `last_value`, `nth_value`\n\n**String**: `length`, `lower`, `upper`, `trim`, `ltrim`, `rtrim`, `lpad`,\n`rpad`, `starts_with`, `ends_with`, `contains`, `substr`, `substring`,\n`replace`, `reverse`, `repeat`, `split_part`, `concat`, `concat_ws`,\n`initcap`, `ascii`, `chr`, `left`, `right`, `strpos`, `translate`, `overlay`,\n`levenshtein`\n\n`F.substr(str, start)` returns the tail of the string from `start` onward;\n`F.substr(str, start, length=n)` returns `n` characters from `start`, like\nSQL's `SUBSTRING(str FROM start FOR length)`. `start` and `length` accept\nnative ints. `F.substring(str, start, length)` is the same with `length`\nrequired. For a fixed-length prefix, `F.left(col(\"s\"), lit(n))` is cleanest.\n\n*The `length` argument to `substr` requires datafusion-python 55 or newer.\nOn earlier versions a third argument raises `TypeError: substr() takes 2\npositional arguments but 3 were given`; use `F.substring` there.*\n\n```python\nF.substr(col(\"c_phone\"), 1, length=2)         # first 2 characters\nF.substring(col(\"c_phone\"), lit(1), lit(2))   # same, on any version\nF.left(col(\"c_phone\"), lit(2))                # prefix shortcut\n```\n\n**Math**: `abs`, `ceil`, `floor`, `round`, `trunc`, `sqrt`, `cbrt`, `exp`,\n`ln`, `log`, `log2`, `log10`, `pow`, `signum`, `pi`, `random`, `factorial`,\n`gcd`, `lcm`, `greatest`, `least`, sin\u002Fcos\u002Ftan and inverse\u002Fhyperbolic variants\n\n**Date\u002FTime**: `now`, `today`, `current_date`, `current_time`,\n`current_timestamp`, `date_part`, `date_trunc`, `date_bin`, `extract`,\n`to_timestamp`, `to_timestamp_millis`, `to_timestamp_micros`,\n`to_timestamp_nanos`, `to_timestamp_seconds`, `to_unixtime`, `from_unixtime`,\n`make_date`, `make_time`, `to_date`, `to_time`, `to_local_time`, `date_format`\n\n**Conditional**: `case`, `when`, `coalesce`, `nullif`, `ifnull`, `nvl`, `nvl2`\n\n**Array\u002FList**: `array`, `make_array`, `array_agg`, `array_length`,\n`array_element`, `array_slice`, `array_append`, `array_prepend`,\n`array_concat`, `array_contains`, `array_has`, `array_has_all`, `array_has_any`, `array_position`,\n`array_remove`, `array_distinct`, `array_sort`, `array_reverse`, `flatten`,\n`array_to_string`, `array_intersect`, `array_union`, `array_except`,\n`generate_series`\n(Most `array_*` functions also have `list_*` aliases.)\n\n**Struct\u002FMap**: `struct`, `named_struct`, `get_field`, `make_map`, `map_keys`,\n`map_values`, `map_entries`, `map_extract`\n\n**Regex**: `regexp_like`, `regexp_match`, `regexp_replace`, `regexp_count`,\n`regexp_instr`\n\n**Hash**: `md5`, `sha224`, `sha256`, `sha384`, `sha512`, `digest`\n\n**Type**: `arrow_typeof`, `arrow_cast`, `arrow_try_cast`, `arrow_field`,\n`arrow_metadata`, `cast_to_type`, `with_metadata`\n\nNote: ``cast_to_type(value, type_ref, *, try_cast=False)`` is the single\nPython entry point for both upstream ``cast_to_type`` and ``try_cast_to_type``;\npass ``try_cast=True`` for the variant that returns NULL on failure.\n\n**Other**: `in_list`, `order_by`, `alias`, `col`, `encode`, `decode`,\n`to_hex`, `to_char`, `uuid`, `version`, `bit_length`, `octet_length`\n\n### Spark-Compatible Functions\n\nA separate `datafusion.functions.spark` namespace mirrors the\n`pyspark.sql.functions` API for callers porting code from PySpark.\n\n```python\nfrom datafusion.functions import spark\n```\n\nUse it for DataFrame work; for SQL, register the Spark UDFs first:\n\n```python\nctx = SessionContext()\nctx.enable_spark_functions()                  # makes Spark UDFs visible to SQL\nctx.sql(\"SELECT sha2('hello', 256)\").show()\n```\n\nCoverage spans aggregate, array, bitmap, bitwise, datetime, hash, JSON,\nmap, math, string, URL, and conditional categories. The authoritative\nlist of what is currently exposed is the `__all__` in\n`python\u002Fdatafusion\u002Ffunctions\u002Fspark.py`:\n\n```bash\npython -c \"from datafusion.functions import spark; print(sorted(spark.__all__))\"\n```\n\nWhen you need to know whether a specific pyspark function is available,\ncheck `__all__` rather than this skill — the list there moves with the\ncode; any enumeration here would drift.\n\n**Semantic divergences vs the default namespace.** Functions that exist in\nboth `functions` and `functions.spark` may behave differently:\n\n| Function | Default `functions` | `functions.spark` |\n|---|---|---|\n| `concat` | NULL inputs treated as empty | NULL inputs propagate to NULL |\n| `round` | HALF_EVEN (banker's) | HALF_UP |\n| `trunc` | Numeric truncation | Date truncation |\n\nPick the namespace whose semantics match your intent — both stay imported\nside by side; `enable_spark_functions()` only affects SQL.\n\n**Parameter names match pyspark exactly.** The spark namespace uses\npyspark parameter names (`col`, `str`, `numBits`, `partToExtract`, ...) so\nyou can paste pyspark code and keep keyword arguments working. The default\nnamespace keeps DataFusion's parameter names.\n",{"name":5,"description":7},"main","878e5d3b8d373bca19671af43e49dfb1890f8da2","skills\u002Fdatafusion_python",184,"https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-python","https:\u002F\u002Fdatafusion.apache.org\u002Fpython","https:\u002F\u002Fgithub.com\u002Fapache","https:\u002F\u002Fwww.apache.org\u002F","2026-11-09 03:42:43",{"source":8,"skillName":5,"commands":58,"prompt":68,"zipAvailable":69},[59,62,65],{"tool":60,"command":61},"npx","npx skills add apache\u002Fdatafusion-python --skill datafusion-python",{"tool":63,"command":64},"bunx","bunx skills add apache\u002Fdatafusion-python --skill datafusion-python",{"tool":66,"command":67},"pnpm","pnpm dlx skills add apache\u002Fdatafusion-python --skill datafusion-python","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-python\u002Fblob\u002Fmain\u002Fskills\u002Fdatafusion_python\u002FSKILL.md",true,[71,77,81],{"path":72,"resourceType":73,"resourceTypeDesc":74,"sizeBytes":75,"sourceHtmlUrl":76},"references\u002FcomponentWillMount.md","REFERENCE","参考文档",3776,"https:\u002F\u002Fgithub.com\u002Fgithub\u002Fawesome-copilot\u002Fblob\u002Fmain\u002Fskills\u002Freact18-lifecycle-patterns\u002Freferences\u002FcomponentWillMount.md",{"path":78,"resourceType":73,"resourceTypeDesc":74,"sizeBytes":79,"sourceHtmlUrl":80},"references\u002FcomponentWillReceiveProps.md",5085,"https:\u002F\u002Fgithub.com\u002Fgithub\u002Fawesome-copilot\u002Fblob\u002Fmain\u002Fskills\u002Freact18-lifecycle-patterns\u002Freferences\u002FcomponentWillReceiveProps.md",{"path":82,"resourceType":73,"resourceTypeDesc":74,"sizeBytes":83,"sourceHtmlUrl":84},"references\u002FcomponentWillUpdate.md",4789,"https:\u002F\u002Fgithub.com\u002Fgithub\u002Fawesome-copilot\u002Fblob\u002Fmain\u002Fskills\u002Freact18-lifecycle-patterns\u002Freferences\u002FcomponentWillUpdate.md",[86,93,104,111],{"skillNo":87,"name":88,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":89,"categories":90,"keywords":91,"publishedAt":43},"01M6ACM17FMWAHVBVMWQZ816NF","make-pythonic","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-python\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Fmake-pythonic\u002FSKILL.md",[],[92],"make pythonic",{"skillNo":94,"name":95,"nameZh":6,"description":96,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":97,"categories":98,"keywords":99,"publishedAt":43},"01M6ACKZ12QZPH2YSN7ENQW13J","ffi-capsule-protocol","TRIGGER — read before adding, changing, or reviewing any __datafusion_*__ capsule getter, any FFI_* export that asks for a TaskContextProvider or an extension codec, or any code that calls FFI_QueryPlanner::new \u002F FFI_TableProvider::new \u002F FFI_{Logical,Physical}ExtensionCodec::new. These methods are one protocol with a settled convention. Do not design it fresh; do not construct a SessionContext inside an extension library.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-python\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Fffi-capsule-protocol\u002FSKILL.md",[],[100,101,102,103,40,36],"protocol","ffi capsule protocol","design","review",{"skillNo":105,"name":106,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":107,"categories":108,"keywords":109,"publishedAt":43},"01M6ACKX5JQ505Q8YX7BJTGQBM","check-upstream","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-python\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Fcheck-upstream\u002FSKILL.md",[],[110],"check upstream",{"skillNo":112,"name":113,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":114,"categories":115,"keywords":119,"publishedAt":43},"01M6ACKVJSNMT7YDTXEFJNZMY3","audit-skill-md","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-python\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Faudit-skill-md\u002FSKILL.md",[116],{"categoryNo":117,"categoryName":118,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发",[120],"audit skill md"]