[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6XETDDN3EXP7GF2KH71E638":3,"skill-same:01M6XETDDN3EXP7GF2KH71E638":71},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":22,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":44,"commitSha":45,"skillDirectory":46,"forksCount":13,"repositoryHtmlUrl":47,"repositoryHomepageUrl":6,"ownerHtmlUrl":48,"ownerBlogUrl":6,"ownerDescription":49,"updatedAt":50,"install":51,"resources":64},"01M6XETDDN3EXP7GF2KH71E638","data_cleaning",null,"Clean messy tabular datasets with deduplication, missing value imputation, outlier handling, and text processing. Use when dealing with dirty data that has duplicates, nulls, or inconsistent formatting.","Waaaalnut\u002Fskillsbench","Waaaalnut","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F103995656?v=4","01M6XCS9K1V51EY3JMB7DGMX24","01M6XCS9Q4WPW88W62C55G1H44",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks\u002Ftrend-anomaly-causal-inference\u002Fenvironment\u002Fskills\u002Fdata_cleaning\u002FSKILL.md",[17],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000033","数据分析师","PROFESSION","适用职业",[23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"data","data cleaning","deduplication","text","use","duplicates","datasets","clean","formatting","missing","imputation","processing","with","value","tabular","when","2026-11-10 10:12:30",false,"---\nname: data_cleaning\ndescription: Clean messy tabular datasets with deduplication, missing value imputation, outlier handling, and text processing. Use when dealing with dirty data that has duplicates, nulls, or inconsistent formatting.\n---\n\n# Data Cleaning Framework\n\nComprehensive, modular data cleaning framework for tabular datasets. Provides strategy-based cleaning operations including deduplication, missing value imputation, text processing, and outlier handling through a configurable pipeline.\n\n## Core Components\n\n### CleaningStrategies\nCollection of static methods for specific cleaning operations:\n\n#### Deduplication\n- `remove_duplicates(df, subset=None)` - Remove duplicate rows given the context of the prompt or the context of the dataset, Carefully evaluate if it still makes sense to allow duplicates exist in the downstream analysis. You should always check the data and remove them based on the logic of meta data and downstream goal.\n\n#### Missing Value Handling\nLogic: If the column is not critical to affect the dowstream task and the cost of dropping the whole line is not neccessary, you should try to impute them, otherwise you can drop.\n- `drop_missing(df, columns)` - Drop rows with missing critical values (that will affect the downstream task's peformance) in specified columns.\n- `impute_median(df, columns)` - Impute with median for numerical columns if the column is numerical data\n- `impute_knn(df, target_features, n_neighbors=5)` - KNN-based imputation such that the target column value potentially can be inferred by other columns. I.e. you have to carefully see the semantic and logical relationship between columns in terms of real life cases and make decisions to use this method for better data imputation performance.\n- `impute_mode(df, columns)` - Impute with mode for categorical columns if the target column is relatively independent to other columns and should just be filled with an acceptable value.\n\n#### Text Processing\n- `process_text(df, columns, operation)` - Text operations: some columns might need conduct text related processing to get numbers or remove unnecessary content, you need to dertermine if this is needed based on preview the column values regarding the down stream goal. (extract_numbers, clean_whitespace, extract_email, lowercase, remove_special)\n\n#### Outlier Handling\nLogic: Again you need to make sure if removing outliers is costly or not in terms of the sample size or critical columns under your investigation. Keep the rows by replacing the outlier value with a more reasonable value, or simply remove the whole row in one of the two ways if it barely has any impact.\n- `cap_outliers_iqr(df, columns, multiplier=1.5)` - Winsorization using IQR to keep reasonable signals of the data;\n- `remove_outliers_iqr(df, columns, multiplier=1.5)` - Remove outliers using IQR\n- `remove_outliers_zscore(df, columns, threshold=3.0)` - Remove outliers using Z-score\n\n### DataCleaningPipeline\nOrchestrates multiple cleaning steps with logging and progress tracking.\n\n## Usage\n\n```python\nfrom data_cleaning import CleaningStrategies, DataCleaningPipeline\n\n# Create pipeline\npipeline = DataCleaningPipeline(name=\"Transaction Data\")\n\n# Add cleaning steps\npipeline.add_step(\n    CleaningStrategies.remove_duplicates,\n    subset=['customer_id', 'order_date'],\n    description=\"Remove duplicate orders\"\n).add_step(\n    CleaningStrategies.drop_missing,\n    columns=['customer_id', 'amount'],\n    description=\"Drop rows with missing critical fields\"\n).add_step(\n    CleaningStrategies.impute_knn,\n    target_features={\n        'income': {'features': ['age', 'education'], 'type': 'numeric'}\n    },\n    description=\"KNN imputation for income\"\n).add_step(\n    CleaningStrategies.cap_outliers_iqr,\n    columns=['amount'],\n    multiplier=1.5,\n    description=\"Cap price outliers\"\n)\n\n# Execute pipeline\ndf_clean = pipeline.execute(raw_df, verbose=True)\n\n# Get execution log\nlog_df = pipeline.get_log()\n```\n\n## Input\n- Any tabular DataFrame requiring cleaning\n\n## Output\n- Cleaned DataFrame with execution log available\n\n## Best Practices\n- Chain multiple steps using method chaining\n- Use verbose=True to monitor pipeline progress\n- Review execution log for quality assurance\n- Extend CleaningStrategies with domain-specific methods as needed\n","# Data Cleaning Framework\n\nComprehensive, modular data cleaning framework for tabular datasets. Provides strategy-based cleaning operations including deduplication, missing value imputation, text processing, and outlier handling through a configurable pipeline.\n\n## Core Components\n\n### CleaningStrategies\nCollection of static methods for specific cleaning operations:\n\n#### Deduplication\n- `remove_duplicates(df, subset=None)` - Remove duplicate rows given the context of the prompt or the context of the dataset, Carefully evaluate if it still makes sense to allow duplicates exist in the downstream analysis. You should always check the data and remove them based on the logic of meta data and downstream goal.\n\n#### Missing Value Handling\nLogic: If the column is not critical to affect the dowstream task and the cost of dropping the whole line is not neccessary, you should try to impute them, otherwise you can drop.\n- `drop_missing(df, columns)` - Drop rows with missing critical values (that will affect the downstream task's peformance) in specified columns.\n- `impute_median(df, columns)` - Impute with median for numerical columns if the column is numerical data\n- `impute_knn(df, target_features, n_neighbors=5)` - KNN-based imputation such that the target column value potentially can be inferred by other columns. I.e. you have to carefully see the semantic and logical relationship between columns in terms of real life cases and make decisions to use this method for better data imputation performance.\n- `impute_mode(df, columns)` - Impute with mode for categorical columns if the target column is relatively independent to other columns and should just be filled with an acceptable value.\n\n#### Text Processing\n- `process_text(df, columns, operation)` - Text operations: some columns might need conduct text related processing to get numbers or remove unnecessary content, you need to dertermine if this is needed based on preview the column values regarding the down stream goal. (extract_numbers, clean_whitespace, extract_email, lowercase, remove_special)\n\n#### Outlier Handling\nLogic: Again you need to make sure if removing outliers is costly or not in terms of the sample size or critical columns under your investigation. Keep the rows by replacing the outlier value with a more reasonable value, or simply remove the whole row in one of the two ways if it barely has any impact.\n- `cap_outliers_iqr(df, columns, multiplier=1.5)` - Winsorization using IQR to keep reasonable signals of the data;\n- `remove_outliers_iqr(df, columns, multiplier=1.5)` - Remove outliers using IQR\n- `remove_outliers_zscore(df, columns, threshold=3.0)` - Remove outliers using Z-score\n\n### DataCleaningPipeline\nOrchestrates multiple cleaning steps with logging and progress tracking.\n\n## Usage\n\n```python\nfrom data_cleaning import CleaningStrategies, DataCleaningPipeline\n\n# Create pipeline\npipeline = DataCleaningPipeline(name=\"Transaction Data\")\n\n# Add cleaning steps\npipeline.add_step(\n    CleaningStrategies.remove_duplicates,\n    subset=['customer_id', 'order_date'],\n    description=\"Remove duplicate orders\"\n).add_step(\n    CleaningStrategies.drop_missing,\n    columns=['customer_id', 'amount'],\n    description=\"Drop rows with missing critical fields\"\n).add_step(\n    CleaningStrategies.impute_knn,\n    target_features={\n        'income': {'features': ['age', 'education'], 'type': 'numeric'}\n    },\n    description=\"KNN imputation for income\"\n).add_step(\n    CleaningStrategies.cap_outliers_iqr,\n    columns=['amount'],\n    multiplier=1.5,\n    description=\"Cap price outliers\"\n)\n\n# Execute pipeline\ndf_clean = pipeline.execute(raw_df, verbose=True)\n\n# Get execution log\nlog_df = pipeline.get_log()\n```\n\n## Input\n- Any tabular DataFrame requiring cleaning\n\n## Output\n- Cleaned DataFrame with execution log available\n\n## Best Practices\n- Chain multiple steps using method chaining\n- Use verbose=True to monitor pipeline progress\n- Review execution log for quality assurance\n- Extend CleaningStrategies with domain-specific methods as needed\n",{"name":5,"description":7},"main","7eeb258ada72b7f58119b0e7a2c00ea89d9beedd","tasks\u002Ftrend-anomaly-causal-inference\u002Fenvironment\u002Fskills\u002Fdata_cleaning","https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench","https:\u002F\u002Fgithub.com\u002FWaaaalnut","student of Zhejiang University","2026-04-21 23:16:23",{"source":8,"skillName":5,"commands":52,"prompt":62,"zipAvailable":63},[53,56,59],{"tool":54,"command":55},"npx","npx skills add Waaaalnut\u002Fskillsbench --skill data_cleaning",{"tool":57,"command":58},"bunx","bunx skills add Waaaalnut\u002Fskillsbench --skill data_cleaning",{"tool":60,"command":61},"pnpm","pnpm dlx skills add Waaaalnut\u002Fskillsbench --skill data_cleaning","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks\u002Ftrend-anomaly-causal-inference\u002Fenvironment\u002Fskills\u002Fdata_cleaning\u002FSKILL.md",true,[65],{"path":66,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":69,"sourceHtmlUrl":70},"scripts\u002Fdata_cleaning.py","SCRIPT","脚本",15777,"https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks\u002Ftrend-anomaly-causal-inference\u002Fenvironment\u002Fskills\u002Fdata_cleaning\u002Fscripts\u002Fdata_cleaning.py",[72,81,95,121,143,161],{"skillNo":73,"name":74,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":75,"categories":76,"keywords":78,"publishedAt":39},"01M6XEWMJ6VY3ZC53MPDCPG3GH","data-reconciliation","https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks\u002Fxlsx-recover-data\u002Fenvironment\u002Fskills\u002Fdata-reconciliation\u002FSKILL.md",[77],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},[79,80,23],"data reconciliation","reconciliation",{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":39},"01M6XEW8MNPMF1KRXV6ATGFRQF","speech-to-text","Transcribe video to timestamped text using Whisper tiny model (pre-installed).","https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks\u002Fvideo-tutorial-indexer\u002Fenvironment\u002Fskills\u002Fspeech-to-text\u002FSKILL.md",[],[88,26,89,90,91,92,93,94],"speech","speech to text","transcribe video","whisper","model","Video","transcribe",{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":99,"categories":100,"keywords":106,"publishedAt":39},"01M6XEW3FQZP0T1RKQQJFXVY9T","video-processor","Process videos by removing segments and concatenating remaining parts. Use when you need to remove detected pauses\u002Fopenings from videos, create highlight reels, or batch process segment removals using ffmpeg filter_complex.","https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks\u002Fvideo-silence-remover\u002Fenvironment\u002Fskills\u002Fvideo-processor\u002FSKILL.md",[101],{"categoryNo":102,"categoryName":103,"taxonomyType":104,"taxonomyTypeDesc":105},"01JBSEEDCAT000000000000016","图像与多媒体","SCENARIO","使用场景",[93,107,108,109,110,111,38,112,113,114,115,116,117,118,119,27,120],"processor","video processor","batch","create","reels","parts","segments","process","videos","you","segment","filter","ffmpeg","remove",{"skillNo":122,"name":123,"nameZh":6,"description":124,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":125,"categories":126,"keywords":128,"publishedAt":39},"01M6XEVXJ5KZ0NTY3XZKRY4AS1","silence-detector","Detect initial silence segments in audio\u002Fvideo using energy-based analysis. Use when you need to find low-energy periods at the start of recordings (title slides, setup time, pre-roll silence).","https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks\u002Fvideo-silence-remover\u002Fenvironment\u002Fskills\u002Fsilence-detector\u002FSKILL.md",[127],{"categoryNo":102,"categoryName":103,"taxonomyType":104,"taxonomyTypeDesc":105},[129,130,131,132,38,116,133,134,135,136,137,113,138,139,93,140,141,142,27],"detector","silence detector","title","slides","time","energy","at","recordings","detect","find","Audio","analysis","start","setup",{"skillNo":144,"name":145,"nameZh":6,"description":146,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":147,"categories":148,"keywords":149,"publishedAt":39},"01M6XEVT3NGRN1FT7JPMAPXS6M","segment-combiner","Combine multiple segment detection results into a unified list. Use when you need to merge segments from different detectors, prepare removal lists for video processing, or consolidate detection outputs.","https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks\u002Fvideo-silence-remover\u002Fenvironment\u002Fskills\u002Fsegment-combiner\u002FSKILL.md",[],[117,150,93,151,152,153,154,155,38,156,157,113,116,34,158,27,159,160],"segment combiner","detection","prepare","merge","video processing","for","outputs","combine","consolidate","results","list",{"skillNo":162,"name":163,"nameZh":6,"description":164,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":165,"categories":166,"keywords":170,"publishedAt":39},"01M6XEVMPVASGS2KX86QK4FMT7","report-generator","Generate compression reports for video processing. Use when you need to create structured JSON reports with duration statistics, compression ratios, and segment details after video processing.","https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks\u002Fvideo-silence-remover\u002Fenvironment\u002Fskills\u002Freport-generator\u002FSKILL.md",[167],{"categoryNo":168,"categoryName":169,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000034","行政与办公人员",[171,172,173,117,38,155,116,35,34,174,175,176,27,154,177,93,178,179,110],"generator","report generator","report","statistics","structured","reports","json","compression","generate"]