pandoc-unicode-sanitize
Pre-process Markdown to replace non-ASCII characters before pandoc PDF conversion
- License 许可
- MIT
- 所属仓库
- HKUDS/AgentPilot
- 最近更新
- 2026-11-12
- Stars
- ★ 8.0k
- Forks
- 944
01 前言 Front Matter
- name
- pandoc-unicode-sanitize
- description
- Pre-process Markdown to replace non-ASCII characters before pandoc PDF conversion
02 资源文件 1
- 其他.skill_id37 B
03 SKILL.md 正文
Pandoc Unicode Sanitization for PDF Conversion
Problem
LaTeX (used by pandoc for PDF output) may fail or produce errors when encountering non-ASCII Unicode characters like currency symbols (₹, €, £), special quotes, or other Unicode characters not supported by the default LaTeX engine.
Solution
Pre-process Markdown files to replace problematic Unicode characters with ASCII equivalents before running pandoc PDF conversion.
Steps
1. Identify Problematic Characters
Common problematic characters include:
- Currency symbols: ₹, €, £, ¥
- Special quotes: " " ' '
- Arrows: →, ←, ↑, ↓
- Bullets: •, ◦
- Other symbols: ©, ®, ™, ±, ×, ÷, —, –
2. Create a Sed Substitution Script
Create a shell script for reusable sanitization:
#!/bin/bash
# sanitize-for-pdf.sh
sed -e 's/₹/Rs/g' \
-e 's/€/EUR/g' \
-e 's/£/GBP/g' \
-e 's/¥/JPY/g' \
-e 's/"/"/g' \
-e 's/"/"/g' \
-e "s/'/'/g" \
-e "s/'/'/g" \
-e 's/→/->/g' \
-e 's/←/<-/g' \
-e 's/•/-/g' \
-e 's/©/(c)/g' \
-e 's/®/(R)/g' \
-e 's/™/(TM)/g' \
-e 's/±/+/ -/g' \
-e 's/×/x/g' \
-e 's/÷/\//g' \
-e 's/—/--/g' \
-e 's/–/-/g' \
"$1"
3. Run Sanitization Before Pandoc
# Using the script
./sanitize-for-pdf.sh input.md > sanitized.md
pandoc sanitized.md -o output.pdf
# Or inline
cat input.md | sed 's/₹/Rs/g; s/€/EUR/g; s/£/GBP/g' | pandoc -o output.pdf
4. Alternative: Use iconv for Automatic Transliteration
For broader Unicode coverage:
iconv -f UTF-8 -t ASCII//TRANSLIT < input.md > sanitized.md
pandoc sanitized.md -o output.pdf
5. Alternative: Use XeLaTeX or LuaLaTeX Engine
If you want to preserve Unicode characters, use a Unicode-capable engine:
pandoc input.md -o output.pdf --pdf-engine=xelatex
# or
pandoc input.md -o output.pdf --pdf-engine=lualatex
Best Practices
- Sanitize files before conversion rather than debugging LaTeX errors after
- Keep a mapping of your organization's common symbols and preferred replacements
- Test with a small sample before processing large documents
- Consider using XeLaTeX/LuaLaTeX if Unicode preservation is required
- Document which characters cause issues in your specific environment
Common Error Signs
Look for these LaTeX errors that indicate Unicode issues:
Package inputenc Error: Unicode character ... not set upLaTeX Error: Unicode character ... cannot be used with pdflatex- Missing or garbled characters in the output PDF
04 同仓库更多 Skills
agentpilot
Set up AgentPilot and use its single entry to prepare missing inputs and run configured RSI experiments. Use when the user requests AgentPilot installation or work with AgentPilot.
remember
Review auto-memory entries and propose promotions to OPENSPACE.md, OPENSPACE.local.md, or retained auto-memory topics; detect duplicates, stale facts, conflicts, and ambiguous entries before making changes.
skill-discovery
Search reusable OpenSpace skills locally, and browse cloud packages step by step when a cloud skill may help. Reusing proven skills saves tokens, improves reliability, and extends your capabilities beyond built-in tools.
delegate-task
Delegate tasks to OpenSpace — a full-stack autonomous worker for coding, DevOps, web research, and desktop automation, backed by an extensive MCP tool and skill library. Skills auto-improve through use, reducing token consumption over time. A cloud community lets agents share and collectively evolve reusable skills.
write-special-chars-ts-check
Workaround for write_file failures caused by special characters (apostrophes, backticks, template literals) using a Python heredoc script, plus a reliable TypeScript type-checking step that filters output to only the newly-written files to avoid being misled by pre-existing errors.
write-special-chars-readback
Workaround for write_file failures caused by special characters (apostrophes, backticks, template literals) using a Python heredoc script, with mandatory post-write verification via read_file and cleanup of /tmp helper scripts.