unicode-to-plain-text

Convert fancy Unicode text to plain ASCII with smart language preservation

Install

npm i unicode-to-plain-text

Usage

Basic usage:

import { toPlainText } from 'unicode-to-plain-text'

// Mathematical styles
toPlainText('𝐇𝐞𝐥𝐥𝐨 𝐖𝐨𝐫𝐥𝐝') // => 'Hello World'

// Enclosed characters
toPlainText('🅣🅔🅢🅣') // => 'TEST'

// Fullwidth forms
toPlainText('ＨＥＬＬＯ') // => 'HELLO'

Language preservation:

// Preserve specific writing systems
toPlainText('Привет 𝐖𝐨𝐫𝐥𝐝', { preserve: ['cyrillic'] }) // => 'Привет World'
toPlainText('你好 𝐖𝐨𝐫𝐥𝐝', { preserve: ['cjk'] })       // => '你好 World'

// But lookalike characters are converted when not preserved
toPlainText('Α test')  // => 'A test' (Greek Alpha → Latin A)

Sanitize user input:

import { sanitize } from 'unicode-to-plain-text'

// Clean and validate display names
sanitize('𝐉𝐨𝐡𝐧 𝐃𝐨𝐞')
// => { text: 'John Doe', valid: true, length: 8 }

sanitize('   ')
// => { text: '', valid: false, length: 0, error: 'whitespace_only' }

sanitize('AB', { minLength: 3 })
// => { text: 'AB', valid: false, length: 2, error: 'too_short' }

Custom pipelines:

import {
  pipe,
  normalizeFlipped,
  convertCharacters,
  normalizeDiacritics,
  normalizeDecorations,
  normalizeCasing
} from 'unicode-to-plain-text'

// Create a custom pipeline
const customTransform = pipe(
  normalizeFlipped,
  convertCharacters,
  normalizeDiacritics,
  normalizeDecorations
)

const result = customTransform('𝐓𝐄𝐒𝐓')

API

toPlainText(text, options?)

Converts fancy Unicode text to plain ASCII

Property	Type	Description
text	string	Input text with Unicode characters
options	object	Optional configuration object

Options

Option	Type	Default	Description
normalizeSpaces	boolean	true	Collapse spaces, convert whitespace, strip invisible
skipEmoji	boolean	false	Preserve emoji characters
preserve	PreserveOption	[]	Writing systems to preserve: 'all' or array
trim	TrimOption	'all'	Trim mode: 'all', 'start', 'end', or 'none'
asciiOnly	boolean	false	Strip all non-ASCII characters after normalization

PreserveOption: 'all' | WritingSystem[]

WritingSystem: 'latin' | 'greek' | 'cyrillic' | 'arabic' | 'hebrew' | 'cjk' | 'ethiopic' | 'thai' | 'devanagari'

Note: 'latin' preserves all European language diacritics (French, German, Spanish, Portuguese, Polish, Czech, Hungarian, Romanian, Turkish, Nordic, etc.)

Examples

// Default behavior - emojis removed
toPlainText('Hello 🎉 World') // => 'Hello World'

// Preserve emojis
toPlainText('Hello 🎉 World', { skipEmoji: true }) // => 'Hello 🎉 World'

// Preserve spacing
toPlainText('Hello   World', { normalizeSpaces: false }) // => 'Hello   World'

// Preserve specific writing systems
toPlainText('שלום 𝐖𝐨𝐫𝐥𝐝', { preserve: ['hebrew'] }) // => 'שלום World'

// Preserve all writing systems
toPlainText('καλημέρί', { preserve: 'all' }) // => 'καλημέρί'

// Preserve European diacritics (French, German, Turkish, Polish, etc.)
toPlainText('Ömer Şahin', { preserve: ['latin'] }) // => 'Ömer Şahin'
toPlainText('François Müller', { preserve: ['latin'] }) // => 'François Müller'

// Control trimming
toPlainText('  hello  ', { trim: 'start' }) // => 'hello '
toPlainText('  hello  ', { trim: 'none' })  // => ' hello '

sanitize(text, options?)

Sanitizes and validates text for use as display names

Property	Type	Description
text	string	Input text to sanitize
options	object	Optional configuration object

Options

Option	Type	Default	Description
minLength	number	1	Minimum length count
maxLength	number	64	Maximum length count
preserve	WritingSystem[]	-	Writing systems to preserve
skipEmoji	boolean	false	Preserve emoji characters
trim	TrimOption	'all'	Trim mode
truncate	boolean	false	Auto-truncate to maxLength instead of error
allowedWritingSystems	WritingSystem[]	-	Strict whitelist of allowed writing systems
rejectHomoglyphs	boolean	false	Fail validation if homoglyphs detected
asciiOnly	boolean	false	Ensure output contains only ASCII

Returns

{
  text: string        // Sanitized text
  valid: boolean      // Whether validation passed
  length: number
  error?: 'empty' | 'too_short' | 'too_long' | 'whitespace_only' | 'homoglyphs' | 'disallowed_writing_system'
}

Detection Utilities

• detectWritingSystems(text) - Returns { writingSystems: WritingSystem[], mixed: boolean, primary: WritingSystem | null }
• getWritingSystem(char) - Returns the writing system for a single character
• hasHomoglyphs(text) - Checks if text contains cross-script visually confusable characters
• analyzeHomoglyphs(text) - Returns detailed analysis of homoglyph matches
• isSuspiciousMix(text) - Checks for Latin + Cyrillic/Greek mixing (common spoofing)

Individual Functions

• normalizeFlipped(text) - Handles upside-down and mirrored text
• convertCharacters(text, options?) - Maps Unicode to ASCII equivalents
• normalizeDiacritics(text) - Removes diacritics from Latin text
• normalizeDecorations(text, options?) - Removes emojis and decorations
• decodeUnicodeId(text) - Converts comma-separated code points to string
• normalizeSpaces(text, options?) - Normalizes whitespace with trim control
• normalizeCasing(text) - Normalizes inconsistent casing
• pipe(...fns) - Composes functions into a pipeline
• pipeWith(...fns) - Pipe with shared context
• when(fn, condition) - Conditional pipeline step

License

Apache-2.0