Differences from MeCab
Suzume takes a fundamentally different approach to Japanese tokenization than MeCab. This page documents the intentional differences in token boundaries — where the two tools merge and split — plus known constraints and guidance on choosing between them.
The source of truth for intentional differences is the Python MeCab normalization pipeline used to generate test expectations. The rule families below are grouped by their practical purpose.
Comparison Baseline
Every MeCab boundary and displayed feature on this page was recorded from MeCab 0.996 with mecab-ipadic 2.7.0-20070801 (the UTF-8 IPA dictionary). The labels abbreviate MeCab's comma-separated feature fields for readability; the comparison does not rewrite its token boundaries. It invokes mecab with the default configuration and no user dictionary. You can inspect the active setup with:
mecab --version
mecab -DMeCab's output depends on the selected dictionary, its costs, and user dictionaries. UniDic, NEologd, or a customized IPA dictionary can produce different boundaries and labels from the rows shown here. The normalization pipeline linked above starts from this IPA-dictionary output and applies Suzume's generalized comparison rules when generating test expectations.
Looking for POS label differences?
This page covers where tokens begin and end. For how the two tools label the same tokens — pronouns, na-adjectives, auxiliaries, and other POS assignments — see POS Classification.
Reading the comparisons
In each comparison the MeCab row uses MeCab's Japanese POS names, and the Suzume row uses the public API tags (NOUN, VERB, ADJ, …) with the Japanese POS name shown next to each code. A colored underline marks every token, so where the two tokenizers split or merge is visible at a glance.
Design Philosophy
Suzume is a Tokenizer, Not a Morphological Analyzer
MeCab is a morphological analyzer — its goal is to decompose text into morphemes with detailed grammatical information. Suzume is a tokenizer — its goal is to split text into meaningful units for practical applications like search, display, and text processing. Morphological analysis is not a goal of Suzume. This fundamental difference in purpose explains many of the behavioral differences described below.
| MeCab | Suzume | |
|---|---|---|
| Approach | Dictionary-driven | Feature-driven |
| Dictionary | External dictionary required; size and output depend on the selected dictionary | Compact dictionaries included (~227KiB gzipped with the WASM package) |
| Unknown words | Falls back to character types | Pattern-based candidate generation |
| Compound handling | Boundaries follow the selected dictionary | Dictionary and structural rules; unknown runs may merge |
| Target | Detailed dictionary-based analysis | Compact search/display tokenization across browser, edge, and native runtimes |
Core Trade-off
MeCab follows the entries and costs in the selected dictionary, so changing the dictionary can change boundaries and labels. Suzume combines compact dictionaries with shared character, grammar, and connection rules; unknown same-script runs may stay merged when there is no evidence for an internal boundary.
Merging Rules
These rules keep sequences together as one token where a dictionary-driven analysis splits them — or where fragments would be useless as search units.
Kanji Compounds
An unregistered kanji run with no dictionary or grammatical evidence for an internal boundary is normally kept as one noun candidate. Structural rules can still split forms such as 神奈川県 / 横浜市 and 会議 / 中.
Katakana Compounds
An unknown ordinary katakana run is normally kept as one noun candidate. Dictionary and shape rules can assign another POS, as with mimetic ドキドキ(ADV).
Mixed-Script Compounds
An alphabetic term followed by a katakana term is one compound noun.
Numbers and Units
Cardinal numbers followed by counters or units are normally merged into one quantity token. This includes large number units (万, 億, 兆), decimal numbers, percentages, and alphabetic units. Ordinal and structural suffix rules can split forms such as 第三 / 回.
Suzume also preserves search-unit boundaries around quantities.
Comma-grouped numerals stay whole. A following counter remains a separate SUFFIX, while a currency amount stays one search unit.
The same merging applies beyond Arabic numerals.
Kana-spelled quantities:
Distributive quantities:
Address and lot numbers:
Ordinal 第 versus approximate 約: the ordinal prefix 第 merges with its number, while the following counter stays a separate SUFFIX token. The approximation prefix 約 instead stays a separate PREFIX, and the number merges with its counter.
Dates
Full date expressions are merged into a single token.
Proper Nouns and Place Names
Many place-name components with region suffixes are merged. The structural 県+市 rule is an exception and splits the prefecture from the city.
place name
Compound Verbs
Verb stems in 連用形 followed by subsidiary verbs are merged into compound verbs.
Supported V2 elements include: 込む, 出す, 続く, 返す, 合う, 直す, 切る, 上がる, 抜く, こもる, 続ける, つける, 替える, 合わせる, 上げる, 下げる, 掛ける, 入れる, etc. (40+ patterns)
Grammaticalized subsidiaries such as 過ぎる and かねる are deliberately not merged — see Subsidiary Auxiliary Splitting.
Closed Compound Particles
Closed expressions that function as one particle are kept whole.
The selected IPA dictionary already emits expressions such as と共に and, in some contexts, につれて as one token, so they are not boundary differences in this baseline. Other dictionaries may split them.
Negative ずに
The negative auxiliary ず and the following particle に are merged as one compound grammatical unit.
Desiderative たがる
The desiderative-observation auxiliary たがる, including its inflected forms, is kept as a single auxiliary token.
Fixed Colloquial Function Words
Fixed function words are kept whole even when a dictionary-driven analysis fragments them.
がてら ("while, on the occasion of") is one particle, and the colloquial そんなら ("in that case") is one conjunction.
Deverbal Compound Nouns
A verb continuative plus 会, and the destination suffix 行き, form single event/route nouns.
Nai-Adjectives
Certain adjectives ending in ない are treated as single lexical units rather than being split.
Main examples handled as one token: だらしない, つまらない, もったいない, くだらない, いたたまれない, ものたりない, こころもとない
This is a closed word list — productive "stem + ない" combinations still split; see Productive Negative Splitting.
Slang and Modern Words
Modern colloquial adjectives and verbs are recognized natively.
Recognized examples include エモい, キモい, ウザい, ダサい, イタい, ヤバい, their hiragana variants, and compound i-adjectives.
Recognized verb examples include バズる, ググる, and パクる.
Tari-Adverbs
Tari-conjugation adverb stems followed by と are merged into a single adverb.
Applies to: 泰然, 堂々, 悠々, 淡々, 粛々, 颯爽, 毅然, 漫然, 茫然, 呆然, 唖然, 愕然, 断然, 俄然, 歴然, 整然, 雑然, 騒然, 憮然, 黙然, 昂然, 凛然, 厳然
Noun + Single-Character Suffixes
The following closed set of noun + single-character suffix combinations is merged.
Applies to suffixes: 書, 誌, 時, 率, 性
Verb Stem + 方
When the formal noun 方 follows a short verb stem, Suzume merges the expression into one search unit denoting a method.
Only short stems merge: the continuative must be at most two characters (走り方, やり方). Longer continuatives keep the boundary — 打ち合わせ方 stays 打ち合わせ / 方.
Everyday Hiragana Words
Common words normally written in hiragana are kept intact through pattern rules and the compact L2 dictionary. Rules cover forms such as おととい, ひこうき, みっつ, and calendar compounds such as 翌営業日. Lexical evidence resolves ambiguous all-hiragana nouns whose characters can also be particles or inflectional endings: みず, てがみ, ひらがな, にわ, いりぐち, はにわ, あけぼの, and くだもの.
Prolonged Sound Marks
Prolonged sound marks (ー) are merged with the preceding token.
Technical Text
Technical identifiers are merged into single tokens.
Snake_case identifiers:
Version numbers:
ASCII name + number:
ASCII dot notation:
ASCII word-internal separators:
The same rule covers apostrophes, ampersands, and slashes when they occur between ASCII word characters.
URLs, Mentions, and Hashtags
URLs, @mentions, and #hashtags are merged into single tokens. Hashtags containing Japanese text are kept whole as well.
Splitting Rules
Suzume also draws boundaries a dictionary-driven analysis may not — and deliberately stops its own merging where a boundary carries meaning.
お/ご Prefixes
Suzume splits お/ご honorific prefixes from nouns but keeps them merged when they form inseparable lexemes.
split — separable prefix
Inseparable exceptions (omitted from the diff examples because both MeCab and Suzume keep them as one token) include: お金, お前, おかず, おでん, おもちゃ, おすすめ, おいら, おっさん, お疲れ様, お出で/おいで, and family terms (お母さん, お父さん, お兄ちゃん, お姉さん, おじさん, おばさん, おじいさん, おばあさん, etc.)
Honorific Suffixes and Hiragana Nicknames
Honorific suffixes are split from names.
Applies to suffixes: さん, ちゃん, 様, 君, 殿, さま
Exceptions: family terms like お兄ちゃん and お母さん, and the collective forms 皆様 / 皆さん, are kept as single tokens.
Short nicknames made from a two- or three-character hiragana stem followed by ちゃん or くん, along with lexicalized family terms, can merge as a search unit. Ordinary さん remains a separate suffix, and names with a kanji stem split from their honorific.
Subsidiary Auxiliary Splitting
Grammaticalized subsidiary verbs — 過ぎる (excess), かねる (inability), そびれる (missed chance), 尽くす (exhaustive) — split off from the main verb and are tagged AUX, unlike the lexical compound verbs that merge. MeCab's own treatment varies with the dictionary: some combinations come out merged (飲み過ぎ), others split with the 非自立 subcategory.
Emphatic Colloquial Particles
Colloquial emphatic particles are split as single units instead of being fragmented.
ったら topic particle:
ってば emphatic particle:
Productive Function-Word Chains
Suzume preserves the internal verb and particle boundaries of productive constructions instead of treating the whole spelling as a fixed function word.
The same rule gives を / もっ / て for をもって. Closed expressions with no productive verb boundary remain whole, as in や否や.
Adverbial Noun + Particle
Some sequences that the selected dictionary lexicalizes as one function word are kept as noun + particle.
Leading Fixed Units
The determiner わが is split from the following noun. Similar fixed units such as 以下 and 程度 also keep their own token next to adjacent nouns.
Verb Stem + Productive Suffix
Productive suffixes after a verb stem — がち (tendency), たて (freshness), っぱなし (left as-is) — keep their boundary and are tagged SUFFIX, so the verb stem stays searchable with its lemma.
The negative construction 読みっこない follows the same principle: 読み (VERB) / っこ (SUFFIX) / ない (ADJ).
Quantity and State Suffixes
Even though Suzume merges kanji compounds aggressively, productive quantity/state suffixes keep their boundary and are tagged SUFFIX.
Pure relabeling suffixes with no boundary change, such as the nominalizer さ, are covered in POS Classification.
Productive Negative Splitting
Unlike the closed nai-adjective word list, productive "stem + ない" combinations split, keeping the stem searchable.
Kango + として
For fixed tari-adverb constructions such as 依然として, MeCab treats the expression as one adverb. Suzume splits the adverb form from the する conjugation. Productive expressions such as 名詞として already have grammatical boundaries in both analyzers.
Prefecture + City
Prefecture-city compound nouns are split at administrative boundaries.
split at the 県 / 市 boundary
Note: This split rule applies only to the 県+市 pattern. Other combinations like 都+区 (東京都新宿区) or 府+市 (大阪府大阪市) are merged into single tokens by the Proper Nouns and Place Names rule.
Classical and Literary Endings
Classical inflections keep the same grammatical boundaries as their modern counterparts. The supported families include negative forms, conjectural and obligation auxiliaries, past and perfect auxiliaries, prohibitives, and imperatives. A compact sample:
- Classical negative ぬ splits from the 未然形: 知らぬ → 知ら + ぬ (
AUX) - Literary volitional ん splits the same way: 乗り越えん → 乗り越え + ん (
AUX) - Classical past き keeps its own auxiliary boundary: 行かざりき → 行か + ざり + き
Normalization
Suzume normalizes grammatical constructions to one consistent shape, even where dictionary output varies between equivalent forms.
Content Symbols and Punctuation
Currency and unit signs, arrows, mathematical or technical marks, and emoji remain in the default output as OTHER. They carry text content and keep the token offsets covering that content. Punctuation-like characters are SYMBOL tokens and are omitted by default; enable preserveSymbols to keep them.
| Options | 価格は€50🎉。 |
|---|---|
| Default | 価格(NOUN) / は(PARTICLE) / €(OTHER) / 50(NOUN) / 🎉(OTHER) |
preserveSymbols: true | 価格(NOUN) / は(PARTICLE) / €(OTHER) / 50(NOUN) / 🎉(OTHER) / 。(SYMBOL) |
Copula Negation
After a nominal predicate, Suzume treats じゃ as the copula and ない as the negative auxiliary. An isolated じゃない is ambiguous and may instead be one adjective token, so the nominal host is part of the comparison.
Causative-Passive
MeCab sometimes merges godan verb 未然形 + causative さ into one token. Suzume normalizes this inconsistency.
Filler Decomposition
Fixed conversational phrases lexicalized as fillers are decomposed into their grammatical parts.
Formal Noun ふう
ふうに after a demonstrative determiner is split into its grammatical determiner, formal-noun, and particle units.
Indefinite か
The indefinite particle か is split from an interrogative pronoun. A following existential いる is then treated as a main verb.
Constraints
These are known limitations arising from Suzume's feature-based architecture.
Cannot Split Merged Compounds
Suzume cannot infer arbitrary lexical boundaries inside an otherwise unknown same-script compound. It can still split boundaries licensed by grammatical rules or compact-dictionary entries.
Suzume cannot determine the internal boundaries; MeCab splits them from its dictionary
Workaround: Use the runtime-loading examples in the user-dictionary guide to register boundaries required by your application. That page covers JavaScript, Python, Go, C++, C, and the native CLI.
When to Use Which
| Use Case | Recommendation |
|---|---|
| Browser / client-side apps | Suzume — no server required |
| Search indexing / tag extraction | Suzume — compound merging is often desirable |
| Compatibility with a particular MeCab dictionary/corpus | MeCab — preserves that dictionary's boundaries and taxonomy |
| Real-time UI (input-as-you-type) | Suzume — fast, no network latency |
| Dictionary-defined compound word splitting | MeCab — boundaries come from the selected dictionary |
| Pattern candidates for words absent from the dictionary | Suzume — can analyze character sequences without a lexical entry |