间隔号-divided transcriptions keep source order - #305
Open
derek73 wants to merge 8 commits into
Open
Conversation
Each dot is read by its own typography's standard. The Japanese nakaguro (U+30FB/U+FF65) is roster/divider typography whose pieces the script license already reads correctly (#272: 高橋・一郎 family-first, マイケル・ジャクソン positional) -- it stays an unconditional pure separator that records nothing. Only the Chinese 间隔号 U+00B7 carries the transcription meaning (威廉·莎士比亚 keeps source order), so only it records, as interpunct_offsets mirroring comma_offsets: main-region only, gated by the same flag that keeps nickname commas out of comma_offsets (record_commas renamed record_offsets). U+00B7 cannot separate unconditionally -- it is also the Catalan punt volat, interior to legitimate names (Gal·la) -- so it is the first context-sensitive separator: it divides only between classified-script neighbors (region-local flanks; an edge B7 stays token text), handled in _tokenize_region where the index exists. The consumers (order and segmentation suppression) arrive in the following commits. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A B7-divided name is a transcription: the dot plays for Han exactly the role pure katakana plays in the kana license -- orthography naming the convention -- so the family-first license (Policy.script_orders) yields to the positional name_order. The check sits before the script_orders lookup in _effective_order because the suppression outranks the license. Codepoint-scoped per spec decision 5 (2026-07-30 design doc): only U+00B7 records interpunct_offsets. The Japanese nakaguro (U+30FB) is roster typography whose divided pieces the license already reads correctly, so #272's family-first reading of 高橋・一郎 is byte-identical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The transcription doctrine's second consumer, after _effective_order: a B7-divided name's pieces are foreign-name syllable groups, not surname+given, so the segmentation stage has no surname site to find -- neither the vocabulary nor a configured segmenter applies. Reachable today: script_segment takes the FIRST script token of the name segment with no single-token gate, so 马丁·路德·金 (Martin Luther King) would split 马 + 丁 under a zh vocabulary listing 马. The gate sits directly beside the FAMILY_COMMA one -- both are the-writer-already-decided opt-outs that run before either split mechanism is consulted. Codepoint-scoped like the rest of #298: the Japanese nakaguro records no offsets and therefore gates nothing. The gate is state-global, matching the order twin: a marker anywhere in the name suppresses segmentation of every token, so an un-dotted hangul token beside a dotted Han one stays whole too ("김민준 马丁·路德" no longer splits 김 + 민준) -- a name carrying the transcription marker reads as a transcription listing throughout. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…298) Five case rows land in tests/v2/cases.py, in the table's canonical given/family/middle spelling: - zh_interpunct_transcription_source_order: 威廉·莎士比亚 divides on the B7 and keeps source order (fix(#298)). - zh_interpunct_nakaguro_typed_stays_roster: the SAME transcription typed with the Japanese nakaguro reads as a 姓・名 roster pair, family-first (fix(#272)). A limitation row, chosen not accidental: the design is codepoint-scoped (spec 2026-07-30 decision 5), so cross-convention input reads by the convention of the codepoint it was typed with -- only the Chinese B7 marks a transcription. - ja_interpunct_b7_katakana: sloppy-IME B7 between katakana divides like the nakaguro; order was already positional (fix(#298)). - latin_punt_volat_is_name_interior: Gal·la Marcet, the flank guard's reason -- Latin-flanked B7 is interior, never a divider (parity). - zh_interpunct_suppresses_segmentation: 马丁·路德·金 under zh -- the dot gates segmentation off, 马丁 stays whole (fix(#298)). The differential CJK rule's character class gains U+00B7, the second sanctioned extra beside U+FF65: both dots change parses without being classified by any Script (B7 doubles as Catalan's punt volat). test_differential_cjk_rule_matches_the_script_ranges pins the pair as sanctioned_extras; the compound rule's lookahead copies classified spans only and is untouched. corpus_cjk.jsonl regenerates 31 -> 35: the four CJK-bearing rows enter (Gal·la Marcet correctly does not -- no classified codepoint). Harness hand-run: 689 names, 48 intentional diffs, zero unexplained; the CJK rule classifies 27 including all the B7 names. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
docs/release_log.rst gains the 2.1.0 Behavior Changes bullet for U+00B7: the 间隔号 is a token separator between characters of a classified script, the name it divides keeps its source order and is never segmented, the Catalan punt volat is untouched, the nakaguro is deliberately not a transcription marker, and rendering stays space-joined with a custom string_format reinstating the dot. Placed directly after the katakana-dot bullet, whose closing sentence hands off to #298. docs/usage.rst gains a paragraph in the East Asian section's Chinese flow — the 间隔号 convention as background, then the separator behavior and the never-segmented reading, with the nakaguro distinguished — and the Boundaries limits paragraph states the two limitations: the spaced form (威廉 莎士比亚) and the nakaguro-typed transcription (威廉・莎士比亚) both keep family-first, because only the Chinese dot marks a transcription. No new doctest blocks; the case rows carry the examples. docs/migrate.rst completes the "one exception" sentence: two dots are now decided in tokenization rather than by policy — the katakana dot unconditionally, the interpunct only between classified-script characters. The nakaguro doctest and its explanation are unchanged. docs/concepts.rst extends the first/last decider list: the interpunct walks the script_orders entry back, marking a transcription that keeps its source order. docs/customize.rst needs no edit: its tokenization-not-policy sentence is scoped to the nakaguro and stays true, and under script_orders={} everything is positional already, so the dot adds nothing switchable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The differential rule now explains #298's B7 diffs, so its issue slug says so -- compare.py output labels a B7 diff with the issue that caused it. The sync test's substring selection (#271/#272) still matches this one rule uniquely. And the corpus generator's docstring spoke of #298 as future work it has since harvested. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #305 +/- ##
==========================================
+ Coverage 98.43% 98.45% +0.01%
==========================================
Files 41 41
Lines 2750 2776 +26
==========================================
+ Hits 2707 2733 +26
Misses 43 43 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The B7 flank guard read text[i-1]/text[i+1] raw, so a policy-stripped
invisible adjacent to the dot defeated it: parse("威廉·莎士比亚")
left the literal dot in the family field, lost the order suppression,
and the polluted token defeated the script license too. That input is
the real-world case strip_bidi exists for -- RTL documents quoting
Chinese transcriptions get U+200E/200F inserted at script transitions,
in text visually identical to the clean form.
The guard now walks each flank past characters the strip policy
removes (bidi when strip_bidi, emoji when strip_emoji) before
classifying: a character that vanishes from the token stream cannot
occupy a flank position either. Whitespace, commas, and the name-dot
separators remain guard-defeating -- a dot beside a space is not
between characters -- and the region-local bounds are unchanged, so
the pinned edge behavior stays.
Also pins the guard's edge coverage the surviving mutants demanded:
the trailing-dot upper bound (whose loss is an IndexError, not a
wrong answer), the minimal one-char flanks, the one-sided flank
(the AND, and that no spurious offset records), and hangul flanks
(마틴·루터·킹 -- Korean writes interpunct transcriptions too).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The B7 sanctioned-extra span was a pure loss, removed: the flank guard guarantees every name the dot can change contains a classified character, so B7 names already matched the CJK rule through their flanks -- the span's only actual effect was letting the rule claim first/middle/last diffs on punt-volat Latin names (Gal·la Marcet), pre-excusing a regression on exactly the input class the guard protects. The sync test goes back to one sanctioned extra, and its overlap loop now checks whole spans instead of discarding each span's high end. Three case rows close review-found gaps: the Korean transcription 마이클·잭슨 (마 is a listed census surname, and the spaced form really does mis-split 마|이클 today -- the dot is what rescues it), the suffix-comma composition 威廉·莎士比亚, PhD, and the half-flanked 王·Smith parity row. The suffix-comma row's corpus entry surfaced a genuine compound diff -- comma routing (title to suffix) union the CJK flip (first/last) -- that neither single-change rule may claim, each excluding the other's fields on purpose; the new fix(cjk-comma-compound) rule takes exactly that union, gated by the same two-lookahead shape as the delimiter compound, and the compound sync pin is parametrized over both slugs so neither span copy goes unpinned. Docs: usage.rst and migrate.rst state the per-flank judgment (a hangul character beside a Han one qualifies), the state-global breadth (a dot anywhere reads the whole name as a transcription listing), and the half-flanked outcome (the undivided dot remains part of the word); AGENTS.md's doctrine paragraph counts five defaults and names the 间隔号 marker among them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #298, codepoint-scoped per the design amendment recorded in the issue (comment below).
U+00B7 — the Chinese 间隔号 — becomes the parser's first context-sensitive separator: it divides only between characters of a classified script, so the Catalan punt volat interior to
Gal·lais untouched. A name it divides is a transcription —威廉·莎士比亚is William Shakespeare — and reads in source order (given威廉, family莎士比亚) with segmentation gated off (马丁·路德·金under the zh pack keeps马丁whole even though马is a listed surname). The dot's offsets rideParseState.interpunct_offsets, recorded by tokenize exactly as comma offsets are (main region only, so a dot inside a nickname doesn't mark the outer name).The Japanese nakaguro is deliberately NOT a transcription marker.
高橋・一郎is roster formatting (姓・名) and keeps its #272 family-first reading, byte-identical — the two dots carry opposite conventions in exactly this position, so each is read by its own typography's standard: no language detection, the codepoint itself settles it. The cost is pinned as a chosen limitation:威廉・莎士比亚(a transcription typed with the Japanese dot) reads by the dot it was typed with, like the spaced form.Guards: five case rows (including the limitation row and a Gal·la parity row), the CJK corpus regenerated (35 names), U+00B7 joining U+FF65 as the differential rule's second sanctioned extra with the sync test extended, and the rule slug now owning #298.
Test plan
str(HumanName("威廉·莎士比亚"))== "威廉 莎士比亚" andstring_format="{first}·{last}"reinstating the dot🤖 Generated with Claude Code