screening system checks a reputation towards a watchlist, it faces a silent failure mode that no person talks about. Sort “Владимир Путин” right into a system listed on “Vladimir Putin” and most name-matching approaches return nothing. The 2 strings share zero characters, so edit distance is meaningless, phonetic codes fail (they assume Latin), and BM25 offers up completely.
This isn’t an obscure edge case. Immigration databases, hospital file techniques, and monetary compliance pipelines cope with this each day. And but, the dominant approaches to this downside are both classical (edit distance, Soundex variants) or heavyweight (fine-tune a multilingual LLM on just a few hundred manually labeled pairs). On this publish, I’ll stroll you thru how we educated a compact transformer encoder from scratch on uncooked UTF-8 bytes, with no tokenizer, no pretrained spine, and no script detection, to resolve cross-script phonetic identify retrieval. We achieved 0.775 MRR and 0.897 R@10 throughout 8 non-Latin scripts, lowering the efficiency hole between Latin and non-Latin queries by 10x over the most effective classical baseline.
The complete code is on GitHub. This publish covers the concepts and the engineering.
Why is this tough?
The issue sits on the intersection of three issues that don’t cooperate:
Scripts are disjoint image units. “Schwarzenegger” and “שוורצנגר” (Hebrew) don’t have any shared characters. Edit distance, the go-to for fuzzy matching, produces a maximum-distance rating each time a script boundary is crossed. Phonetic hashing (Double Metaphone, Soundex) encodes approximate English pronunciation, so it’s ineffective for non-Latin queries by design.
Romanization shouldn’t be a perform. The Chinese language identify written as “张” maps to Zhang, Chang, and Cheung relying on dialect, romanization customary, and historic conference. The Korean “박” maps to Park, Pak, and Bak. Any method that tries to normalize to a canonical Latin kind (like ICU transliterate) will get the appropriate reply for one conference and fail for the others.
Names carry no semantic context. Dense retrieval strategies like DPR and BGE-M3 are highly effective for sentence-level duties as a result of surrounding phrases present semantic grounding. For a 2-word particular person identify there isn’t any context to compensate for floor mismatch. Chari et al. (2025) confirmed that even robust multilingual retrievers degrade severely when queries are transliterated slightly than written of their native script.
The perception behind our method: each Unicode character decomposes deterministically into 1 to 4 bytes from a set 256-symbol alphabet. “Владимир” and “Vladimir” are totally different byte sequences, however a mannequin educated contrastively on sufficient phonetic pairs can study to map them to close by vectors. The vocabulary is common by building.
Constructing Coaching Information at Scale
You may’t prepare this mannequin with out knowledge, and there’s no dataset of 4 million cross-script phonetic identify pairs mendacity round. We constructed one with a 4-stage LLM pipeline.
Stage 1: Stratified sampling from Wikidata
We began with 2 million person-name entities from Wikidata, which supplies canonical English names plus partial cross-script labels (some entities have Russian or Arabic names of their Wikidata file, most don’t). Naively sampling from this produces a dataset dominated by English-only names. We stratified by script-coverage bucket (0, 1-2, 3-4, 5+ non-English labels) and sampled proportionally inside every bucket, yielding 119,040 entities with balanced protection.
Stage 2: Phonetic Latin variants (Llama-3.1-8B)
For every English anchor identify, we requested Llama-3.1-8B-Instruct to generate 4 phonetic spelling variants — the sorts of mishearings and misspellings actual folks produce. The immediate was strict:
Generate 4 DISTINCT phonetic spelling variants of this identify
because it sounds when spoken: "Catherine"
Guidelines:
- Every variant should be spelled in another way from all others and from the unique
- Simulate how totally different folks would possibly mishear or misspell the identify phonetically
- Do NOT use nicknames, abbreviations, or shortened varieties
- Do NOT change language (keep in Latin script)
Return a JSON array of precisely 4 strings, no clarification:
["variant1", "variant2", ...]Outcome for “Catherine”: ["Kathryn", "Katerin", "Kathrin", "Katharine"]
Stage 3: Cross-script transliteration (Qwen3-30B)
For every English identify and every of its Latin variants, we generated transliterations into 8 scripts: Arabic, Russian, Chinese language, Japanese, Hebrew, Hindi, Greek, Korean. We used Qwen3-Coder-30B-A3B-Instruct-FP8:
{
"Catherine": {"ar": "كاثرين", "ru": "Катрин", "he": "קתרין", ...},
"Kathryn": {"ar": "كاثرين", "ru": "Катрин", ...},
"Katharine": {"ar": "...", "ru": "...", ...}
}Each stage is independently resumable: it reads present output, builds a set of already-processed entity IDs, and skips them. A crash loses at most one in-flight batch.
Stage 4: Merge and tag
The ultimate stage merges Wikidata ground-truth labels with LLM output, deduplicates, and tags every optimistic pair by kind:
- phonetic: Latin spelling variant of the English anchor (“Catherine” → “Kathryn”)
- script: direct transliteration right into a non-Latin script (“Catherine” → “كاثرين”)
- mixed: a phonetic Latin variant that was then transliterated (“Katharine” → “كاثرين”)
Positives are saved per entity; negatives should not saved in any respect, they’re mined dynamically throughout coaching. Splits are assigned on the entity degree (80/10/10, deterministic MD5 hash of entity ID) so all variants of an identification go to 1 partition.
Closing dataset: 119,040 entities, 4.67 million optimistic pairs.
The Mannequin
The encoder is genuinely small: 6 transformer layers, 8 consideration heads, hidden dim 256, FFN dim 1024, dropout 0.1, max size 256 bytes. Whole parameters: ~4M.
class ByteLevelEncoder(PreTrainedModel):
def __init__(self, config: ByteEncoderConfig):
tremendous().__init__(config)
self.embedding = nn.Embedding(
config.vocab_size, # 256 — uncooked UTF-8 bytes
config.hidden_dim,
padding_idx=config.pad_token_id,
)
self.pos_embedding = nn.Embedding(config.max_len, config.hidden_dim)
encoder_layer = nn.TransformerEncoderLayer(
d_model=config.hidden_dim,
nhead=config.n_heads,
dim_feedforward=config.ffn_dim,
dropout=config.dropout,
batch_first=True,
norm_first=True, # pre-norm: extra secure when coaching from scratch
)
self.transformer = nn.TransformerEncoder(
encoder_layer, num_layers=config.n_layers,
enable_nested_tensor=False,
)
def ahead(self, input_ids, attention_mask):
B, L = input_ids.form
positions = torch.arange(L, machine=input_ids.machine).unsqueeze(0)
x = self.embedding(input_ids) + self.pos_embedding(positions)
padding_mask = ~attention_mask # TransformerEncoder makes use of True = ignore
x = self.transformer(x, src_key_padding_mask=padding_mask)
# imply pool over actual tokens solely
mask_f = attention_mask.unsqueeze(-1).float()
pooled = (x * mask_f).sum(dim=1) / mask_f.sum(dim=1).clamp(min=1)
return F.normalize(pooled, p=2, dim=-1) # unit vectorsWhy pre-norm (norm_first=True)? When coaching a transformer from scratch (no pretrained initialization), pre-norm stabilizes gradient circulation in early coaching. Submit-norm tends to diverge except you’re cautious with studying fee warmup and initialization. For a fine-tuning situation, you in all probability don’t want to consider this, however right here it mattered.
The output is a unit vector in 256 dimensions. Cosine similarity = inside product on unit vectors, so retrieval is only a dot product.
Coaching: InfoNCE and Onerous Destructive Mining
The InfoNCE loss
The loss is customary: an (anchor, optimistic) pair ought to have a excessive inside product; the anchor’s inside product with each different optimistic within the batch (the in-batch negatives) needs to be low.
def infonce_loss(anchor, optimistic, temperature=0.07):
# anchor, optimistic: (B, D), L2-normalized
logits = (anchor @ optimistic.T) / temperature # (B, B)
labels = torch.arange(len(anchor), machine=anchor.machine) # diagonal = appropriate
return F.cross_entropy(logits, labels)With batch measurement 256 and temperature 0.07, that is 255 negatives per anchor per step. The temperature controls how peaked the distribution is: too excessive and the loss ignores laborious negatives, too low and coaching turns into unstable.
Why in-batch negatives aren’t sufficient
In-batch negatives are low-cost however shallow: they’re random names from the dataset, which are typically straightforward to separate. A mannequin that has been coaching for just a few hundred steps can distinguish “Catherine” from “Zhao Wei” effortlessly. What it struggles with is “Katarina” vs “Katherine” — names which are phonetically shut however seek advice from totally different folks. These are the circumstances the place the gradient sign is definitely informative.
That is the motivation for ANCE (Approximate Nearest Neighbour Contrastive Estimation): periodically rebuild a FAISS index from the present mannequin’s embeddings, then for every anchor, discover the present nearest non-matching neighbors and use these as negatives. They’re laborious exactly as a result of the mannequin presently thinks they’re related.
The laborious destructive schedule
class ANCEBatchSampler(Sampler):
def _current_mix_ratio(self) -> float:
if self._step < self.warmup or self.index is None:
return 0.0
steps_past_warmup = self._step - self.warmup
# ramp from 0 → target_mix_ratio over mix_ramp_steps
return min(
self.target_mix_ratio,
self.target_mix_ratio * steps_past_warmup / max(1, self.mix_ramp_steps)
)In the course of the first 200 steps: random batches solely. The mannequin has no significant construction but; a FAISS index over random embeddings would produce ineffective laborious negatives.
After step 200: the FAISS index is rebuilt periodically from contemporary embeddings (each refresh_every steps). Every batch is constructed by taking a seed anchor, discovering its nearest neighbors within the present index, filling n_hard = batch_size * mix_ratio slots with these neighbors, and padding the remainder with random samples. The combination ratio ramps linearly from 0 to 0.7 over 500 steps after warmup, so the transition is gradual.
The coaching loop:
for batch in train_loader:
anchor = mannequin(batch["anchor"].to(machine), batch["anchor_mask"].to(machine))
optimistic = mannequin(batch["positive"].to(machine), batch["positive_mask"].to(machine))
loss = loss_fn(anchor, optimistic)
loss.backward()
optimizer.step()
optimizer.zero_grad()
scheduler.step()
if global_step % refresh_every == 0:
embs, ids = encode_all(mannequin, train_ds, train_batch_size, machine)
train_sampler.update_index(embs, ids)Analysis
The retrieval setup is an ordinary dense IR analysis. The corpus is all 11,974 test-split anchor names, every encoded to a unit vector and saved in a FAISS FlatIP index. Every optimistic variant within the take a look at set is issued as a question; retrieval succeeds if the proper anchor seems within the top-k outcomes.
We report MRR, R@1, R@5, R@10, and NDCG@10, damaged down 3 ways: total, by question kind, and by script.
General outcomes:
The classical baselines (Levenshtein, Double Metaphone, BM25) cluster at MRR ~0.09. This appears to be like horrible, but it surely’s an artifact of what’s being measured: 70% of the analysis queries are cross-script (script or mixed kind), on which these strategies rating close to zero as a result of they share no characters with Latin-indexed names. On Latin-only queries, Levenshtein achieves 0.894 MRR — a wonderfully respectable quantity for a classical baseline.
Why total MRR misleads
The mixed kind is each the toughest and the most typical (70% of queries): the question is a phonetic variant of the anchor that was then transliterated right into a non-Latin script (“Katharine” → “كاثرين”, English anchor “Catherine”). Breaking down by question kind reveals the place every technique truly fails.
The mannequin must deal with phonetic variation and script change concurrently. Transliterate, which applies a set canonical romanization, drops to 0.485 right here as a result of a set mapping can’t account for phonetic variants within the question.
The byte encoder maintains robust efficiency throughout all three varieties (0.937 / 0.827 / 0.738). The contrastive coaching sign, which sees all three pair varieties, efficiently aligns phonetically equal byte sequences no matter script.
The script hole
The script hole is the R@10 distinction between Latin and non-Latin queries. Classical baselines have gaps of 0.88 to 0.94: they retrieve properly inside Latin script however fail completely throughout script boundaries. The byte encoder reduces this to 0.096.
Importantly, the mannequin additionally improves Latin R@10 from 0.944 to 0.983. The contrastive goal generalizes within-script in addition to throughout scripts.
The remaining hole (0.096) is nearly completely defined by two scripts:
Scripts with constant romanization conventions (Arabic, Russian, Hebrew, Hindi, Greek) attain above 0.95. Chinese language (0.666) and Korean (0.728) are the outliers. Each have extreme romanization ambiguity: “张” maps to Zhang, Chang, and Cheung; “박” maps to Park, Pak, and Bak. The LLM-generated coaching knowledge accommodates all of those as positives for a similar entity, which produces conflicting gradient sign. The mannequin can’t totally resolve which embedding area a reputation belongs to when its romanization is genuinely ambiguous.
Discover additionally that BM25 performs barely higher on Chinese language and Korean than different baselines. This isn’t as a result of BM25 understands phonetics. When the question is already within the goal script (Chinese language querying a Chinese language-indexed corpus), equivalent CJK characters could seem in each question and doc, producing incidental character n-gram overlap. This impact disappears for true cross-script retrieval (Latin question, CJK corpus) and shouldn’t be mistaken for phonetic matching.
FAISS index ablation
HNSW matches precise search recall (0.896 vs 0.897 R@10) at 5.7x decrease latency. For deployment, HNSW is the selection: the small recall penalty is negligible and the latency enchancment compounds at scale. IVF-PQ cuts index measurement by 96% at a 6.4% R@10 penalty — price contemplating should you’re indexing thousands and thousands of entities and reminiscence is constrained.
At 11,974 entities the distinction between 0.03 ms and 0.17 ms is tutorial. At 50 million entities in an actual deployment, HNSW’s recall benefit over IVF-Flat turns into extra pronounced because the variety of index partitions grows.
What doesn’t work (and why)
The mannequin fails to totally shut the hole on Chinese language and Korean, and the reason being price dwelling on. The pipeline generates non-Latin variants solely by transliterating from Latin: “Catherine” → Latin variant → Arabic/Chinese language/and many others. It by no means generates native-script spelling variation. Various Arabic orthographies, Korean spacing conventions, or variant Chinese language character varieties that seek advice from the identical identify don’t seem in coaching knowledge. The mannequin learns to map Latin byte sequences to non-Latin byte sequences, but it surely hasn’t seen non-Latin spelling variation inside a single script.
It is a recognized limitation. The repair can be a fifth pipeline stage: given a generated Chinese language or Arabic identify, ask the LLM to supply native-script phonetic variants of it. We didn’t do that, so the mannequin is probably going underperforming on queries that characterize real-world native-script variation.
A second limitation: 99.5% of optimistic pairs are LLM-generated. The analysis makes use of the identical LLM-generated pairs. If the LLM systematically mistransliterates a category of names, each coaching and analysis sign can be unsuitable in the identical course, and we might not catch it. The 0.5% Wikidata floor reality supplies a sanity examine however not an entire one.
Key takeaways
Byte-level tokenization is an underused software for multilingual duties. It eliminates out-of-vocabulary tokens by building, requires no language-specific tokenizer, and offers you a common 256-symbol vocabulary that covers each Unicode character. For duties the place floor kind issues greater than semantics — like identify matching — it’s a pure match.
LLMs are a viable knowledge engine for low-resource retrieval duties. We generated 4.67 million optimistic pairs throughout 8 scripts utilizing two open-weight fashions. The pipeline is 4 phases, every independently resumable. This method is generalizable to different low-resource entity matching issues the place ground-truth labels are scarce however a succesful LLM can synthesize practical variation.
ANCE laborious destructive mining issues. The transition from random negatives to ANN-mined laborious negatives noticeably sharpens the embedding area. With out it, the mannequin would study to separate straightforward circumstances (totally different names in the identical script) however wrestle on the laborious ones (phonetically related names throughout scripts).
Report outcomes by question kind and script, not simply total MRR. An total MRR of 0.775 masks enormous variation: 0.937 on phonetic queries, 0.738 on mixed. A system that appears mediocre on headline metrics could also be near-perfect for one use case and damaged for one more.
The code, dataset pipeline, educated checkpoint, and analysis scripts are at github.com/vedant-jumle/cross-language-phonetic-text-alignment.
Observe about Wikidata: Wikidata is launched below CC0 1.0 Common (public area) — no restrictions on use, together with industrial.
