tokenizers 0.23.3

Source: https://github.com/huggingface/tokenizers
Tag scanned: v0.23.3
Directory: bindings/python (only this part of the repository was scanned)
Compared with: v0.23.2 (1 signals introduced since)
Files: 77 scanned, 10 skipped (binary, oversized, or unrecognised)
Signals: 0 critical · 0 warn · 5 info
Scan time: 2.6 s
LLM review: Reviewed by claude-haiku-5-5 (Anthropic): 1 of 1 warn and critical signal; 1 demoted to info.
Disclude version: 4.0.0
Scanned (UTC): 2026-10-14 18:43

Signals

Ordered by severity, then signals introduced in this release. Hidden characters are shown as ⟨U+…⟩.

info unicode-mixed-script bindings/python/py_src/tokenizers/normalizers/__init__.pyi:158:39 new in this release

identifier `ﬁne` mixes scripts: Other + Latin (in string/comment) (raw pass, confidence 0.40)
```
        >>> normalizer = NFKC()
        >>> normalizer.normalize_str("ﬁne caf\\u00e9")
        'fine café'
    """
```
info ~~warn~~ → info by review dynamic-execution bindings/python/README.md:52:1

[embedded bash] pipeline feeds into `sh` — piped data is executed as shell code (ast pass, confidence 0.80)
```
# Install with:
curl https://sh.rustup.rs -sSf | sh -s -- -y
export PATH="$HOME/.cargo/bin:$PATH"
```
likely benign LLM, confidence 0.90 Documentation snippet showing the official rustup installer via curl piped to sh.
Reasoning
This is a standard rustup installation instruction in a README code block. The pipeline curl https://sh.rustup.rs | sh is the officially documented Rust installer, a well-known legitimate pattern. The finding is a documentation example, not executed code, and the domain is the genuine rustup host, so it is benign though worth noting as a curl-pipe-sh practice.
info unicode-mixed-script bindings/python/py_src/tokenizers/normalizers/__init__.pyi:176:39

identifier `ﬁne` mixes scripts: Other + Latin (in string/comment) (raw pass, confidence 0.40)
```
        >>> normalizer = NFKD()
        >>> normalizer.normalize_str("ﬁne")
        'fine'
    """
```
info unicode-mixed-script bindings/python/src/normalizers.rs:361:39

identifier `ﬁne` mixes scripts: Other + Latin (in string/comment) (raw pass, confidence 0.40)
```
///     >>> normalizer = NFKD()
///     >>> normalizer.normalize_str("ﬁne")
///     'fine'
///
```
info unicode-mixed-script bindings/python/src/normalizers.rs:410:39

identifier `ﬁne` mixes scripts: Other + Latin (in string/comment) (raw pass, confidence 0.40)
```
///     >>> normalizer = NFKC()
///     >>> normalizer.normalize_str("ﬁne caf\u00e9")
///     'fine café'
///
```

Reproduce This Scan

Run disclude locally to verify. Add --format sarif for SARIF output. Add --llm (with an API key in the environment) to include the LLM review.

cargo install disclude --version 4.0.0
git clone https://github.com/huggingface/tokenizers tokenizers && cd tokenizers
git checkout v0.23.3
disclude scan --severity info --diff v0.23.2 bindings/python