disclude.ioPyPI / tokenizers / 1.0.0rc3
tokenizers 1.0.0rc3
- Source
- https://github.com/huggingface/tokenizers
- Artifact scanned
tokenizers-1.0.0rc3.tar.gz- Compared with
tokenizers-1.0.0rc2.tar.gz(2 signals introduced since)- Files
- 195 scanned, 19 skipped (binary, oversized, or unrecognised)
- Signals
- 0 critical · 0 warn · 9 info
- Scan time
- 4.6 s
- LLM review
- Reviewed by claude-haiku-5-5 (Anthropic): 5 of 5 warn and critical signals; 5 demoted to info.
- Disclude version
- 4.3.0
- Scanned (UTC)
- 2026-10-23 07:37
Signals
Ordered by severity, then signals introduced in this release. Hidden characters are shown as ⟨U+…⟩.
-
-
# tutorial, not a binding test. -
static GROUP: =likely benign Generated Rust static lookup table of byte class values for a tokenizer.Reasoning
Long single-line static table literal of numeric class codes; this is generated Unicode lookup data for a tokenizer classifier. The long-line signal alone reflects formatting, with no execution, decoding or network behaviour.
-
static FAST3_MIXED: =likely benign Large generated static Unicode classification table in Rust.Reasoning
Very long static array of numeric tables generated for Unicode character classification in a Rust tokenizer. Pure data constant with no code paths that execute or exfiltrate anything.
-
static BMP_RLE: =likely benign Generated run-length encoded BMP Unicode class table.Reasoning
Run-length encoded Unicode BMP class table as a static array of (codepoint, class) tuples. This is typical generated data for a lexer and is inert.
-
static ASTRAL: =likely benign Generated astral-plane Unicode class range table.Reasoning
Astral-plane Unicode range table of (codepoint, class) pairs, generated data for a tokenizer. The long line is only a formatting artifact of a data constant.
-
//! 1. **The case split is a scalar escape, on purpose.** `[UC]*[LC]+ | [UC]+[LC]*` has no local //! form: `中Qz` is ONE token (a U after a C is not a boundary) but `ʰABC` is two (it is, when no //! L follows) — the difference is whether an L appears LATER in the run, so no `bk` decides it. -
// The one backtrack: alt-1 needs an `lc` AFTER `uc*`, so a `uc`-run that ENDS the letter // run cuts after its last `cm` (`ʰABC` -> `ʰ`,`ABC`). `中Qz` is untouched because an `l` // follows. "is there an X later in this run" is `fill_to_last` on reversed streams. -
/// Mixed enough that every branch is live, and long enough to span several blocks. const MIXED: &str = "Hello, world! 42 items — naïve café 3.14\tX\r\n\n ¡Hola! \ 日本語テスト、句読点。 Ⅻ ½ ٣٤٥ देवनागरी ｆｕｌｌ width 007 a_b-c'd \likely benign Test fixture string containing mixed Unicode text for classifier tests.Reasoning
A test fixture string deliberately mixing many Unicode scripts, whitespace and punctuation to exercise every classifier branch. The high compression ratio reflects repetitive test content, not an encoded payload, and the file is a test.
Reproduce This Scan
Run disclude locally to verify. These commands scan the published file; the "new in this release" marks come from comparing it with the previous release's file, which they don't repeat. Add --format sarif for SARIF output. Add --llm (with an API key in the environment) to include the LLM review.
|
&&