tiktoken 0.14.0

Source: https://github.com/openai/tiktoken
Tag scanned: 0.14.0
Compared with: 0.13.0 (0 signals introduced since)
Files: 22 scanned, 6 skipped (binary, oversized, or unrecognised)
Signals: 0 critical · 0 warn · 1 info
Scan time: 2.3 s
LLM review: Reviewed by claude-haiku-5-5 (Anthropic): 1 of 1 warn and critical signal; 1 demoted to info.
Disclude version: 4.1.0
Scanned (UTC): 2026-10-22 04:27

Signals

Ordered by severity, then signals introduced in this release. Hidden characters are shown as ⟨U+…⟩.

info ~~warn~~ → info by review dynamic-execution tiktoken/_educational.py:212:10

`open(__file__)` — file reads its own source, common pattern for extracting payloads hidden in comments (ast pass, confidence 0.85)
```
    )
    with open(__file__) as f:
        data = f.read()
```
likely benign LLM, confidence 0.90 The code opens the module's own source file and reads its contents, with no execution or decoding visible in the snippet.
Reasoning
The snippet shows only a plain read of the file's own source via open(__file__) followed by f.read(). No exec, eval, decoding, or network activity appears. In an educational tokenizer module, reading its own source is most plausibly for demonstration or inspection. The scanner's claim about payload extraction is speculative from this snippet alone, and the flagged call is a common benign idiom.

Reproduce This Scan

Run disclude locally to verify. Add --format sarif for SARIF output. Add --llm (with an API key in the environment) to include the LLM review.

cargo install disclude --version 4.1.0
git clone https://github.com/openai/tiktoken tiktoken && cd tiktoken
git checkout 0.14.0
disclude scan --severity info --diff 0.13.0 .