tiktoken 0.14.0

Source: https://github.com/openai/tiktoken
Artifact scanned: tiktoken-0.14.0.tar.gz (sdist · 38.0 KB · SHA-256 231dec90efcd…)
Compared with: tiktoken-0.13.0.tar.gz (0 signals introduced since)
Files: 26 scanned, 10 skipped (binary, oversized, or unrecognised)
Signals: 0 critical · 0 warn · 1 info
Scan time: 2.3 s
LLM review: Reviewed by claude-haiku-5-5 (Anthropic): 1 of 1 warn and critical signal; 1 demoted to info.
Disclude version: 4.3.0
Scanned (UTC): 2026-10-23 07:19

Signals

Ordered by severity, then signals introduced in this release. Hidden characters are shown as ⟨U+…⟩.

info ~~warn~~ → info by review dynamic-execution tiktoken/_educational.py:212:10

`open(__file__)` — file reads its own source, common pattern for extracting payloads hidden in comments (ast pass, confidence 0.85)
```
    )
    with open(__file__) as f:
        data = f.read()
```
likely benign LLM, confidence 0.90 The module reads its own source file into a variable, with no visible decoding or execution of the contents.
Reasoning
The snippet shows only open(__file__) followed by a plain read into data. Reading the module's own source is a common pattern in educational or demo code, for example to count lines, display source, or parse docstrings. The snippet shows no decoding, execution, or network use of the data. The scanner's message about extracting hidden payloads is speculative here. The file name suggests an educational module, so the read is most likely benign, though the surrounding code is not visible.

Reproduce This Scan

Run disclude locally to verify. These commands scan the published file; the "new in this release" marks come from comparing it with the previous release's file, which they don't repeat. Add --format sarif for SARIF output. Add --llm (with an API key in the environment) to include the LLM review.

cargo install disclude --version 4.3.0
curl -fsSL -o tiktoken-0.14.0.tar.gz https://files.pythonhosted.org/packages/66/62/167a842aa0429d45f5e797354fd4343a96f6043d67d0513c675c7b8d36e6/tiktoken-0.14.0.tar.gz
printf '%s  %s\n' 231dec90efcdccf1b565a1416107736f1e09b1a08fe736ef9d6363e626d03874 tiktoken-0.14.0.tar.gz | shasum -a 256 -c -
mkdir pkg && tar -xzf tiktoken-0.14.0.tar.gz -C pkg --strip-components 1
disclude scan --untrusted --severity info pkg