pyspark 4.0.5

Source: https://github.com/apache/spark
Tag scanned: v4.0.5
Directory: python (only this part of the repository was scanned)
Compared with: v4.0.4 (0 signals introduced since)
Files: 1221 scanned, 41 skipped (binary, oversized, or unrecognised)
Signals: 0 critical · 0 warn · 30 info
Scan time: 21.7 s
LLM review: Reviewed by claude-haiku-5-5 (Anthropic): 26 of 26 warn and critical signals; 26 demoted to info.
Disclude version: 4.3.0
Scanned (UTC): 2026-10-28 14:28

Signals

Ordered by severity, then signals introduced in this release. Hidden characters are shown as ⟨U+…⟩.

info ~~warn~~ → info by review ifs-manipulation python/docs/source/getting_started/install.rst:175:64

[embedded bash] `IFS` set to `:` — non-standard separator enables word-splitting obfuscation (ast pass, confidence 0.80)
```
    export SPARK_HOME=`pwd`
    export PYTHONPATH=$(ZIPS=("$SPARK_HOME"/python/lib/*.zip); IFS=:; echo "${ZIPS[*]}"):$PYTHONPATH
```
likely benign LLM, confidence 0.85 Docs snippet using IFS=: to join zip paths into PYTHONPATH.
Reasoning
Documentation example in an rst file setting PYTHONPATH with a bash IFS join of zip globs. Standard shell idiom for building a classpath; not obfuscation.
info ~~critical~~ → info by review dynamic-execution python/packaging/classic/setup.py:48:5

`exec` at module scope (runs on import) on a non-literal expression (ast pass, confidence 0.80)
```
try:
    exec(open("pyspark/version.py").read())
except IOError:
    print(
```
likely benign LLM, confidence 0.90 Setup script execs the package's local version.py to read the version.
Reasoning
Standard PySpark setup.py idiom exec-ing pyspark/version.py to get __version__ at build time. Legitimate, though exec on a local file is a code smell; the file is part of the source tree.
info ~~critical~~ → info by review dynamic-execution python/packaging/client/setup.py:51:5

`exec` at module scope (runs on import) on a non-literal expression (ast pass, confidence 0.80)
```
try:
    exec(open("pyspark/version.py").read())
except IOError:
    print(
```
likely benign LLM, confidence 0.90 Client setup script execs local version.py for the version string.
Reasoning
Same version-file exec pattern as the classic packaging setup; reads a local source file with no network or encoded payload.
info ~~critical~~ → info by review dynamic-execution python/packaging/connect/setup.py:74:9

`exec` at module scope (runs on import) on a non-literal expression (ast pass, confidence 0.80)
```
    try:
        exec(open("pyspark_connect/version.py").read())
    except IOError:
        print(
```
likely benign LLM, confidence 0.90 Connect setup script execs local version.py for the version string.
Reasoning
Same benign version.py exec pattern in the connect packaging script.
info ~~critical~~ → info by review dynamic-import python/pyspark/cloudpickle/cloudpickle.py:469:5

`__import__` called on a non-literal expression (ast pass, confidence 0.70)
```
    # module.
    __import__(name)
    return sys.modules[name]
```
likely benign LLM, confidence 0.90 cloudpickle dynamic import of a module by name during unpickling of subimports.
Reasoning
cloudpickle's subimport helper imports a module by name, a documented and necessary dynamic import in a serialization library.
info ~~warn~~ → info by review frame-introspection python/pyspark/errors/utils.py:205:17

`inspect.currentframe` — call-stack frame introspection (ast pass, confidence 0.70)
```
    def inspect_stack() -> Iterator[FrameType]:
        frame = inspect.currentframe()
        while frame:
            yield frame
```
likely benign LLM, confidence 0.88 Frame walking helper used to capture call-site info for PySpark errors.
Reasoning
Walks the call stack to build error context for PySpark error messages; ordinary diagnostic introspection.
info ~~critical~~ → info by review dynamic-import python/pyspark/ml/connect/io_utils.py:67:9

`__import__` called on a non-literal expression (ast pass, confidence 0.70)
```
    module = ".".join(parts[:-1])
    m = __import__(module, fromlist=[parts[-1]])
    return getattr(m, parts[-1])
```
likely benign LLM, confidence 0.88 Imports a class by dotted path when loading ML model metadata.
Reasoning
Resolves a dotted class path from saved metadata to load ML stages; standard plugin-style import in PySpark ML persistence.
info ~~critical~~ → info by review dynamic-import python/pyspark/ml/connect/readwrite.py:264:21

`__import__` called on a non-literal expression (ast pass, confidence 0.70)
```
                module = ".".join(parts[:-1])
                m = __import__(module, fromlist=[parts[-1]])
                return getattr(m, parts[-1])
```
likely benign LLM, confidence 0.88 Dotted-path class import for ML model loading in connect readwrite.
Reasoning
Same class-path resolution for ML read/write of Spark Connect models; expected behavior.
info ~~critical~~ → info by review dynamic-import python/pyspark/ml/util.py:907:13

`__import__` called on a non-literal expression (ast pass, confidence 0.70)
```
        module = ".".join(parts[:-1])
        m = __import__(module, fromlist=[parts[-1]])
        return getattr(m, parts[-1])
```
likely benign LLM, confidence 0.88 Dotted-path class import for ML persistence in util.py.
Reasoning
Standard dotted-path import used to reconstruct ML pipeline stages from metadata; legitimate.
info ~~critical~~ → info by review dynamic-import python/pyspark/ml/wrapper.py:313:17

`__import__` called on a non-literal expression (ast pass, confidence 0.70)
```
            module = ".".join(parts[:-1])
            m = __import__(module, fromlist=[parts[-1]])
            return getattr(m, parts[-1])
```
likely benign LLM, confidence 0.88 Dotted-path class import when loading ML Java/Python wrapper stages.
Reasoning
Same ML persistence class-path import pattern in wrapper.py; legitimate framework code.
info ~~critical~~ → info by review frame-introspection python/pyspark/pandas/sql_processor.py:208:16

`inspect.stack` — call-stack frame introspection — file also calls `sys.exit`/`exec`/`eval` (anti-analysis shape) (ast pass, confidence 0.85)
```
    try:
        return inspect.stack()[_CAPTURE_SCOPES][0].f_locals
    except IndexError:
        return {}
```
likely benign LLM, confidence 0.80 Uses inspect.stack to fetch caller locals for SQL variable substitution.
Reasoning
Reads caller's local variables via inspect.stack to resolve variable references in pandas-on-Spark SQL. The sys.exit/exec/eval co-occurrence is a weak anti-analysis signal that does not fit this function's purpose.
info ~~warn~~ → info by review dynamic-execution python/pyspark/serializers.py:523:16

`marshal.loads` deserializes Python bytecode from bytes — code-object sink (ast pass, confidence 0.75)
```
    def loads(self, obj):
        return marshal.loads(obj)
```
likely benign LLM, confidence 0.85 Marshal serializer loads for function transport between Spark processes.
Reasoning
marshal-based serializer is a deliberate PySpark feature for shipping functions between driver and workers; only sends code between trusted cluster components.
info ~~warn~~ → info by review dynamic-execution python/pyspark/serializers.py:548:20

`marshal.loads` deserializes Python bytecode from bytes — code-object sink (ast pass, confidence 0.75)
```
        if _type == b"M":
            return marshal.loads(obj[1:])
        elif _type == b"P":
            return pickle.loads(obj[1:])
```
likely benign LLM, confidence 0.85 Type-tagged deserialization dispatch including marshal and pickle.
Reasoning
Dispatch on a type byte choosing marshal or pickle for internal Spark serialization; design of trusted inter-process protocol, not hidden payload.
info ~~warn~~ → info by review decoder-decompress-payload python/pyspark/serializers.py:569:38

file imports decoder module(s) marshal, zlib, pickle and calls `.decompress()` — embedded payload decompressed at runtime (ast pass, confidence 0.75)
```
    def loads(self, obj):
        return self.serializer.loads(zlib.decompress(obj))

    def __repr__(self):
```
likely benign LLM, confidence 0.85 Decompresses serialized data before deserializing in Spark's serializer.
Reasoning
zlib-wrapped serializer for compressing Spark data between trusted workers; ordinary compression, not a dropper.
info ~~critical~~ → info by review dynamic-execution python/pyspark/shell.py:142:16

`compile` at module scope (runs on import) on a non-literal expression (ast pass, confidence 0.80)
```
    with open(_pythonstartup) as f:
        code = compile(f.read(), _pythonstartup, "exec")
        exec(code)
```
likely benign LLM, confidence 0.85 Interactive shell executes the file named by PYTHONSTARTUP.
Reasoning
Shell startup honors PYTHONSTARTUP by compiling and executing the user's own startup file, the standard Python interactive behavior. Not hidden code; it runs only where the user sets the variable.
info ~~critical~~ → info by review dynamic-execution python/pyspark/shell.py:143:9

`exec` at module scope (runs on import) on a non-literal expression (ast pass, confidence 0.80)
```
        code = compile(f.read(), _pythonstartup, "exec")
        exec(code)
```
likely benign LLM, confidence 0.85 Executes user-supplied PYTHONSTARTUP code in the PySpark shell.
Reasoning
Companion exec of the PYTHONSTARTUP compiled code, same legitimate user-configured startup hook.
info ~~critical~~ → info by review dynamic-execution python/pyspark/sql/tests/test_catalog.py:493:17

`os.system` called on a non-literal command — runtime-determined shell invocation (ast pass, confidence 0.85)
```
                os.system("rm -rf {}/*".format(tmp_dir))
                self.assertEqual(spark.table("my_tab").count(), 1)
```
likely benign LLM, confidence 0.90 Test cleanup removing a temp directory via os.system with a formatted path.
Reasoning
This is a test in the PySpark catalog suite that removes a temporary directory with os.system and a formatted path. The command is built from a tmp_dir the test controls, not from external input. Unsafe-looking shell use, but it is ordinary test cleanup rather than malicious.
info ~~warn~~ → info by review decoder-import-with-exec python/pyspark/sql/tests/test_types.py:1951:30

file imports decoder module(s) pickle and calls `exec`/`eval`/`compile` — multi-stage payload shape (ast pass, confidence 0.80)
```
        for instance in instances:
            self.assertEqual(eval(repr(instance)), instance)

    def test_daytime_interval_type_constructor(self):
```
likely benign LLM, confidence 0.90 Round-trip repr/eval check of type instances in a unit test.
Reasoning
The test calls eval(repr(instance)) to check round-trip representation of Spark types. The pickle import is incidental; this is a round-trip assertion in a test, not a multi-stage payload.
info ~~warn~~ → info by review dynamic-execution python/pyspark/sql/tests/test_types.py:1951:30

`eval` called on a non-literal expression (ast pass, confidence 0.70)
```
        for instance in instances:
            self.assertEqual(eval(repr(instance)), instance)

    def test_daytime_interval_type_constructor(self):
```
likely benign LLM, confidence 0.90 eval of repr output of test-created type instances.
Reasoning
eval on repr of locally constructed type objects in a test assertion. Legitimate test technique, not attacker-controlled input.
info ~~warn~~ → info by review dynamic-execution python/pyspark/sql/tests/test_types.py:2272:48

`eval` called on a non-literal expression (ast pass, confidence 0.70)
```
        # check repr
        self.assertEqual(str(variants[0]), str(eval(repr(variants[0]))))

        metadata = bytes([1, 0, 0])
```
likely benign LLM, confidence 0.90 eval of repr output of a type object in a test assertion.
Reasoning
Same pattern: eval(repr(variants[0])) inside a test checking string representation of a type. Benign test code.
info ~~critical~~ → info by review dynamic-import python/pyspark/sql/types.py:1726:13

`__import__` called on a non-literal expression (ast pass, confidence 0.70)
```
        pyClass = pyUDT[split + 1 :]
        m = __import__(pyModule, globals(), locals(), [pyClass])
        if not hasattr(m, pyClass):
            raise PySparkValueError(
```
likely benign LLM, confidence 0.85 Dynamic __import__ used to resolve a UDT class from its module path.
Reasoning
This is the UDT deserialization path that imports a module named from a stored class path, then checks hasattr. It is a dynamic import that can be abused in principle, but it mirrors the expected design for loading user-defined types by module name and is a known legitimate pattern in PySpark.
info ~~critical~~ → info by review dynamic-execution python/pyspark/testing/utils.py:260:16

`os.popen` called on a non-literal command — runtime-determined shell invocation (ast pass, confidence 0.85)
```
        script = "$(test $(tput colors)) && $(test $(tput colors) -ge 8) && echo true || echo false"
        return os.popen(script).read()
    except Exception:
        return False
```
likely benign LLM, confidence 0.85 Fixed tput shell probe to detect terminal color support.
Reasoning
The command is a fixed string checking terminal color support via tput, with no external input. os.popen on a constant is benign, even though the finding labels it non-literal.
info ~~warn~~ → info by review dynamic-execution python/pyspark/worker.py:1408:32

`eval` called on a non-literal expression (ast pass, confidence 0.70)
```
                    # iterator is a tuple in the form of (pandas.DataFrame, arrow_return_type).
                    yield from eval(*[a[o] for o in args_kwargs_offsets])
                if terminate is not None:
                    yield from terminate()
```
likely benign LLM, confidence 0.90 Worker applies the user's UDF to batched arguments via a local eval name.
Reasoning
In the worker, eval is used as the UDF function invoked on batched arguments; this is the core Python UDF execution mechanism in PySpark, operating on user-supplied functions deliberately. Expected behavior for this component.
info ~~warn~~ → info by review dynamic-execution python/pyspark/worker.py:1500:27

`eval` called on a non-literal expression (ast pass, confidence 0.70)
```
                for a in it:
                    yield eval(*[a[o] for o in args_kwargs_offsets])
                if terminate is not None:
                    yield terminate()
```
likely benign LLM, confidence 0.90 Worker invokes the UDF on batched arguments, part of normal UDF execution.
Reasoning
Same as the sibling worker finding: eval here invokes the pickled UDF callable in the Spark worker, the designed execution path for Python UDFs.

info dynamic-execution python/run-tests-with-coverage:58:1

command name is a variable (`$COV_EXEC`) that this file sets only to plain text (ast pass, confidence 0.40)

echo "Combining collected coverage data under $COVERAGE_DIR/coverage_data"
$COV_EXEC combine
echo "Creating XML report file at python/coverage.xml"
$COV_EXEC xml --ignore-errors --include "pyspark/*" --omit "*/pyspark/cloudpickle/*,*/pyspark/sql/connect/proto/*,*/python/pyspark/loose_version.py"

info dynamic-execution python/run-tests-with-coverage:60:1

command name is a variable (`$COV_EXEC`) that this file sets only to plain text (ast pass, confidence 0.40)

echo "Creating XML report file at python/coverage.xml"
$COV_EXEC xml --ignore-errors --include "pyspark/*" --omit "*/pyspark/cloudpickle/*,*/pyspark/sql/connect/proto/*,*/python/pyspark/loose_version.py"
echo "Reporting the coverage data at $COVERAGE_DIR/coverage_data/coverage"

info dynamic-execution python/run-tests-with-coverage:62:1

command name is a variable (`$COV_EXEC`) that this file sets only to plain text (ast pass, confidence 0.40)

echo "Reporting the coverage data at $COVERAGE_DIR/coverage_data/coverage"
$COV_EXEC report --include "pyspark/*" --omit "*/pyspark/cloudpickle/*,*/pyspark/sql/connect/proto/*,*/python/pyspark/loose_version.py"
echo "Generating HTML files for PySpark coverage under $COVERAGE_DIR/htmlcov"

info dynamic-execution python/run-tests-with-coverage:64:1

command name is a variable (`$COV_EXEC`) that this file sets only to plain text (ast pass, confidence 0.40)

echo "Generating HTML files for PySpark coverage under $COVERAGE_DIR/htmlcov"
$COV_EXEC html --ignore-errors --include "pyspark/*" --directory "$COVERAGE_DIR/htmlcov" --omit "*/pyspark/cloudpickle/*,*/pyspark/sql/connect/proto/*,*/python/pyspark/loose_version.py"

popd

info ~~critical~~ → info by review dynamic-execution python/run-tests.py:151:17

`os.system` called on a non-literal command — runtime-determined shell invocation (ast pass, confidence 0.85)
```
            if platform.system() == "Darwin":
                os.system("rm -rf " + tmp_dir)
            else:
                shutil.rmtree(tmp_dir, ignore_errors=True)
```
likely benign LLM, confidence 0.85 Test runner removes a temporary directory via os.system.
Reasoning
Test runner cleanup removing a temp directory with os.system on a concatenated path. The path comes from the runner's own temp directory, so it is a poor-practice shell call but not malicious.
info ~~warn~~ → info by review narrow-file-charset python/test_support/sql/streaming/time/text-with-time-test.txt:1:1

file uses only 10 distinct printable characters ("-0123456:;") — JSF*ck-style or character-constrained obfuscation (raw pass, confidence 0.90)
```
2024-05-24 15:03:20;1
2024-05-24 15:03:21;2
2024-05-24 15:03:24;3
```
likely benign LLM, confidence 0.90 Plain-text streaming test fixture of timestamps and counters.
Reasoning
The file is timestamped test data with digits, colons, dashes and semicolons, which naturally uses a small character set. Not obfuscation.

Reproduce This Scan

Run disclude locally to verify. Add --format sarif for SARIF output. Add --llm (with an API key in the environment) to include the LLM review.

cargo install disclude --version 4.3.0
git clone https://github.com/apache/spark spark && cd spark
git checkout v4.0.5
disclude scan --untrusted --severity info --diff v4.0.4 python