pyspark 4.1.4

Source: https://github.com/apache/spark
Tag scanned: v4.1.4
Directory: python (only this part of the repository was scanned)
Compared with: v4.1.3 (0 signals introduced since)
Files: 1324 scanned, 43 skipped (binary, oversized, or unrecognised)
Signals: 0 critical · 0 warn · 38 info
Scan time: 27.1 s
LLM review: Reviewed by claude-haiku-5-5 (Anthropic): 32 of 32 warn and critical signals; 32 demoted to info.
Disclude version: 4.3.0
Scanned (UTC): 2026-10-28 14:29

Signals

Ordered by severity, then signals introduced in this release. Hidden characters are shown as ⟨U+…⟩.

info ~~warn~~ → info by review ifs-manipulation python/docs/source/getting_started/install.rst:179:64

[embedded bash] `IFS` set to `:` — non-standard separator enables word-splitting obfuscation (ast pass, confidence 0.80)
```
    export SPARK_HOME=`pwd`
    export PYTHONPATH=$(ZIPS=("$SPARK_HOME"/python/lib/*.zip); IFS=:; echo "${ZIPS[*]}"):$PYTHONPATH
```
likely benign LLM, confidence 0.90 Docs bash example that sets IFS to colon to build PYTHONPATH from zip files.
Reasoning
Documentation example of shell IFS to join a list of zip paths for PYTHONPATH. Standard word-splitting usage in a docs snippet, not obfuscation.
info ~~critical~~ → info by review dynamic-execution python/packaging/classic/setup.py:48:5

`exec` at module scope (runs on import) on a non-literal expression (ast pass, confidence 0.80)
```
try:
    exec(open("pyspark/version.py").read())
except IOError:
    print(
```
likely benign LLM, confidence 0.85 setup.py executes the local version.py to obtain the package version.
Reasoning
Standard PySpark packaging idiom: exec of pyspark/version.py to read the version string from a file in the source tree. Flagged as critical due to exec pattern, but the source is a local version file, not remote or encoded data.
info ~~critical~~ → info by review dynamic-execution python/packaging/client/setup.py:51:5

`exec` at module scope (runs on import) on a non-literal expression (ast pass, confidence 0.80)
```
try:
    exec(open("pyspark/version.py").read())
except IOError:
    print(
```
likely benign LLM, confidence 0.85 setup.py execs pyspark/version.py to read the version.
Reasoning
Same benign version-file exec idiom as the classic packaging setup script; reads a local module to set the version.
info ~~critical~~ → info by review dynamic-execution python/packaging/connect/setup.py:74:9

`exec` at module scope (runs on import) on a non-literal expression (ast pass, confidence 0.80)
```
    try:
        exec(open("pyspark_connect/version.py").read())
    except IOError:
        print(
```
likely benign LLM, confidence 0.85 setup.py execs the local connect version file to set the version.
Reasoning
Same benign pattern: exec of local pyspark_connect/version.py for version metadata during packaging.
info ~~critical~~ → info by review dynamic-import python/pyspark/cloudpickle/cloudpickle.py:469:5

`__import__` called on a non-literal expression (ast pass, confidence 0.70)
```
    # module.
    __import__(name)
    return sys.modules[name]
```
likely benign LLM, confidence 0.90 cloudpickle helper that imports a module by its dotted name during deserialization.
Reasoning
cloudpickle's subimport helper imports a module by name, a well-known legitimate mechanism for reconstructing modules during unpickling.
info ~~warn~~ → info by review dynamic-execution python/pyspark/errors/exceptions/tblib.py:209:17

`exec` called on a non-literal expression (ast pass, confidence 0.70)
```
            try:
                exec(
                    code, dict(current.tb_frame.f_globals), dict(current.tb_frame.f_locals)
                )  # noqa: S102
```
likely benign LLM, confidence 0.85 tblib re-raises traceback code via exec with the original frame globals and locals.
Reasoning
tblib's reraise/exec of traceback code objects is a known library pattern used to re-raise exceptions with original frames; the noqa comment indicates intentional use, not malicious.
info ~~warn~~ → info by review frame-introspection python/pyspark/errors/utils.py:242:17

`inspect.currentframe` — call-stack frame introspection (ast pass, confidence 0.70)
```
    def inspect_stack() -> Iterator[FrameType]:
        frame = inspect.currentframe()
        while frame:
            yield frame
```
likely benign LLM, confidence 0.90 Walks the call stack via inspect.currentframe to build error context.
Reasoning
Error-reporting utility walking the call stack to find the user's frame for error context; ordinary introspection.
info ~~warn~~ → info by review frame-introspection python/pyspark/logger/worker_io.py:233:17

`inspect.currentframe` — call-stack frame introspection (ast pass, confidence 0.70)
```
    try:
        frame = inspect.currentframe()
        is_in_pyspark_module = False
```
likely benign LLM, confidence 0.90 Uses inspect.currentframe to check whether logging originates inside pyspark.
Reasoning
Logging helper inspecting frames to determine whether the call originates from pyspark; benign.
info ~~critical~~ → info by review dynamic-import python/pyspark/ml/connect/io_utils.py:67:9

`__import__` called on a non-literal expression (ast pass, confidence 0.70)
```
    module = ".".join(parts[:-1])
    m = __import__(module, fromlist=[parts[-1]])
    return getattr(m, parts[-1])
```
likely benign LLM, confidence 0.88 Loads a class by dotted path using __import__ for ML model deserialization.
Reasoning
Standard class-path loader that imports a module from a dotted name taken from saved metadata, a common ML persistence pattern; the name comes from the library's own serialized class names rather than attacker-controlled network input.
info ~~critical~~ → info by review dynamic-import python/pyspark/ml/connect/readwrite.py:269:21

`__import__` called on a non-literal expression (ast pass, confidence 0.70)
```
                module = ".".join(parts[:-1])
                m = __import__(module, fromlist=[parts[-1]])
                return getattr(m, parts[-1])
```
likely benign LLM, confidence 0.88 Dynamic import of an ML class from its dotted name during model loading.
Reasoning
Same class-by-name import helper for ML read/write; legitimate reflection on library class names.
info ~~critical~~ → info by review dynamic-import python/pyspark/ml/util.py:990:13

`__import__` called on a non-literal expression (ast pass, confidence 0.70)
```
        module = ".".join(parts[:-1])
        m = __import__(module, fromlist=[parts[-1]])
        return getattr(m, parts[-1])
```
likely benign LLM, confidence 0.88 Resolves an ML class by dotted name via __import__.
Reasoning
Standard ML utility resolving a class from a dotted name for reconstruction; legitimate reflection.
info ~~critical~~ → info by review dynamic-import python/pyspark/ml/wrapper.py:313:17

`__import__` called on a non-literal expression (ast pass, confidence 0.70)
```
            module = ".".join(parts[:-1])
            m = __import__(module, fromlist=[parts[-1]])
            return getattr(m, parts[-1])
```
likely benign LLM, confidence 0.88 Resolves an ML wrapper class by dotted name via __import__.
Reasoning
Same ML class-resolution pattern in the Java wrapper; legitimate reflection.
info ~~critical~~ → info by review frame-introspection python/pyspark/pandas/sql_processor.py:208:16

`inspect.stack` — call-stack frame introspection — file also calls `sys.exit`/`exec`/`eval` (anti-analysis shape) (ast pass, confidence 0.85)
```
    try:
        return inspect.stack()[_CAPTURE_SCOPES][0].f_locals
    except IndexError:
        return {}
```
likely benign LLM, confidence 0.80 Retrieves caller frame locals via inspect.stack to resolve SQL variable references.
Reasoning
Pandas-on-Spark SQL helper reads caller locals via inspect.stack to resolve variables for query interpolation. The anti-analysis note is a heuristic artifact of exec/eval appearing in the file; the code itself is legitimate frame introspection.
info ~~warn~~ → info by review frame-introspection python/pyspark/pipelines/source_code_location.py:65:37

`inspect.currentframe` — call-stack frame introspection (ast pass, confidence 0.70)
```
    # Stack is ordered from the caller first to the callee last.
    stack = traceback.extract_stack(inspect.currentframe())

    frame = stack[-(stacklevel + 2)]
```
likely benign LLM, confidence 0.90 Extracts the caller's source location via traceback and currentframe.
Reasoning
Computes source location for pipeline definitions by inspecting the stack; ordinary introspection.
info ~~warn~~ → info by review dynamic-execution python/pyspark/serializers.py:523:16

`marshal.loads` deserializes Python bytecode from bytes — code-object sink (ast pass, confidence 0.75)
```
    def loads(self, obj):
        return marshal.loads(obj)
```
likely benign LLM, confidence 0.85 Marshal-based serializer loads for PySpark internal data transfer.
Reasoning
PySpark's marshal serializer for Python bytecode exchange between its own driver and workers; the marshal use is the documented internal serialization format, not attacker-facing input.
info ~~warn~~ → info by review dynamic-execution python/pyspark/serializers.py:548:20

`marshal.loads` deserializes Python bytecode from bytes — code-object sink (ast pass, confidence 0.75)
```
        if _type == b"M":
            return marshal.loads(obj[1:])
        elif _type == b"P":
            return pickle.loads(obj[1:])
```
likely benign LLM, confidence 0.85 Type-tagged deserializer choosing marshal or pickle for internal PySpark data.
Reasoning
Type-prefixed deserialization dispatching to marshal or pickle for internal PySpark worker communication; the same legitimate serializer design as the adjacent code.
info ~~warn~~ → info by review decoder-decompress-payload python/pyspark/serializers.py:569:38

file imports decoder module(s) marshal, zlib, pickle and calls `.decompress()` — embedded payload decompressed at runtime (ast pass, confidence 0.75)
```
    def loads(self, obj):
        return self.serializer.loads(zlib.decompress(obj))

    def __repr__(self):
```
likely benign LLM, confidence 0.90 A serializer wrapper decompresses zlib-compressed data before deserializing it.
Reasoning
The loads method wraps a serializer's payload with zlib.decompress in a compressed-serializer wrapper. This is ordinary data compression for Spark's serialization, not a hidden payload.
info ~~critical~~ → info by review dynamic-execution python/pyspark/shell.py:142:16

`compile` at module scope (runs on import) on a non-literal expression (ast pass, confidence 0.80)
```
    with open(_pythonstartup) as f:
        code = compile(f.read(), _pythonstartup, "exec")
        exec(code)
```
likely benign LLM, confidence 0.85 The PySpark shell compiles and executes the file named by PYTHONSTARTUP.
Reasoning
The shell runs the user's own PYTHONSTARTUP file via compile and exec, which is the documented startup-script behaviour of the PySpark interactive shell. The code is user-supplied configuration, not an embedded payload, though exec at import time is worth noting.
info ~~critical~~ → info by review dynamic-execution python/pyspark/shell.py:143:9

`exec` at module scope (runs on import) on a non-literal expression (ast pass, confidence 0.80)
```
        code = compile(f.read(), _pythonstartup, "exec")
        exec(code)
```
likely benign LLM, confidence 0.85 Executes the compiled PYTHONSTARTUP code in the shell.
Reasoning
This exec call is the second half of the same PYTHONSTARTUP handling, executing the compiled user startup file. It is the expected behaviour of the shell.
info encoding-base64 python/pyspark/sql/tests/pandas/streaming/test_transform_with_state_state_variable_checkpoint_v2.py:26:7

base64-like blob (66 bytes, compression ratio 1.17) (outside string/comment) (raw pass, confidence 0.30)
```
class TransformWithStateInPySparkStateVariableWithCheckpointV2TestsMixin(
    TransformWithStateInPySparkStateVariableTestsMixin
):
```
info encoding-base64 python/pyspark/sql/tests/pandas/streaming/test_transform_with_state_state_variable_checkpoint_v2.py:37:5

base64-like blob (66 bytes, compression ratio 1.17) (outside string/comment) (raw pass, confidence 0.30)
```
class TransformWithStateInPySparkStateVariableWithCheckpointV2Tests(
    TransformWithStateInPySparkStateVariableWithCheckpointV2TestsMixin, ReusedSQLTestCase
):
    pass
```
info ~~critical~~ → info by review dynamic-execution python/pyspark/sql/tests/test_catalog.py:493:17

`os.system` called on a non-literal command — runtime-determined shell invocation (ast pass, confidence 0.85)
```
                os.system("rm -rf {}/*".format(tmp_dir))
                self.assertEqual(spark.table("my_tab").count(), 1)
```
likely benign LLM, confidence 0.85 A test cleans up a temp directory via a shell rm command.
Reasoning
A unit test removes a temporary directory with os.system using a format string built from a test-created temp path. This is test cleanup with no untrusted input.
info ~~warn~~ → info by review decoder-import-with-exec python/pyspark/sql/tests/test_types.py:2088:30

file imports decoder module(s) pickle and calls `exec`/`eval`/`compile` — multi-stage payload shape (ast pass, confidence 0.80)
```
        for instance in instances:
            self.assertEqual(eval(repr(instance)), instance)

    def test_daytime_interval_type_constructor(self):
```
likely benign LLM, confidence 0.90 A test uses eval on repr output to verify type round-tripping.
Reasoning
The test imports pickle and calls eval(repr(instance)) to check round-tripping of type objects. The pickle import and eval are in a test checking repr behaviour, not a multi-stage payload.
info ~~warn~~ → info by review dynamic-execution python/pyspark/sql/tests/test_types.py:2088:30

`eval` called on a non-literal expression (ast pass, confidence 0.70)
```
        for instance in instances:
            self.assertEqual(eval(repr(instance)), instance)

    def test_daytime_interval_type_constructor(self):
```
likely benign LLM, confidence 0.90 A test evaluates the repr of type instances to compare them.
Reasoning
The eval call is the repr round-trip assertion in the same test; its input is the object's own repr in a unit test.
info ~~warn~~ → info by review dynamic-execution python/pyspark/sql/tests/test_types.py:2409:48

`eval` called on a non-literal expression (ast pass, confidence 0.70)
```
        # check repr
        self.assertEqual(str(variants[0]), str(eval(repr(variants[0]))))

        metadata = bytes([1, 0, 0])
```
likely benign LLM, confidence 0.90 A test evaluates a variant's repr to verify its string form.
Reasoning
The test evals the repr of a variant to check its string form. Input is generated in the test itself, so this is a benign self-check.
info ~~critical~~ → info by review dynamic-import python/pyspark/sql/types.py:1991:13

`__import__` called on a non-literal expression (ast pass, confidence 0.70)
```
        pyClass = pyUDT[split + 1 :]
        m = __import__(pyModule, globals(), locals(), [pyClass])
        if not hasattr(m, pyClass):
            raise PySparkValueError(
```
likely benign LLM, confidence 0.70 Dynamically imports the module of a user-defined type from its JSON class path.
Reasoning
The UDT loader imports the module named in a serialized UDT class path to recover the class. This is dynamic import driven by schema data, which carries some risk from untrusted JSON, but it matches the documented UDT deserialization design.
info ~~critical~~ → info by review dynamic-execution python/pyspark/testing/utils.py:332:16

`os.popen` called on a non-literal command — runtime-determined shell invocation (ast pass, confidence 0.85)
```
        script = "$(test $(tput colors)) && $(test $(tput colors) -ge 8) && echo true || echo false"
        return os.popen(script).read()
    except Exception:
        return False
```
likely benign LLM, confidence 0.90 Runs a constant shell check to see whether the terminal supports colours.
Reasoning
The os.popen argument is a fixed shell test string checking terminal colour support, so the command is not attacker-controlled despite the flag.
info ~~warn~~ → info by review dynamic-execution python/pyspark/worker.py:2245:32

`eval` called on a non-literal expression (ast pass, confidence 0.70)
```
                    # iterator is a tuple in the form of (pandas.DataFrame, arrow_return_type).
                    yield from eval(*[a[o] for o in args_kwargs_offsets], num_rows=len(a[0]))
                if terminate is not None:
                    yield from terminate()
```
likely benign LLM, confidence 0.90 Calls a UDTF's eval method on batch arguments.
Reasoning
eval here is a local UDTF evaluation function invoked on Arrow batch arguments, not Python's builtin eval. The non-literal flag is a false positive from the name.

info ~~warn~~ → info by review dynamic-execution python/pyspark/worker.py:2420:32

`eval` called on a non-literal expression (ast pass, confidence 0.70)

                    # iterator is a tuple in the form of (pyarrow.RecordBatch, arrow_return_type).
                    yield from eval(*[pylist[o] for o in args_kwargs_offsets], num_rows=a.num_rows)
                if terminate is not None:
                    yield from terminate()

likely benign LLM, confidence 0.90 Calls a UDTF's eval method on Arrow record batch arguments.

Reasoning

Same pattern: a UDTF eval method applied to RecordBatch arguments within the worker's data path, not builtin eval.

info ~~warn~~ → info by review dynamic-execution python/pyspark/worker.py:2529:32

`eval` called on a non-literal expression (ast pass, confidence 0.70)
```
                    # For PyArrow UDTFs, pass RecordBatches directly (no row conversion needed)
                    yield from eval(*[a[o] for o in args_kwargs_offsets])
                if terminate is not None:
                    yield from terminate()
```
likely benign LLM, confidence 0.90 Calls a UDTF's eval method on Arrow record batches.
Reasoning
Same UDTF eval method call, applied to Arrow record batches. The name collides with builtin eval and there is no dynamic code evaluation.
info ~~warn~~ → info by review dynamic-execution python/pyspark/worker.py:2631:27

`eval` called on a non-literal expression (ast pass, confidence 0.70)
```
                for a in it:
                    yield eval(*[a[o] for o in args_kwargs_offsets])
                if terminate is not None:
                    yield terminate()
```
likely benign LLM, confidence 0.90 Calls a UDTF's eval method over argument rows.
Reasoning
Same pattern again: a UDTF eval method call over row arguments, a legitimate worker function, not dynamic execution.

info dynamic-execution python/run-tests-with-coverage:58:1

command name is a variable (`$COV_EXEC`) that this file sets only to plain text (ast pass, confidence 0.40)

echo "Combining collected coverage data under $COVERAGE_DIR/coverage_data"
$COV_EXEC combine
echo "Creating XML report file at python/coverage.xml"
$COV_EXEC xml --ignore-errors --include "pyspark/*" --omit "*/pyspark/cloudpickle/*,*/pyspark/sql/connect/proto/*,*/python/pyspark/loose_version.py"

info dynamic-execution python/run-tests-with-coverage:60:1

command name is a variable (`$COV_EXEC`) that this file sets only to plain text (ast pass, confidence 0.40)

echo "Creating XML report file at python/coverage.xml"
$COV_EXEC xml --ignore-errors --include "pyspark/*" --omit "*/pyspark/cloudpickle/*,*/pyspark/sql/connect/proto/*,*/python/pyspark/loose_version.py"
echo "Reporting the coverage data at $COVERAGE_DIR/coverage_data/coverage"

info dynamic-execution python/run-tests-with-coverage:62:1

command name is a variable (`$COV_EXEC`) that this file sets only to plain text (ast pass, confidence 0.40)

echo "Reporting the coverage data at $COVERAGE_DIR/coverage_data/coverage"
$COV_EXEC report --include "pyspark/*" --omit "*/pyspark/cloudpickle/*,*/pyspark/sql/connect/proto/*,*/python/pyspark/loose_version.py"
echo "Generating HTML files for PySpark coverage under $COVERAGE_DIR/htmlcov"

info dynamic-execution python/run-tests-with-coverage:64:1

command name is a variable (`$COV_EXEC`) that this file sets only to plain text (ast pass, confidence 0.40)

echo "Generating HTML files for PySpark coverage under $COVERAGE_DIR/htmlcov"
$COV_EXEC html --ignore-errors --include "pyspark/*" --directory "$COVERAGE_DIR/htmlcov" --omit "*/pyspark/cloudpickle/*,*/pyspark/sql/connect/proto/*,*/python/pyspark/loose_version.py"

popd

info ~~critical~~ → info by review dynamic-execution python/run-tests.py:153:17

`os.system` called on a non-literal command — runtime-determined shell invocation (ast pass, confidence 0.85)
```
            if platform.system() == "Darwin":
                os.system("rm -rf " + tmp_dir)
            else:
                shutil.rmtree(tmp_dir, ignore_errors=True)
```
likely benign LLM, confidence 0.85 The test runner deletes a temp directory with a shell rm on macOS.
Reasoning
The test runner removes a temp directory via os.system with a path from tempfile. The path is internally generated, and the non-Darwin branch uses shutil.rmtree, so this is ordinary cleanup.
info ~~critical~~ → info by review dynamic-execution python/run-with-viztracer:69:1

`exec` called with a dynamic (variable/substitution) path (ast pass, confidence 0.85)
```
exec "$@"
```
likely benign LLM, confidence 0.90 Bash launcher that execs its arguments under the viztracer profiler.
Reasoning
A wrapper script ending in exec "$@" is the standard idiom for running a command with a prefix tool (viztracer) and passing through arguments. The argument source is the caller's own command line, so the dynamic path is expected for a launcher wrapper, not a dropper.
info ~~warn~~ → info by review narrow-file-charset python/test_support/sql/streaming/time/text-with-time-test.txt:1:1

file uses only 10 distinct printable characters ("-0123456:;") — JSF*ck-style or character-constrained obfuscation (raw pass, confidence 0.90)
```
2024-05-24 15:03:20;1
2024-05-24 15:03:21;2
2024-05-24 15:03:24;3
```
dismissed LLM, confidence 0.97 Plain-text test fixture containing timestamped rows for SQL streaming tests.
Reasoning
The file is test data for Spark streaming timestamp parsing: lines of dates, times, and row numbers. A narrow digit-and-punctuation charset is the natural result of timestamp content and is not an obfuscation technique.

Reproduce This Scan

Run disclude locally to verify. Add --format sarif for SARIF output. Add --llm (with an API key in the environment) to include the LLM review.

cargo install disclude --version 4.3.0
git clone https://github.com/apache/spark spark && cd spark
git checkout v4.1.4
disclude scan --untrusted --severity info --diff v4.1.3 python