disclude.iopyspark 4.2.1
- Source
- https://github.com/apache/spark
- Tag scanned
v4.2.1- Directory
python- Compared with
v4.2.0(0 signals introduced since)- Files
- 1367 scanned, 58 skipped (binary, oversized, or unrecognised)
- Signals
- 0 critical · 90 warn · 123 info
- Scan time
- 82.4 s
- LLM review
- Reviewed by claude-haiku-5-5 (Anthropic): the 117 most severe of 207 warn and critical signals (117 with a verdict); 117 demoted to info. The rest are unreviewed.
- Disclude version
- 4.3.0
- Scanned (UTC)
- 2026-10-28 14:30
Signals
Ordered by severity, then signals introduced in this release. Hidden characters are shown as ⟨U+…⟩.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
# iterator is a tuple in the form of (pandas.DataFrame, arrow_return_type). yield from yield from -
# iterator is a tuple in the form of (pyarrow.RecordBatch, arrow_return_type). yield from yield from -
# For PyArrow UDTFs, pass RecordBatches directly (no row conversion needed) yield from yield from -
yield yield -
= -
2024-05-24 15:03:20;1 2024-05-24 15:03:21;2 2024-05-24 15:03:24;3 -
; IFS=:;likely benign Docs example building PYTHONPATH from py4j zips using IFS=:.Reasoning
IFS=: is a standard idiom for building a colon-separated PYTHONPATH from a glob array in a documentation shell example. No obfuscation or malicious intent.
-
likely benign Build script exec-ing the package's own version.py to read the version.
Reasoning
This is PySpark's standard setup.py reading pyspark/version.py to get __version__ at build time. The exec is on a local, repo-controlled file, a common and legitimate packaging pattern.
-
likely benign Client packaging setup.py exec-ing local version.py for the version string.
Reasoning
Same legitimate version-reading pattern in the client packaging setup script, exec of a local version file.
-
likely benign Connect packaging setup.py exec-ing local version.py.
Reasoning
Same legitimate version-file exec in the connect packaging script; reads a repo-local file at build time.
-
# module. returnlikely benign cloudpickle subimport helper importing a module by name.Reasoning
cloudpickle's subimport helper imports a module by name to ensure it is loaded before lookup. Dynamic import of a module name is a standard, legitimate pattern in this library.
-
= . # type: ignorelikely benign Vendored tblib exec of traceback code objects while rebuilding frames.Reasoning
tblib reconstructs traceback frames and re-executes code objects in captured frame globals for error reconstruction. Vendored traceback-handling code; the exec is of a code object from the traceback, not attacker-supplied input, so it is a legitimate pattern here.
-
= yieldlikely benign Error utility walking the stack to find call-site frames.Reasoning
Frame walking via inspect.currentframe is used to locate the user's call site for error messages. Ordinary introspection in a library.
-
= = Falselikely benign Logger checks call-stack frames to detect pyspark-internal callers.Reasoning
Uses currentframe to determine whether the caller is inside the pyspark module for logging. Legitimate introspection with no network or execution behavior.
-
in : returnlikely benign Module __getattr__ resolving names from module globals.Reasoning
Module-level __getattr__ fallback resolving names from globals is a standard lazy-attribute pattern. Benign.
-
= = Truelikely benign Populating module globals from locals during conditional import setup.Reasoning
Copies optional-import names into module globals when the profiler dependency is available. Standard conditional-import plumbing, no malicious behavior.
-
= = Truelikely benign Reading locals() to export conditionally imported names.Reasoning
Same conditional-import globals population; reading locals() here is benign.
-
= = returnlikely benign Loads a class from a dotted path for ML persistence.Reasoning
Resolves a class path string to a class for ML model loading, a standard reflection pattern. The name comes from the library's own metadata, not an external payload.
-
= = returnlikely benign Dynamic class loading by dotted name during ML model read.Reasoning
Same class-path resolution for ML read/write of estimators; standard reflection in the library.
-
= = returnlikely benign Dotted-path class import for ML persistence utilities.Reasoning
Dotted-path class loading used to restore ML pipeline stages; standard reflection with no suspicious payload.
-
= = returnlikely benign Dynamic import of ML wrapper classes by dotted name.Reasoning
Resolves Java/Python wrapper class names by path for ML wrappers; ordinary library reflection.
-
return . returnlikely benign Captures caller local variables via inspect.stack for SQL parameter binding.Reasoning
inspect.stack is used to capture the caller's locals for pandas-on-Spark SQL variable resolution, a legitimate feature. The co-occurrence of exit/exec/eval in the file is common in a large library and does not indicate anti-analysis here.
-
info
warn→ info by reviewframe-introspectionpython/pyspark/pipelines/source_code_location.py:41:37# Stack is ordered from the caller first to the callee last. = =likely benign PySpark pipeline helper that reads the call stack to find the user's source location for a declaration.Reasoning
Using inspect.currentframe with traceback.extract_stack to locate the caller's source location is a standard way for PySpark pipeline code to record where user declarations originate. The snippet shows only stack indexing for a stacklevel offset, with no network, exec, or process activity. This is a legitimate use of frame introspection.
-
returnlikely benign Marshal-based deserializer method used by PySpark's internal serializer.Reasoning
marshal.loads is used in PySpark's serializer for its own cloudpickle/marshal protocol on data exchanged between PySpark's own Python workers; it is a known and documented trust boundary within the framework, not an injected payload.
-
return == b: returnlikely benign Dispatches deserialization by a type byte to marshal or pickle loaders.Reasoning
Type-tagged dispatch between marshal and pickle for PySpark's internal framed serialization. Legitimate framework code; the deserialization trust is inherent to Spark's worker protocol rather than hidden malice.
-
returnlikely benign Decompresses zlib data before delegating to an inner serializer.Reasoning
zlib decompression wrapping a serializer's loads is a standard compressed-stream pattern in PySpark's serializer stack; the payload is ordinary serialized data flowing through Spark's own protocol.
-
=likely benign Compiles and executes the user's PYTHONSTARTUP file when the Spark shell starts.Reasoning
The PYTHONSTARTUP handling in the PySpark interactive shell mirrors the standard Python interpreter behavior of executing the user's own startup file. It runs a user-configured local file, not remote or hidden content, so it is expected for a shell.
-
=likely benign Executes the PYTHONSTARTUP script in the Spark shell namespace.Reasoning
Executes the code compiled from the user's PYTHONSTARTUP file, matching the interpreter's own startup behavior; no network or encoding involved. Critical severity is overstated for this legitimate shell feature.
-
info
encoding-base64python/pyspark/sql/tests/pandas/streaming/test_transform_with_state_state_variable_checkpoint_v2.py:25:7 -
info
encoding-base64python/pyspark/sql/tests/pandas/streaming/test_transform_with_state_state_variable_checkpoint_v2.py:36:5pass -
likely benign Test helper shells out to rm -rf to clear a temporary directory.
Reasoning
A test cleaning up a temporary directory with rm -rf on a path formed from a tmp_dir variable. Test-only cleanup; the command is fixed apart from a local temp path, a common though not ideal pattern.
-
likely benign Round-trip test evaluating repr of data type instances.
Reasoning
The test evaluates repr() of type instances to check round-tripping. The pickle import is incidental to the test module; eval here is on a local repr, not attacker data.
-
likely benign Test uses eval on repr output of local type instances.
Reasoning
eval of repr(instance) inside a unit test asserting round-trip equality; the input is locally constructed type objects, not untrusted data.
-
# check repr =likely benign Test evaluates repr of a local type variant to verify round-tripping.Reasoning
Same round-trip repr/eval test pattern on locally created variants, asserting str equality. No external input.
-
= =likely benign Imports the module named in a UDT's serialized Python class path to restore the type.Reasoning
Dynamic import of a UDT class from a serialized class path is how PySpark restores user-defined types from JSON schema; it imports a module named in the schema, which is a known design trait but not hidden malice in itself.
-
= return return Falselikely benign Checks terminal color capability by running a fixed tput command.Reasoning
Runs a fixed shell probe (tput colors) via os.popen to detect terminal color support; the command is a constant string, so the non-literal flag is a false positive.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:1:1likely benign Golden markdown table of PyArrow scalar cast results with very long lines.Reasoning
A golden file of a wide markdown table of cast compatibility; long lines are expected for wide tables and carry no code.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:2:1likely benign Markdown table separator line in a golden data file.Reasoning
Table separator row of the same golden markdown file; wide-table formatting with no executable or hidden content.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:3:1likely benign Golden markdown table of PyArrow scalar cast outcomes with a very long row.Reasoning
A wide markdown table of golden cast results from a PySpark upstream test fixture. Long rows are expected for generated tables, and the raw long-line signal carries no obfuscation or malicious content.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:4:1likely benign Golden markdown table row for int8 negative-value PyArrow casts.Reasoning
Same golden-file table row format: int8 negative-value cast results. The long line is generated test data, not code or an encoded payload.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:5:1likely benign Golden markdown table row for int8 max/min PyArrow casts.Reasoning
Golden test table row listing boundary-value cast results for int8. The length comes from the table layout of the generated fixture and is not hidden behaviour.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:6:1likely benign Golden markdown table row for int16 standard-value PyArrow casts.Reasoning
Standard-value golden row for int16 casts in a PySpark test fixture. It is plain tabular data with no executable or encoded content.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:7:1likely benign Golden markdown table row for int16 negative-value PyArrow casts.Reasoning
Negative-value golden row for int16 casts. This is generated expected-output text in a test directory, consistent with a legitimate test fixture.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:8:1likely benign Golden markdown table row for int16 max/min PyArrow casts.Reasoning
Boundary-value golden row for int16 casts in a PySpark upstream fixture. The long line is a product of the wide table and not a sign of obfuscation.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:9:1likely benign Golden markdown table row for int32 standard-value PyArrow casts.Reasoning
Standard-value golden row for int32 casts. It is expected-output data for a test suite with no code-execution or hidden-content indicators.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:10:1likely benign Golden markdown table row for int32 negative-value PyArrow casts.Reasoning
Negative-value golden row for int32 casts in a test fixture. Long generated table rows like this are normal and carry no malicious intent.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:11:1likely benign Golden markdown table row for int32 max/min PyArrow casts.Reasoning
Boundary-value golden row for int32 casts. The content is test expectation data in a markdown fixture, and the raw long-line flag is a formatting artifact.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:12:1likely benign Golden markdown table row for int64 standard-value PyArrow casts.Reasoning
Standard-value golden row for int64 casts in a PySpark upstream test file. It is plain generated test data and the long-line signal is benign.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:13:1likely benign A markdown golden file holding a wide table of PyArrow scalar cast results.Reasoning
Raw pass long-line signal on a markdown golden test file containing a wide table of PyArrow cast results. Wide tables are a normal way to store golden test output; no executable content or obfuscation is present.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:14:1likely benign A markdown golden file row for int64 max/min cast results.Reasoning
Same golden-table long line in a markdown test fixture for PyArrow safe casts. The content is tabular expected-output data, not code or obfuscation.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:15:1likely benign A markdown golden file row for uint8 standard cast results.Reasoning
Wide golden-table row of expected cast outputs in a test fixture. Long lines in generated markdown tables are routine and carry no malicious intent.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:16:1likely benign A markdown golden file row for uint8 max cast results.Reasoning
Expected-output table row from a PyArrow cast test golden file. The long length is due to many columns of result values, consistent with benign generated test data.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:17:1likely benign A markdown golden file row for uint16 standard cast results.Reasoning
Same pattern: a wide golden-table row of cast results in a markdown test fixture. No code execution or encoded payload is visible.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:18:1likely benign A markdown golden file row for uint16 max cast results.Reasoning
Golden test table row for uint16 max cast outputs. Long generated table lines are a normal feature of such fixtures and do not indicate obfuscation.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:19:1likely benign A markdown golden file row for uint32 standard cast results.Reasoning
Expected-output row in a PyArrow golden markdown file. The raw long-line flag is a length artifact of tabular test data.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:20:1likely benign A markdown golden file row for uint32 max cast results.Reasoning
Wide golden-table row of PyArrow cast outcomes for uint32 max values. Benign test fixture content flagged only by length.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:21:1likely benign A markdown golden file row for uint64 standard cast results.Reasoning
Golden test table row for uint64 standard cast results in a markdown fixture. Long lines reflect many columns, not malicious content.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:22:1likely benign A markdown golden file row for uint64 max cast results.Reasoning
Expected-output table row for uint64 max casts in a PyArrow test golden file. The flag is only a length heuristic on benign tabular data.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:23:1likely benign A wide markdown table row in a PySpark PyArrow golden test file listing scalar cast outcomes for float16 values.Reasoning
Raw long-line signal on a markdown golden-file table of PyArrow cast results. Wide tabular test fixtures naturally produce very long lines; no obfuscation or malicious content is present.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:24:1likely benign A wide markdown table row of PyArrow cast expectations for float16 special values.Reasoning
Same golden-file table layout with a long line of expected cast results for float16 special values; a formatting artifact, not a threat.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:25:1likely benign A wide markdown table row of PyArrow cast expectations for float16 fractional values.Reasoning
Long table row from a generated golden test fixture; the raw long-line signal is a known false positive for such tables.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:26:1likely benign A wide markdown table row of PyArrow cast expectations for float32 standard values.Reasoning
Generated golden-file table row; long width is expected for a matrix of types and cast results.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:27:1likely benign A wide markdown table row of PyArrow cast expectations for float32 special values.Reasoning
Generated golden-file table with a long line; no executable or obfuscated content.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:28:1likely benign A wide markdown table row of PyArrow cast expectations for float32 fractional values.Reasoning
Long line is inherent to the wide golden-table format of cast results.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:29:1likely benign A wide markdown table row of PyArrow cast expectations for float64 standard values.Reasoning
Generated test fixture table; the long-line flag reflects table width only.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:30:1likely benign A wide markdown table row of PyArrow cast expectations for float64 special values.Reasoning
Golden-file table row with long width; benign test data.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:31:1likely benign A wide markdown table row of PyArrow cast expectations for float64 fractional values.Reasoning
Generated golden-table row; the long line is a width artifact with no code or payload.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:32:1likely benign A wide markdown table row of PyArrow cast expectations for bool values.Reasoning
Same golden-file table format; the bool row contains only cast result text, so the long-line signal is a false positive.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:33:1likely benign A generated markdown golden-file table row listing PyArrow scalar cast outcomes.Reasoning
Raw long-line signal on a generated golden markdown table of PyArrow cast results. Wide tables with many columns naturally produce long lines; no executable content or obfuscation is present.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:34:1likely benign A generated markdown golden-file table row of PyArrow string cast results.Reasoning
Same generated golden table pattern with long rows of error/result strings from casting tests; the long-line warning is a formatting artifact of a raw byte-level check.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:35:1likely benign A generated markdown golden-file table row of PyArrow unicode string cast results.Reasoning
Generated golden test data table row; long line is expected for a wide matrix of cast outcomes and carries no malicious content.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:36:1likely benign A generated markdown golden-file table row of PyArrow large_string cast results.Reasoning
Generated golden table row with the same structure as other rows; the long-line flag is a raw-pass false positive on wide tabular test output.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:37:1likely benign A generated markdown golden-file table row of PyArrow large_string alpha cast results.Reasoning
Generated golden test table; long line reflects a wide matrix of cast results and is not obfuscation.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:38:1likely benign A generated markdown golden-file table row of PyArrow large_string unicode cast results.Reasoning
Generated golden table row; the long-line warning is a formatting artifact of a wide test matrix.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:39:1likely benign A generated markdown golden-file table row of PyArrow binary cast results.Reasoning
Generated golden test data for PyArrow binary casts; long line from a wide table, no executable or hidden content.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:40:1likely benign A generated markdown golden-file table row of PyArrow large_binary cast results.Reasoning
Generated golden test data for PyArrow large_binary casts; the flag is only a line-length heuristic on wide tabular output.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:41:1likely benign A generated markdown golden-file table row of PyArrow fixed_size_binary cast results.Reasoning
Generated golden test table row for fixed_size_binary casts; long line is a formatting artifact with no malicious content.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:42:1likely benign Generated golden test-output markdown table of pyarrow scalar cast results with very wide rows.Reasoning
This is a markdown golden-output table from PySpark's upstream pyarrow test suite, with long rows of repeated ERR@ArrowInvalid cells for scalar cast results. Long lines in generated tables are ordinary, and the content is data rather than code or hidden instructions.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:43:1likely benign Wide markdown table of PyArrow scalar cast outcomes in a PySpark test fixture.Reasoning
A markdown golden-test table of PyArrow cast results with wide rows; the long line is just a wide table, with no code execution or obfuscation.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:44:1likely benign Wide markdown golden table row for decimal256 casts in PySpark tests.Reasoning
Same golden-file table pattern of cast results; long line length from table width only.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:45:1likely benign Wide markdown golden table row for decimal256 large casts.Reasoning
Golden test fixture table row; no executable or encoded content is present.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:46:1likely benign Wide markdown golden table row for date32 cast results.Reasoning
Plain table of expected error strings and values for date casts; the long-line warning is a formatting artifact.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:47:1likely benign Wide markdown golden table row for date32 negative cast results.Reasoning
Same golden table format; benign test expectation data.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:48:1likely benign Wide markdown golden table row for date64 cast results.Reasoning
Golden test table of cast outcomes for date64; no obfuscation present.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:49:1likely benign Wide markdown golden table row for date64 negative cast results.Reasoning
Benign golden-file table row with expected cast values and errors.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:50:1likely benign Wide markdown golden table row for timestamp[s] cast results.Reasoning
Timestamp cast expectations in a markdown test fixture; the long line is only table width.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:51:1likely benign Wide markdown golden table row for timestamp[s] negative cast results.Reasoning
Benign golden table data for timestamp negative casts; no code or hidden content.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:52:1likely benign Wide markdown golden table row for timestamp[ms] cast results.Reasoning
Golden fixture table row for timestamp[ms] casts; the flag is only the raw long-line heuristic on a wide table.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:53:1likely benign Wide markdown table row in a PyArrow scalar cast golden test file.Reasoning
Raw long-line signal on a markdown golden-file table of PyArrow cast results. Wide tables are normal for generated test fixtures; no code execution or hidden content.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:54:1likely benign Wide markdown table row in a PyArrow scalar cast golden test file.Reasoning
Same generated golden-table row with long-line signal; content is cast result strings typical of a test fixture.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:55:1likely benign Wide markdown table row in a PyArrow scalar cast golden test file.Reasoning
Generated golden-file table row; long-line finding is a formatting artifact with no malicious content shown.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:56:1likely benign Wide markdown table row in a PyArrow scalar cast golden test file.Reasoning
Generated golden-file table row; long-line signal only, content is test expectations.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:57:1likely benign Wide markdown table row in a PyArrow scalar cast golden test file.Reasoning
Generated golden-file table row; long-line signal only, no executable or obfuscated content.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:58:1likely benign Wide markdown table row in a PyArrow scalar cast golden test file.Reasoning
Generated golden-file table row; long-line signal only, content is cast expectations for timestamp types.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:59:1likely benign Wide markdown table row in a PyArrow scalar cast golden test file.Reasoning
Generated golden-file table row; long-line signal only, no malicious content.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:60:1likely benign Wide markdown table row in a PyArrow scalar cast golden test file.Reasoning
Generated golden-file table row; long-line signal only, no malicious content.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:61:1likely benign Wide markdown table row in a PyArrow scalar cast golden test file.Reasoning
Generated golden-file table row; long-line signal only, no malicious content.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:62:1likely benign Wide markdown table row in a PyArrow scalar cast golden test file.Reasoning
Generated golden-file table row; long-line signal only, no malicious content.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:63:1likely benign A wide row in a generated PyArrow cast golden-result table.Reasoning
Raw-pass long-line warning on a generated markdown golden table of PyArrow scalar cast results. The row is wide because it lists many cast targets. No executable content or obfuscation is present.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:64:1likely benign A wide row in a generated PyArrow cast golden-result table.Reasoning
Same generated golden table, raw long-line signal on a data row of cast outcomes (ERR@ArrowNotImplementedError and value lists). Expected output for a test fixture.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:65:1likely benign A wide row in a generated PyArrow cast golden-result table.Reasoning
Generated markdown test fixture row with repeated error-name strings and integer lists. The long line only reflects table width; nothing is encoded or executed.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:66:1likely benign A wide row in a generated PyArrow cast golden-result table.Reasoning
Raw long-line warning on a golden-result table row for timestamp casts with timezone labels. This is ordinary test data in a markdown file.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:67:1likely benign A wide row in a generated PyArrow cast golden-result table.Reasoning
Generated table row in a test golden file. The 2630-character length is a consequence of the many columns, not of hidden payloads.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:68:1likely benign A wide row in a generated PyArrow cast golden-result table.Reasoning
Raw long-line signal on a markdown golden table row with timezone-labelled timestamp types and cast results. Consistent with other rows in the same generated file.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:69:1likely benign A wide row in a generated PyArrow cast golden-result table.Reasoning
Generated golden-result row for PyArrow casts. The length reflects table layout and the snippet shows no encoding, execution or network behaviour.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:70:1likely benign A wide row in a generated PyArrow cast golden-result table.Reasoning
Duration-type cast row in a generated markdown test fixture. Raw long-line warning only; the content is plain tabular test data.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:71:1likely benign A wide row in a generated PyArrow cast golden-result table.Reasoning
Generated golden table row for duration casts with negative and standard values. A raw long-line flag on a documentation-style fixture, with no malicious indicators.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:72:1likely benign A wide row in a generated PyArrow cast golden-result table.Reasoning
Duration cast row from the same generated PyArrow golden file. The long line is table width only, with no obfuscation or executable content.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:73:1likely benign A generated golden markdown table recording PyArrow scalar cast outcomes for duration values.Reasoning
Markdown golden-file table of PyArrow cast results with long rows; raw long-line signal on generated test data has no obfuscation or malicious content.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:74:1likely benign A generated golden markdown table row of PyArrow duration cast results.Reasoning
Same generated golden test table row structure; long line is just a wide markdown table of cast results.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:75:1likely benign A generated golden markdown table row of PyArrow duration cast results.Reasoning
Generated golden test data row with repetitive ERR and value entries; no executable or encoded payload.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:76:1likely benign A generated golden markdown table row of PyArrow duration cast results.Reasoning
Generated golden test table; the long line is an artifact of wide tabular output in markdown.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:77:1likely benign A generated golden markdown table row of PyArrow duration cast results.Reasoning
Generated golden test table with no sign of obfuscation; only a long-line length warning.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:78:1likely benign A generated golden markdown table row of PyArrow time32 cast results.Reasoning
Generated golden table of time32 cast outcomes; benign test fixture content.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:79:1likely benign A generated golden markdown table row of PyArrow time32 cast results.Reasoning
Generated golden test table row; long line only reflects wide tabular data.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:80:1likely benign A generated golden markdown table row of PyArrow time32 cast results.Reasoning
Generated golden test table row with no executable or encoded content; benign.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:81:1likely benign A generated golden markdown table row of PyArrow time32 cast results.Reasoning
Generated golden test table row; the warning is a length heuristic on benign fixture data.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:82:1likely benign A generated golden markdown table row of PyArrow time64 cast results.Reasoning
Generated golden test table row for time64 casts; no malicious indicators, only wide-line warning.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:83:1dismissed A generated golden-file table row of PyArrow scalar cast outcomes.Reasoning
The line is a wide markdown table row of PyArrow cast results inside a golden test fixture. Long lines are expected in generated tables, and the content is test data, not code.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:84:1dismissed A generated golden-file table row of PyArrow scalar cast outcomes.Reasoning
Same as the other golden table rows: a long markdown line in a generated PyArrow test fixture with no executable or obfuscated content. The raw long-line signal has no security meaning here.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.md:85:1dismissed A generated golden-file table row of PyArrow scalar cast outcomes.Reasoning
A long line in a generated markdown golden fixture of cast results. It is test data, and the raw long-line flag is a formatting artifact.
-
info
warn→ info by reviewlong-linepython/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_unsafe.md:1:1dismissed A generated golden-file table header for PyArrow unsafe cast results.Reasoning
A wide markdown table header in a golden test fixture for unsafe PyArrow casts. This is documentation/test data with no executable or hidden content.
-
echo "Combining collected coverage data under $COVERAGE_DIR/coverage_data" $COV_EXEC combine echo "Creating XML report file at python/coverage.xml" $COV_EXEC xml --ignore-errors --include "pyspark/*" -
echo "Creating XML report file at python/coverage.xml" $COV_EXEC xml --ignore-errors --include "pyspark/*" echo "Reporting the coverage data at $COVERAGE_DIR/coverage_data/coverage" $COV_EXEC report --include "pyspark/*" -
echo "Reporting the coverage data at $COVERAGE_DIR/coverage_data/coverage" $COV_EXEC report --include "pyspark/*" echo "Generating HTML files for PySpark coverage under $COVERAGE_DIR/htmlcov" $COV_EXEC html --ignore-errors --include "pyspark/*" --directory "$COVERAGE_DIR/htmlcov" -
echo "Generating HTML files for PySpark coverage under $COVERAGE_DIR/htmlcov" $COV_EXEC html --ignore-errors --include "pyspark/*" --directory "$COVERAGE_DIR/htmlcov" popd -
likely benign Test runner cleanup that shells out to rm -rf on a temporary directory on Darwin.
Reasoning
os.system is built from a string concatenation with tmp_dir, which is a non-literal shell command. However, it is a cleanup branch in a test runner that removes a temporary directory on macOS, and the input is locally created temp path data rather than attacker-controlled input. The injection risk is minor and the code looks like a legitimate, if sloppy, pattern.
-
exec "$@"dismissed Launcher script that execs the user-supplied command with viztracer.Reasoning
`exec "$@"` is the standard passthrough idiom in wrapper scripts: it replaces the shell with the command the user supplied. This is expected behavior for a launcher, not a dropper.
-
exec "$@"dismissed Debug launcher script that execs the user-supplied command.Reasoning
`exec "$@"` is a standard argument passthrough in a debugging wrapper script. It runs the caller's own arguments and contains no obfuscation or fetched payload.
Reproduce This Scan
Run disclude locally to verify. Add --format sarif for SARIF output. Add --llm (with an API key in the environment) to include the LLM review.
&&