8000 Bump Spark 4.1.3, Delta 4.4.0, Unity Catalog 0.6.0 and Spark Operator 2.5.2 by ignitz · Pull Request #6 · ignitz/apache-spark · GitHub
[go: up one dir, main page]

Skip to content

Bump Spark 4.1.3, Delta 4.4.0, Unity Catalog 0.6.0 and Spark Operator 2.5.2 - #6

Open
ignitz wants to merge 1 commit into
mainfrom
feature/uc-0.6.0-delta-4.4.0
Open

ignitz wants to merge 1 commit into
mainfrom
feature/uc-0.6.0-delta-4.4.0

Conversation

@ignitz
@ignitz ignitz commented Oct 5, 2026
Copy link
Copy Markdown
Owner

Summary

Upgrades the custom Spark image to Delta 4.4.0 + Unity Catalog 0.6.0 and removes the patched unitycatalog-client jar. Both releases ship the OAuth M2M camelCase fix (unitycatalog#1663, delta#7203), so OAuth against Databricks Unity Catalog works with stock jars.

Component Before After
Apache Spark 4.1.2 4.1.3
Delta Lake 4.3.0 4.4.0
Unity Catalog connector 0.5.1 + patched client 0.6.0
Spark Operator v2.5.0 v2.5.2

Changes

  • Versions: base/custom/operator Dockerfiles and README.
  • Patched jar removed: patched-jars/io.unitycatalog_unitycatalog-client-0.5.1.jar deleted.
  • resolve_jars.py: replaces mv ~/.ivy2.5.2/jars/*.
    • Previously, older copies of jars Spark already ships (e.g. jackson-databind 2.15 next to Spark's 2.21) ended up in /opt/spark/jars.
    • Now Spark's version wins, and commons-logging and RoaringBitmap shims are skipped because they duplicate Spark classes.
    • The Jackson modules pulled transitively are pinned to 2.21.2.
    • The report lands in /opt/spark/extra-jars.txt.
  • Operator Dockerfile: the Go builder runs on $BUILDPLATFORM and cross-compiles (GOARCH=$TARGETARCH, CGO_ENABLED=0). Under QEMU the Go runtime crashed with fatal error: lfstack.push.
  • docker-bake.hcl + make buildx-ecr: multi-arch (amd64/arm64) build that pushes apache-spark-custom and apache-spark-operator to ECR. The base image is only an intermediate bake target. ECR repos are IMMUTABLE, so bump SPARK_TAG / OPERATOR_TAG on every release.

Published images

  • 975049936326.dkr.ecr.eu-central-1.amazonaws.com/apache-spark/apache-spark-custom:4.1.3 (amd64 + arm64)
  • 975049936326.dkr.ecr.eu-central-1.amazonaws.com/apache-spark/apache-spark-operator:2.5.2 (amd64 + arm64)

Built with a two-node buildx builder: local arm64 plus a native amd64 node, to avoid QEMU. A JVM spark-submit under emulation hangs at exit.

Test plan

Validated on a local kind cluster (arm64):

  • No duplicated classes between the added jars and Spark's jars, except the pre-existing aws-msk-iam-auth-2.3.7-all.jar (see Notes).
  • Operator deployed from this repo's image via the same chart keys used in EKS (image.registry/repository/tag), chart 2.5.2. Reports Git Version: v2.5.2 on Spark 4.1.3.
  • PySpark smoke test submitted through that operator against Databricks Unity Catalog (sandbox.spark_k8s) with OAuth M2M and UC credential vending. All steps passed: list schemas/tables, read an existing table, CREATE managed table, INSERT, MERGE, read back, DESCRIBE HISTORY.
  • delta-spark 4.4.0 on pip; delta, confluent_kafka and marimo import.
  • amd64 variant from ECR: x86_64, Spark 4.1.3, same extra-jars.txt as the tested arm64 image. The smoke test itself was not run on amd64.

Notes

  • Catalog config:
    • The catalog uri must be the workspace host only; the client appends /api/2.1/unity-catalog.
    • The Spark catalog name must equal the UC catalog name (delta#7788).
    • The service principal needs the workspace-access entitlement and EXTERNAL USE SCHEMA.
  • aws-msk-iam-auth-2.3.7-all.jar: a fat jar bundling AWS SDK v2, Netty and httpclient classes that overlap with bundle-2.29.52 (S3A). It didn't affect the tests and already exists in 4.1.2. Consider the non-all artifact in a follow-up.
  • Rollout (not part of this PR):
    • Upgrade the spark-operator Helm release to chart 2.5.2 with image.tag=2.5.2.
    • Move SparkApplications from apache-spark-custom:4.1.2 to 4.1.3.

O título do commit fica com mais de 72 caracteres. Se preferir mais curto: ⬆️ Bump Spark 4.1.3, Delta 4.4.0, UC 0.6.0, Operator 2.5.2.

…tor 2.5.2

- Spark 4.1.2 -> 4.1.3, Delta Lake 4.3.0 -> 4.4.0, Unity Catalog 0.5.1 -> 0.6.0,
  Spark Operator v2.5.0 -> v2.5.2.
- Drop the patched unitycatalog-client jar: UC 0.6.0 and Delta 4.4.0 ship the
  OAuth camelCase fix (unitycatalog#1663, delta#7203).
- Add resolve_jars.py: only copy Ivy-resolved jars that Spark doesn't already
  ship, skip commons-logging and RoaringBitmap shims (class clashes), and pin
  jackson-datatype-jdk8 / jackson-dataformat-xml to Spark's Jackson (2.21.2).
  Report written to $SPARK_HOME/extra-jars.txt.
- Operator: build the Go stage on $BUILDPLATFORM and cross-compile with
  GOARCH=$TARGETARCH (Go crashes under QEMU with "fatal error: lfstack.push").
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0