Repository navigation
Conversation
…tor 2.5.2 - Spark 4.1.2 -> 4.1.3, Delta Lake 4.3.0 -> 4.4.0, Unity Catalog 0.5.1 -> 0.6.0, Spark Operator v2.5.0 -> v2.5.2. - Drop the patched unitycatalog-client jar: UC 0.6.0 and Delta 4.4.0 ship the OAuth camelCase fix (unitycatalog#1663, delta#7203). - Add resolve_jars.py: only copy Ivy-resolved jars that Spark doesn't already ship, skip commons-logging and RoaringBitmap shims (class clashes), and pin jackson-datatype-jdk8 / jackson-dataformat-xml to Spark's Jackson (2.21.2). Report written to $SPARK_HOME/extra-jars.txt. - Operator: build the Go stage on $BUILDPLATFORM and cross-compile with GOARCH=$TARGETARCH (Go crashes under QEMU with "fatal error: lfstack.push").
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Upgrades the custom Spark image to Delta 4.4.0 + Unity Catalog 0.6.0 and removes the patched
unitycatalog-clientjar. Both releases ship the OAuth M2M camelCase fix (unitycatalog#1663, delta#7203), so OAuth against Databricks Unity Catalog works with stock jars.Changes
patched-jars/io.unitycatalog_unitycatalog-client-0.5.1.jardeleted.resolve_jars.py: replacesmv ~/.ivy2.5.2/jars/*.jackson-databind2.15 next to Spark's 2.21) ended up in/opt/spark/jars.commons-loggingand RoaringBitmapshimsare skipped because they duplicate Spark classes./opt/spark/extra-jars.txt.$BUILDPLATFORMand cross-compiles (GOARCH=$TARGETARCH,CGO_ENABLED=0). Under QEMU the Go runtime crashed withfatal error: lfstack.push.docker-bake.hcl+make buildx-ecr: multi-arch (amd64/arm64) build that pushesapache-spark-customandapache-spark-operatorto ECR. The base image is only an intermediate bake target. ECR repos are IMMUTABLE, so bumpSPARK_TAG/OPERATOR_TAGon every release.Published images
975049936326.dkr.ecr.eu-central-1.amazonaws.com/apache-spark/apache-spark-custom:4.1.3(amd64 + arm64)975049936326.dkr.ecr.eu-central-1.amazonaws.com/apache-spark/apache-spark-operator:2.5.2(amd64 + arm64)Built with a two-node buildx builder: local arm64 plus a native amd64 node, to avoid QEMU. A JVM
spark-submitunder emulation hangs at exit.Test plan
Validated on a local kind cluster (arm64):
aws-msk-iam-auth-2.3.7-all.jar(see Notes).image.registry/repository/tag), chart 2.5.2. ReportsGit Version: v2.5.2on Spark 4.1.3.sandbox.spark_k8s) with OAuth M2M and UC credential vending. All steps passed: list schemas/tables, read an existing table, CREATE managed table, INSERT, MERGE, read back, DESCRIBE HISTORY.delta-spark4.4.0 on pip;delta,confluent_kafkaandmarimoimport.extra-jars.txtas the tested arm64 image. The smoke test itself was not run on amd64.Notes
urimust be the workspace host only; the client appends/api/2.1/unity-catalog.workspace-accessentitlement andEXTERNAL USE SCHEMA.aws-msk-iam-auth-2.3.7-all.jar: a fat jar bundling AWS SDK v2, Netty and httpclient classes that overlap withbundle-2.29.52(S3A). It didn't affect the tests and already exists in 4.1.2. Consider the non-allartifact in a follow-up.spark-operatorHelm release to chart 2.5.2 withimage.tag=2.5.2.apache-spark-custom:4.1.2to4.1.3.O título do commit fica com mais de 72 caracteres. Se preferir mais curto: ⬆️ Bump Spark 4.1.3, Delta 4.4.0, UC 0.6.0, Operator 2.5.2.