Spark aus den Sourcen bauen

Voraussetzung

• Python 3.8 (Die Python-Vesion muss mit der Version im Pyspark-Image übereinstimmen)

Bauen von Spark aus den Sourcen

Erzeugen der Spark "Build"-Umgebung

mkdir ~/spark-build
cd ~/spark-build
tar -x --strip-components 1 -f ../Downloads/spark-3.5.1.tar
./build/mvn -Pkubernetes -DskipTests clean package

Ausführlichere Informationen finden sich unter https://spark.apache.org/docs/latest/building-spark.html#building-with-kubernetes-support

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 3.5.1:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [  5.932 s]
[INFO] Spark Project Tags ................................. SUCCESS [  7.636 s]
[INFO] Spark Project Sketch ............................... SUCCESS [  9.517 s]
[INFO] Spark Project Local DB ............................. SUCCESS [  9.783 s]
[INFO] Spark Project Common Utils ......................... SUCCESS [ 10.929 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 16.737 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 12.266 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 12.987 s]
[INFO] Spark Project Launcher ............................. SUCCESS [  8.682 s]
[INFO] Spark Project Core ................................. SUCCESS [02:55 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 35.005 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 42.495 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 59.954 s]
[INFO] Spark Project SQL API .............................. SUCCESS [ 31.148 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [02:32 min]
[INFO] Spark Project SQL .................................. SUCCESS [03:45 min]
[INFO] Spark Project ML Library ........................... SUCCESS [02:26 min]
[INFO] Spark Project Tools ................................ SUCCESS [  6.398 s]
[INFO] Spark Project Hive ................................. SUCCESS [01:18 min]
[INFO] Spark Project REPL ................................. SUCCESS [ 21.673 s]
[INFO] Spark Project Kubernetes ........................... SUCCESS [ 55.766 s]
[INFO] Spark Project Assembly ............................. SUCCESS [  9.140 s]
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [ 16.150 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 23.493 s]
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [ 37.577 s]
[INFO] Spark Project Examples ............................. SUCCESS [ 52.559 s]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [  6.266 s]
[INFO] Spark Avro ......................................... SUCCESS [ 35.577 s]
[INFO] Spark Project Connect Common ....................... SUCCESS [ 53.639 s]
[INFO] Spark Protobuf ..................................... SUCCESS [ 37.673 s]
[INFO] Spark Project Connect Server ....................... SUCCESS [ 51.755 s]
[INFO] Spark Project Connect Client ....................... SUCCESS [ 51.584 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  25:01 min
[INFO] Finished at: 2024-03-31T16:09:20+02:00
[INFO] ------------------------------------------------------------------------

Test der Spark-Installation

Spark-Runtime inkl. Test

cd ~/spark-built
export SPARK_HOME=~/spark-built/
export PYSPARK_PYTHON=/usr/local/bin/python3.8
export PYSPARK_DRIVER_PYTHON=/usr/local/bin/python3.8

# Test
./bin/spark-submit examples/src/main/python/pi.py 10

24/03/25 12:30:19 INFO SparkContext: Running Spark version 3.5.1
24/03/25 12:30:19 INFO SparkContext: OS info Mac OS X, 14.3.1, x86_64
24/03/25 12:30:19 INFO SparkContext: Java version 21.0.2
...
24/03/25 12:30:20 INFO SparkContext: Submitted application: PythonPi
...
Pi is roughly 3.140720
24/03/25 12:30:22 INFO SparkContext: SparkContext is stopping with exitCode 0.
...