Spark local mode

Environment

Ubuntu

• HP Z230
• 数量: 1
• Ubuntu: 16.04 / 18.04
• CPU: E3-1230_V3 ^ 1
• RAM: 16GB
• Disk: 250 GB (HDD)
• Network: 1Gbps

Download Spark package

wget http://apache.stu.edu.tw/spark/spark-2.4.7/spark-2.4.7-bin-hadoop2.7.tgz
tar -xvf spark-2.4.7-bin-hadoop2.7.tgz
mv spark-2.4.7-bin-hadoop2.7 spark

ref:http://spark.apache.org/downloads.html

Set Spark Path

vim ~/.bashrc

export SPARK_HOME=/home/ubuntu/spark
export PATH=$PATH:$SPARK_HOME/bin

source ~/.bashrc

Install Java

sudo apt install openjdk-8-jdk openjdk-8-jre
java -version

Set Java Path

vim ~/.bashrc

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export JRE_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre

source ~/.bashrc

Install Pyspark

sudo apt-get install python3
sudo apt-get install python3-pip

pip3 install pyspark

Set Python Path

vim ~/.bashrc

export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.7-src.zip:$PYTHONPATH
export PYSPARK_PYTHON=python3

source ~/.bashrc