Apache Spark 설치

1. Java, scala 설치

# sudo apt update
# sudo apt-get upgrade
sudo apt install default-jdk scala

2. Apache Spark 다운로드

wget https://downloads.apache.org/spark/spark-3.4.0/spark-3.4.0-bin-hadoop3-scala2.13.tgz

3. Spark /opt 경로에 풀기

sudo mkdir /opt/spark
sudo tar -xf spark*.tgz -C /opt/spark --strip-component 1
sudo chmod -R 777 /opt/spark

4. 시스템 경로에 Java, Spark 폴더 추가

echo "export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64" >> ~/.bashrc
echo "export PATH=$PATH:$JAVA_HOME/bin" >> ~/.bashrc
echo "export SPARK_HOME=/home/jooeun.kim/spark" >> ~/.bashrc
echo "export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin" >> ~/.bashrc
echo "export PYSPARK_PYTHON=/usr/bin/python3" >> ~/.bashrc

5. reload shell

source ~/.bashrc

6. linux에서 Apache Spark master 시작하기

/opt/spark/sbin> . start-master.sh (--port 7072 --webui-port 8082)
# port : 7077(master), 임의의 값 (worker)
# webui-port : 8080 (master), 8081 (worker)

7. 웹 인터페이스에서 Spark master 접속하기

http://127.0.0.1:8080

8. 워커 스크립트 실행

#start-worker.sh spark://hostname:port
start-worker.sh spark://ubuntu:7077
start-worker.sh -m 212M spark://ubuntu:7077

9. Spark shell 사용

spark-shell