mac安装Pyspark并毗连Mysql

• 安装Scala, apache-spark, Hadoop
  

1. brew install scala
2. brew install apache-spark
3. brew install hadoop
4. pip install pyspark

复制代码

注意不要自己别的安装jdk, 会造成版本对不上报错。由于安装apache-spark的过程中会自动安装openjdk。

• 设置环境变量
  

1. JAVA_HOME=/opt/homebrew/Cellar/openjdk@11/11.0.26/libexec/openjdk.jdk/Contents/Home
2. export PATH=$PATH:$JAVA_HOME/bin
3. export CLASSPATH=$JAVA_HOME/lib
4. export JAVA_HOME
6. SCALA_HOME=/opt/homebrew/opt/scala
7. export PATH=$PATH:$SCALA_HOME/bin
9. export SPARK_PATH=/opt/homebrew/Cellar/apache-spark/3.5.4
10. export PATH=$PATH:$SPARK_PATH/bin
12. export openjdk_home=/opt/homebrew/Cellar/openjdk@17/17.0.14
14. HADOOP_HOME=/opt/homebrew/Cellar/hadoop/3.4.1
15. export PATH=$PATH:/$HADOOP_HOME/bin

复制代码

注意jdk版本不对会造成Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.: java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed

• 下载jdbc driver
  链接, 选择 Platform Independent
  

将下载的解压包中的jar文件拷贝到apache-spark的目次下: /opt/homebrew/Cellar/apache-spark/3.5.4/libexec/jars

• 使用如下代码来测试
  

1. from pyspark.sql import SparkSession
2. from pyspark import SparkContext
4. # sc = SparkContext.getOrCreate()
6. sparkdriver = SparkSession.builder.\
7.     appName('demo').\
8.     master('local[*]').\
9.     config('spark.driver.extraClassPath', '/opt/homebrew/Cellar/apache-spark/3.5.4/libexec/jars/mysql-connector-j-9.2.0.jar').\
10.     getOrCreate()
11.    
12. df_mysql = sparkdriver.read.format('jdbc').\
13.     option('url', 'jdbc:mysql://localhost:3306').\
14.     option('driver', 'com.mysql.jdbc.Driver').\
15.     option('user', 'root').\
16.     option('password', '123').\
17.     option('query', 'select * from tablename').\
18.     load()
20. df_mysql.show(10)

复制代码

  Connect MySQL with PySpark using JDBC Connector (Mac 2022)