Spark sql with JDBC
./sbin/start-master.sh
you will get master is running on a hostname and a port number. copy that and put in blow command
./bin/spark-shell --driver-memory 4G --master spark://master-host:7077 --executor-memory 8G --executor-cores 1 --num-executors 3
import java.util.Properties
val connectionProperties = new Properties()
connectionProperties.put("user", "actualUsername")
connectionProperties.put("password", "actualPassword")
val jdbcUrl = "jdbc:mysql://hostname/dbname"
val sqlquery = "(select * from t1 limit 10)tmp"
val df = spark.read.jdbc(url=jdbcUrl, table=sqlquery, properties=connectionProperties)
df.show
for ms sql: https://docs.microsoft.com/en-us/sql/connect/jdbc/building-the-connection-url?view=sql-server-2017
Comments
Post a Comment