Connect to spark cluster

from pyspark.conf import SparkConf
from pyspark.sql import SparkSession, Catalog

spark_conf = SparkConf()
spark_conf.setAll([
    ('spark.master', 'spark://<master-ip>:7077'),
    ('spark.app.name', 'myApp'),
    ('spark.submit.deployMode', 'client'),
    ('spark.ui.showConsoleProgress', 'true'),
    ('spark.eventLog.enabled', 'false'),
    ('spark.logConf', 'false'),
    ('spark.driver.bindAddress', '0.0.0.0'),
    ('spark.driver.host', '<master-ip>')
])

spark = SparkSession.builder.config(conf=spark_conf).getOrCreate()

Similar codes

0 vote266 views1 answer
Matplotlib matrix plot
Python
81 votes280 views6 answers
Python write to file
Python
3 votes268 views2 answers
How to edit a specific line in text file in python
Python
0 vote287 views1 answer
Doormat pattern
Python
123 votes287 views6 answers
Rename column in dataframe
Python
-3 votes285 views1 answer
Shooting game with pygame python tutorial
Python
0 vote295 views1 answer
Python iter on a dic key value
Python
-2 votes307 views5 answers
Wow
Python
30 votes289 views6 answers
Mount drive google colab
Python
2 votes258 views1 answer
Numpy array remove scientific notation
Python