Python site-packages pyspark

import os

def myfun(x):`
        os.system("pip install shapely")
        return x
rdd = sc.parallelize([1,2,3,4]) ## assuming 4 worker nodes
rdd.map(lambda x: myfun(x)).collect() 
## call each cluster to run the code to import the library

Source: stackoverflow.com

Similar codes

2 votes198 views1 answer
Set index in datarame
Python
0 vote196 views1 answer
Read data from excel file using openbyxl
Python
0 vote183 views1 answer
Ip address finder script python
Python
54 votes258 views6 answers
Conda create environment python 3.6
Python
0 vote186 views1 answer
\n appears in json dump
Python
0 vote213 views1 answer
Mypy clear cache
Python
0 vote178 views1 answer
Minimumm sort tryed python
Python
107 votes183 views6 answers
Python rename columns
Python
0 vote198 views1 answer
Echo $pythonpath ubuntu set default
Python
1 vote238 views10 answers
Python to java converter online
Python