Pyspark rdd method

>>> df.repartition(10).rdd.getNumPartitions()
10
>>> data = df.union(df).repartition("age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+
>>> data = data.repartition(7, "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  5|  Bob|
|  2|Alice|
|  5|  Bob|
+---+-----+
>>> data.rdd.getNumPartitions()
7
>>> data = data.repartition("name", "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+

Source: spark.apache.org

Similar codes

0 vote173 views1 answer
Printing a range of no one line in python
Python
0 vote193 views1 answer
From flask import Flask app = Flask(__name__) @app.route('/',methods = ('GET')) def home(): return ('Hello World!!1') if __name__== '_main_': app.run()
Python Flask
2 votes213 views2 answers
Fibbonacci python
Python
0 vote182 views1 answer
Linux site-packages location
Python
0 vote175 views1 answer
Python: subset top 5 values in a column
Python
1 vote166 views1 answer
Get page source code selenium python
Python
3 votes172 views2 answers
Pandas normalize columns
Python
3 votes182 views2 answers
How to delete a turtle in python
Python
0 vote171 views1 answer
Python time limit for input
Python
12 votes171 views1 answer
Link shortener
Python