Pyspark rdd method

>>> df.repartition(10).rdd.getNumPartitions()
10
>>> data = df.union(df).repartition("age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+
>>> data = data.repartition(7, "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  5|  Bob|
|  2|Alice|
|  5|  Bob|
+---+-----+
>>> data.rdd.getNumPartitions()
7
>>> data = data.repartition("name", "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+

Source: spark.apache.org

Similar codes

2 votes223 views1 answer
Lcm math python library
Python
0 vote212 views1 answer
The dropping of sediment by water wind and ice or gravity is known as
Python
1 vote216 views1 answer
How to fill na python
Python
10 votes223 views5 answers
Pandas convert float to int
Python
0 vote223 views1 answer
Python check if file is writable
Python
0 vote220 views1 answer
Pandas plot column titles vertical
Python
0 vote220 views1 answer
Tkinter yes no dialogue box
Python
0 vote220 views1 answer
Numpy combine two arrays selecting min
Python
33 votes231 views6 answers
Merge dataframe
Python
1 vote330 views1 answer
Add help text in Django model forms
Python Django