Pyspark rdd method

>>> df.repartition(10).rdd.getNumPartitions()
10
>>> data = df.union(df).repartition("age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+
>>> data = data.repartition(7, "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  5|  Bob|
|  2|Alice|
|  5|  Bob|
+---+-----+
>>> data.rdd.getNumPartitions()
7
>>> data = data.repartition("name", "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+

Source: spark.apache.org

Similar codes

5 votes233 views4 answers
Data normalization python
Python
0 vote220 views1 answer
Python import list from py file
Python
0 vote256 views1 answer
Python inspect.getsource
Python
0 vote324 views1 answer
Cv2.imshow not working in vscode
Python
1 vote217 views1 answer
Python letter arr
Python
0 vote248 views1 answer
Django filter word count greater than
Python Django
0 vote221 views1 answer
Replace transparent pixels python
Python
0 vote254 views1 answer
Ignore transformers warning
Python
0 vote228 views1 answer
How to take out every even number from a list in python
Python
0 vote230 views1 answer
Fb account api grabber
Python