Pyspark rdd method

>>> df.repartition(10).rdd.getNumPartitions()
10
>>> data = df.union(df).repartition("age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+
>>> data = data.repartition(7, "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  5|  Bob|
|  2|Alice|
|  5|  Bob|
+---+-----+
>>> data.rdd.getNumPartitions()
7
>>> data = data.repartition("name", "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+

Source: spark.apache.org

Similar codes

0 vote342 views1 answer
Pronic number program in python
Python
0 vote406 views1 answer
Flask remove file after send_file
Python Flask
86 votes323 views6 answers
Csv in python
Python
0 vote351 views1 answer
Been deprecated, please pass in a Service object
Python
3 votes329 views1 answer
Presentation in jupyter notebook
Python
0 vote396 views1 answer
How to insert value in admin panel in django
Python Django
0 vote292 views1 answer
Multiply every nth element
Python
2 votes326 views10 answers
@7chats
Python
0 vote290 views1 answer
Tuple and dictionary in python
Python
0 vote319 views1 answer
Python tkinter interface exoskeleton
Python