Pyspark rdd method

>>> df.repartition(10).rdd.getNumPartitions()
10
>>> data = df.union(df).repartition("age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+
>>> data = data.repartition(7, "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  5|  Bob|
|  2|Alice|
|  5|  Bob|
+---+-----+
>>> data.rdd.getNumPartitions()
7
>>> data = data.repartition("name", "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+

Source: spark.apache.org

Similar codes

29 votes176 views6 answers
Install jupyter notebook python
Python
0 vote174 views1 answer
Using csv module how to read perticular lines in csv
Python
25 votes190 views6 answers
Python if
Python
0 vote184 views2 answers
Matplotlib subtitle
Python
0 vote239 views1 answer
Matplotlib savefig size
Python
29 votes190 views6 answers
Update all pip packages
Python
-1 vote177 views1 answer
Analyse des fleurs du mal la vision du baudelaire
Python
2 votes180 views1 answer
Change image resolution pillow
Python
5 votes245 views3 answers
Flask port
Python Flask
0 vote169 views1 answer
Add a value to the start of a list python
Python