Pyspark rdd method

>>> df.repartition(10).rdd.getNumPartitions()
10
>>> data = df.union(df).repartition("age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+
>>> data = data.repartition(7, "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  5|  Bob|
|  2|Alice|
|  5|  Bob|
+---+-----+
>>> data.rdd.getNumPartitions()
7
>>> data = data.repartition("name", "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+

Source: spark.apache.org

Similar codes

0 vote188 views1 answer
Import pygame module in pycharm
Python
11 votes220 views3 answers
Python pip install pandas
Python
1 vote158 views1 answer
Python find smallest value in 2d list
Python
2 votes206 views10 answers
Lol
Python
43 votes257 views6 answers
Reverse string python
Python
26 votes190 views2 answers
Import time in python
Python
0 vote196 views1 answer
Onehot encode list of columns pandas
Python
18 votes179 views6 answers
Add to a list python
Python
0 vote204 views1 answer
Filter startswith django
Python Django
12 votes199 views1 answer
Python check if has attribute
Python