Pyspark rdd method

>>> df.repartition(10).rdd.getNumPartitions()
10
>>> data = df.union(df).repartition("age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+
>>> data = data.repartition(7, "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  5|  Bob|
|  2|Alice|
|  5|  Bob|
+---+-----+
>>> data.rdd.getNumPartitions()
7
>>> data = data.repartition("name", "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+

Source: spark.apache.org

Similar codes

0 vote335 views1 answer
Python variable definieren
Python
11 votes315 views3 answers
Logistic regression algorithm in python
Python
3 votes327 views1 answer
Python list of random float numbers
Python
8 votes339 views2 answers
How to merge two variables to get an array in python
Python
8 votes318 views1 answer
Python program to remove duplicate characters of a given string.
Python
12 votes321 views2 answers
Convert torch to numpy
Python
1 vote348 views1 answer
Scatter plot of a dataframe in python
Python
0 vote303 views1 answer
How to change the size of datapoint in plot python
Python
1 vote306 views1 answer
Erreur install pyaudio
Python
5 votes321 views3 answers
Python import graphviz
Python