Pyspark rdd method

>>> df.repartition(10).rdd.getNumPartitions()
10
>>> data = df.union(df).repartition("age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+
>>> data = data.repartition(7, "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  5|  Bob|
|  2|Alice|
|  5|  Bob|
+---+-----+
>>> data.rdd.getNumPartitions()
7
>>> data = data.repartition("name", "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+

Source: spark.apache.org

Similar codes

3 votes309 views2 answers
Postgres Database Django
Python Django
2 votes288 views2 answers
Timestamp e datetime python
Python
60 votes288 views4 answers
Python choose random sample from list
Python
2 votes332 views2 answers
Range python start at 1
Python
0 vote306 views1 answer
Returns all column names as a list
Python
3 votes289 views2 answers
Indentation in python
Python
0 vote286 views1 answer
Python generate sine wave pyaudio
Python
2 votes289 views2 answers
Print progress without next line python
Python
0 vote310 views1 answer
Int var def __init__(self,var=10): Initialize.var=var def display(): print var
Python
0 vote329 views1 answer
Run sql query on pandas dataframe
Python