Pyspark rdd method

>>> df.repartition(10).rdd.getNumPartitions()
10
>>> data = df.union(df).repartition("age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+
>>> data = data.repartition(7, "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  5|  Bob|
|  2|Alice|
|  5|  Bob|
+---+-----+
>>> data.rdd.getNumPartitions()
7
>>> data = data.repartition("name", "age")
>>> data.show()
+---+-----+
|age| name|
+---+-----+
|  5|  Bob|
|  5|  Bob|
|  2|Alice|
|  2|Alice|
+---+-----+

Source: spark.apache.org

Similar codes

1 vote332 views1 answer
How to custom page not found in django
Python
1 vote346 views2 answers
How to convert integer to binary string python
Python
0 vote341 views1 answer
Async, await, python
Python
3 votes317 views1 answer
Pandas plotly backend
Python
0 vote353 views1 answer
Spacy text annotation dict comprehension
Python
1 vote383 views1 answer
How to use query_params in get_object djangorestframework
Python Django
0 vote359 views2 answers
How can i aggregate without group by in pandas
Python
14 votes358 views2 answers
Python get username
Python
3 votes373 views2 answers
Extract column numpy array python
Python
1 vote341 views1 answer
Python sorted dictionary multiple keys
Python