Pyspark mapreduce dataframe

df.rdd \
  .filter(lambda x: x[1] == "france") \ # only french stations
  .map(lambda x: (x[0], x[2])) \ # select station & temp
  .mapValues(lambda x: (x, 1)) \ # generate count
  .reduceByKey(lambda x, y: (x[0]+y[0], x[1]+y[1])) \ # calculate sum & count
  .mapValues(lambda x: x[0]/x[1]) \ # calculate average
  .sortBy(lambda x: x[1], ascending = False) \ # sort
  .take(100)

Source: stackoverflow.com

Similar codes

0 vote262 views1 answer
Young C so new(pro.cashmoneyap x nazz music) soundcloud
Python
2 votes267 views3 answers
How to get a number from a string in python
Python
1 vote265 views2 answers
Use Python to calculate (((1+2)*3)/4)^5
Python
0 vote269 views2 answers
How to convert string labels to numpy array
Python
0 vote251 views1 answer
How to get checkbutton from a list
Python
9 votes271 views6 answers
Elif python
Python
0 vote302 views1 answer
Pymupdf extract all text from pdf
Python
1 vote261 views1 answer
Python pandas dataframe column date to string
Python
5 votes243 views3 answers
How to add button in tkinter
Python
0 vote276 views1 answer
Django Signup urls.py
Python Django