Pyspark mapreduce dataframe

df.rdd \
  .filter(lambda x: x[1] == "france") \ # only french stations
  .map(lambda x: (x[0], x[2])) \ # select station & temp
  .mapValues(lambda x: (x, 1)) \ # generate count
  .reduceByKey(lambda x, y: (x[0]+y[0], x[1]+y[1])) \ # calculate sum & count
  .mapValues(lambda x: x[0]/x[1]) \ # calculate average
  .sortBy(lambda x: x[1], ascending = False) \ # sort
  .take(100)

Source: stackoverflow.com

Similar codes

1 vote235 views1 answer
Adding new column to existing DataFrame in Pandas by assigning a list
Python
1 vote231 views2 answers
What is the ternary operator in python
Python
2 votes235 views1 answer
String function in class python
Python
20 votes224 views6 answers
Lowercase python
Python
0 vote250 views1 answer
Django listview
Python
0 vote229 views1 answer
Keras transfer learning
Python
0 vote228 views1 answer
Concatenate strings of numpy array python
Python
0 vote261 views1 answer
Candle stick with volume plotly
Python
0 vote262 views1 answer
Permission denied when converting dataframe to csv
Python
0 vote251 views1 answer
Using replace() method to remove newlines from a string
Python