Wordcount pyspark

text_file = sc.textFile("hdfs://...")
counts = text_file.flatMap(lambda line: line.split(" ")) \
             .map(lambda word: (word, 1)) \
             .reduceByKey(lambda a, b: a + b)
counts.saveAsTextFile("hdfs://...")

Source: spark.apache.org

Similar codes

-2 votes313 views1 answer
Ode meaning
Python
0 vote328 views1 answer
Pydub get audio length in seconds
Python
0 vote324 views1 answer
Python http request params
Python
6 votes326 views3 answers
Python moving average pandas
Python
1 vote382 views2 answers
Python docstring use
Python
0 vote290 views1 answer
How to get the name of an object in python
Python
7 votes308 views3 answers
Python encoding utf 8
Python
2 votes333 views1 answer
Python pandas replace nan with null
Python
4 votes271 views1 answer
Django listview
Python
2 votes332 views2 answers
List of seaborn color palette
Python