Pyspark name accumulator

# Named accumulators are not available in Python yet
# You can use broadcast variables to see you progress
# Somehow

# Get a spark context sc and read your data into and rdd (or DataFrame)
rows = sc.textFile("/path/to/data/sample_data.txt")
# broadcast an object defined like below
processor = spark.sparkContext.broadcast(Processor())
# Apply this lambda to your rdd
mapped = rows.map(lambda x: processor.value.process(x))
# Other transformations you need to do
sc.stop()

class Processor:
    def process(self, content):
        # either print some output to the screen
        print(content)
        
        # or write to a file
        with open('results.txt', 'a') as f:
            f.write(content + "\n")

        # or call an API to store and count the requests
        requests.get('https://some.external.api/' + content)

        return self.other_processing(content)

Source: stackoverflow.com

Similar codes

0 vote394 views1 answer
Tuto date and time python
Python
6 votes372 views1 answer
Np euclidean distance python
Python
0 vote391 views1 answer
Python create empty dictionary with keys
Python
0 vote337 views1 answer
Django date grater
Python
0 vote368 views2 answers
Logout url
Python
1 vote381 views2 answers
Group by dateime pandas
Python
0 vote341 views1 answer
Get midnight of current day python
Python
0 vote372 views1 answer
Python float till 2 decimal places
Python
0 vote363 views1 answer
Replace string using regex
Python
0 vote427 views2 answers
Error: Could not locate a Flask application. You did not provide the "FLASK_APP" environment variable, and a "wsgi.py" or "app.py" module was not found in the current directory.
Python Flask