Pyspark name accumulator

# Named accumulators are not available in Python yet
# You can use broadcast variables to see you progress
# Somehow

# Get a spark context sc and read your data into and rdd (or DataFrame)
rows = sc.textFile("/path/to/data/sample_data.txt")
# broadcast an object defined like below
processor = spark.sparkContext.broadcast(Processor())
# Apply this lambda to your rdd
mapped = rows.map(lambda x: processor.value.process(x))
# Other transformations you need to do
sc.stop()

class Processor:
    def process(self, content):
        # either print some output to the screen
        print(content)
        
        # or write to a file
        with open('results.txt', 'a') as f:
            f.write(content + "\n")

        # or call an API to store and count the requests
        requests.get('https://some.external.api/' + content)

        return self.other_processing(content)

Source: stackoverflow.com

Similar codes

0 vote388 views1 answer
Django insert bulk data
Python Django
0 vote305 views1 answer
What data types store value by index in python
Python
57 votes316 views6 answers
Python read lines
Python
1 vote318 views4 answers
Check internet speed using python
Python
0 vote349 views1 answer
I=int(input("enter the number")); sum=0; pro=1; while(i>0): d=1%10; if (d%2==0): sum=sum+d; else: pro=pro*d; i=i//10; print("sum=",sum,"product=",pro);
Python
10 votes397 views3 answers
Django make migrations
Python Django
2 votes321 views2 answers
Merge two columns name in one header pandas
Python
0 vote312 views1 answer
Encoding int to chr in python and vice versa
Python
0 vote384 views1 answer
Program for length of the shortest word
Python
0 vote360 views1 answer
Pytesseract.image_to_data into pandas dataframe
Python