/Python/Remove duplicate rows in csv file python

Remove duplicate rows in csv file python

# credit to the Stack Overflow user in the source link

with open('file_with_duplicates.csv','r') as in_file, open('ouput.csv','w') as out_file:
  
    seen = set() # set for fast O(1) amortized lookup
    
    for line in in_file:
        if line in seen: 
          continue # skip duplicate

        seen.add(line)
        out_file.write(line)

Source: stackoverflow.com

Similar codes

24 votes393 views6 answers
Static files in django
Python
0 vote342 views1 answer
Websocket api python on close
Python
0 vote363 views1 answer
Python scipy put more weight to a set value in curve_fit
Python
-2 votes390 views1 answer
Django modelform datetime
Python Django
0 vote374 views1 answer
Http response template
Python
0 vote372 views1 answer
Python convert dict to xml
Python
0 vote334 views1 answer
How to read linux environment variable in python
Python
0 vote365 views1 answer
Python manually trigger exception
Python
13 votes353 views6 answers
Range in python
Python
0 vote394 views1 answer
MULTIPLE Model Search in Django rest framework
Python