/Python/Remove duplicate rows in csv file python

Remove duplicate rows in csv file python

# credit to the Stack Overflow user in the source link

with open('file_with_duplicates.csv','r') as in_file, open('ouput.csv','w') as out_file:
  
    seen = set() # set for fast O(1) amortized lookup
    
    for line in in_file:
        if line in seen: 
          continue # skip duplicate

        seen.add(line)
        out_file.write(line)

Source: stackoverflow.com

Similar codes

1 vote362 views1 answer
Python convert html to text
Python
0 vote336 views1 answer
Pandas is nattype
Python
10 votes342 views6 answers
Swap case python
Python
0 vote296 views1 answer
Computercraft turtle place block
Python
4 votes297 views1 answer
Removing whitespaces from pandas dataframe csv
Python
0 vote369 views1 answer
Disable console messages in Flask server
Python Flask
15 votes343 views4 answers
Csv to numpy array
Python
0 vote287 views1 answer
Ignore error open file python
Python
0 vote317 views1 answer
Numpy convert true false to 0 1
Python
0 vote316 views1 answer
Parse list from string
Python