/Python/Remove duplicate rows in csv file python

Remove duplicate rows in csv file python

# credit to the Stack Overflow user in the source link

with open('file_with_duplicates.csv','r') as in_file, open('ouput.csv','w') as out_file:
  
    seen = set() # set for fast O(1) amortized lookup
    
    for line in in_file:
        if line in seen: 
          continue # skip duplicate

        seen.add(line)
        out_file.write(line)

Source: stackoverflow.com

Similar codes

-1 vote282 views2 answers
Python how to add 2 numbers
Python
31 votes330 views3 answers
Pandas order dataframe by column of other dataframe
Python
0 vote306 views1 answer
Mathtext to regular python
Python
1 vote314 views1 answer
Accept user input in python
Python
0 vote328 views1 answer
Python image to video
Python
3 votes296 views2 answers
How to change role permissions in discord.py
Python
10 votes319 views6 answers
Formatting in python
Python
0 vote334 views1 answer
Affinity propagation cosine similarity python
Python
0 vote312 views1 answer
Python capture desktop as video source
Python
0 vote299 views1 answer
How to alight and place ipywidgets
Python