/Python/Remove duplicate rows in csv file python

Remove duplicate rows in csv file python

# credit to the Stack Overflow user in the source link

with open('file_with_duplicates.csv','r') as in_file, open('ouput.csv','w') as out_file:
  
    seen = set() # set for fast O(1) amortized lookup
    
    for line in in_file:
        if line in seen: 
          continue # skip duplicate

        seen.add(line)
        out_file.write(line)

Source: stackoverflow.com

Similar codes

1 vote254 views1 answer
Python close gile
Python
3 votes208 views2 answers
Pandas read excel with two headers
Python
0 vote229 views1 answer
App = Flask(_name_) NameError: name '_name_' is not defined
Python
4 votes223 views2 answers
How to get the most common number in python
Python
0 vote217 views1 answer
Ejercicios con def en python
Python
0 vote210 views1 answer
Load data(review path) python
Python
4 votes221 views2 answers
Plot distribution seaborn
Python
0 vote205 views1 answer
How to handle missing values in dataset
Python
0 vote218 views2 answers
E unable to locate package python-pip ubuntu 20.04
Python
8 votes221 views2 answers
Pandas filter rows by value in list
Python