Pandas read chunk of csv

import pandas as pd 
import pickle

in_path = "" #Path where the large file is
out_path = "" #Path to save the pickle files to
chunk_size = 400000 #size of chunks relies on your available memory
separator = "~"

reader = pd.read_csv(in_path,sep=separator,chunksize=chunk_size, 
                    low_memory=False)    


for i, chunk in enumerate(reader):
    out_file = out_path + "/data_{}.pkl".format(i+1)
    with open(out_file, "wb") as f:
        pickle.dump(chunk,f,pickle.HIGHEST_PROTOCOL)

Source: stackoverflow.com

Similar codes

4 votes332 views3 answers
How to change port in flask app
Python Flask
0 vote284 views1 answer
Python try script
Python
4 votes283 views3 answers
Read file into list python
Python
4 votes213 views1 answer
Datetime has no attribute now
Python
0 vote324 views1 answer
Set value through serializer django
Python Django
36 votes256 views6 answers
Remove letter from string python
Python
0 vote269 views1 answer
La commande pop python
Python
1 vote274 views1 answer
Each line in a text file into a list in Python
Python
0 vote298 views1 answer
Django populate form from database
Python Django
0 vote326 views1 answer
Df.fillna("tagline",inplace=True) in jupyter notebook
Python