Pandas read chunk of csv

import pandas as pd 
import pickle

in_path = "" #Path where the large file is
out_path = "" #Path to save the pickle files to
chunk_size = 400000 #size of chunks relies on your available memory
separator = "~"

reader = pd.read_csv(in_path,sep=separator,chunksize=chunk_size, 
                    low_memory=False)    


for i, chunk in enumerate(reader):
    out_file = out_path + "/data_{}.pkl".format(i+1)
    with open(out_file, "wb") as f:
        pickle.dump(chunk,f,pickle.HIGHEST_PROTOCOL)

Source: stackoverflow.com

Similar codes

0 vote384 views1 answer
Django email change sender name
Python Django
0 vote368 views1 answer
Call a function onclick tkinter
Python
0 vote359 views1 answer
Deep clone 2d list python
Python
0 vote348 views1 answer
Permcheck codility python
Python
0 vote374 views1 answer
Textrank python implementation
Python
0 vote360 views1 answer
Pathlib remove extension
Python
0 vote392 views3 answers
From unix timestamp to datetime pandas
Python
0 vote336 views1 answer
Support vector machine svm using python numerical example
Python
0 vote381 views1 answer
Gcp functions save BQ
Python
9 votes362 views1 answer
Python delete none from list
Python