Chunk size in python large size

def rows(f, chunksize=1024, sep='|'):
    """
    Read a file where the row separator is '|' lazily.

    Usage:

    >>> with open('big.csv') as f:
    >>>     for r in rows(f):
    >>>         process(r)
    """
    row = ''
    while (chunk := f.read(chunksize)) != '':   # End of file
        while (i := chunk.find(sep)) != -1:     # No separator found
            yield row + chunk[:i]
            chunk = chunk[i+1:]
            row = ''
        row += chunk
    yield row

Source: stackoverflow.com

Similar codes

1 vote263 views1 answer
Does jupyter notebook need internet
Python
0 vote252 views2 answers
Disable network on pytest
Python
22 votes271 views5 answers
Python import file from different directory
Python
0 vote303 views1 answer
Delete migrations django and start over deployment heroku
Python Django
0 vote293 views1 answer
Threshold meaning in pandas dropna
Python
0 vote278 views2 answers
Return returns string with brackets python
Python
1 vote277 views1 answer
How to create text file with python and store a dictionary
Python
0 vote299 views1 answer
Django password field encryption
Python Django
111 votes259 views6 answers
Rename columns
Python
0 vote276 views1 answer
Nth term of gp in python when 2,3 terms given
Python