Chunk size in python large size

def rows(f, chunksize=1024, sep='|'):
    """
    Read a file where the row separator is '|' lazily.

    Usage:

    >>> with open('big.csv') as f:
    >>>     for r in rows(f):
    >>>         process(r)
    """
    row = ''
    while (chunk := f.read(chunksize)) != '':   # End of file
        while (i := chunk.find(sep)) != -1:     # No separator found
            yield row + chunk[:i]
            chunk = chunk[i+1:]
            row = ''
        row += chunk
    yield row

Source: stackoverflow.com

Similar codes

0 vote188 views1 answer
Python list of dictionaries to list
Python
1 vote188 views1 answer
Django datetimefield default
Python
3 votes180 views1 answer
Spark df shape
Python
11 votes194 views2 answers
That does 1:to:3 do Python ?
Python
0 vote233 views1 answer
Minimal requirements.txt django
Python Django
0 vote179 views2 answers
Python test if you can convert to int
Python
4 votes197 views6 answers
How to change python version
Python
2 votes193 views1 answer
How to rotate screen with python
Python
0 vote253 views1 answer
How to calculate log 10 in python
Python
1 vote205 views1 answer
Multiprocessing a for loop python
Python