[Python]How do I read a huge file line by line in Python, without loading the entire

How do I read a huge file line by line in Python, without loading the entire thing into memory first?
http://www.yak.net/fqa/171.html
by rupe

How do I read a huge file line by line in Python, without loading the entire thing into memory first?

In Python, the most common way to read lines from a file is to do the following: 

for line in open('myfile','r').readlines():
do_something(line)


When this is done, however, the readlines() function loads the entire file into memory as it runs. A better approach for large files is to use the fileinput module, as follows:

import fileinput
for line in fileinput.input(['myfile']):
do_something(line)


the fileinput.input() call reads lines sequentially, but doesn't keep them in memory after they've been read.

 

---

Annotation by enki :

as 'file' is iteratable, why not simply iterate on it? (use iter() when you need more control over the iterator's state)

ex:

for line in open('myfile','r'):
  doSomething(line)