Re: Text File vs Database reading performance
On 14.08.2006 08:01, EJP wrote:
antoine wrote:
my problem is that I feel I'm reading the file "in pieces" and
alternating between reading & processing. I'm wandering if there's a
faster way, to read the file at once and store elements in a data
structure, then analyse each entry of that data structure...
any take on that ?
Yes. Don't. What's wrong with 'reading the file in pieces'? The 2nd
technique you propose doesn't scale to arbitrarily large files, and
gives you an I/O-bound first phase followed by a compute-bound 2nd
phase. The way you're doing it now is better from all points of view,
especially that of other users of the system.
A compromise approach would be to have a reader thread and a processor
thread connected by a bounded (!) queue. This gives you parallel
reading and processing or put differently, processing is not stopped by
slow reading. This is actually only worth the effort if processing is
slower than reading. If not, your processing thread is slowed down by
the reader. I doubt though that it's worth the effort - especially
since we haven't seen proof which part of the system is slow.
You could try a larger buffer size when constructing the BufferedReader,
say 64k, but in my experience the default (8k *chars* or 16kB) is pretty
well chosen. If you make this buffer size *really* large you will
effectively get something like your 2nd suggestion anyway.
Yep.
Kind regards
robert