Re: Concurrent Write to a single file (in multi thread code)?

homicanada@gmail.com wrote:


I would not do anything like this on a "just in case" basis. Most jobs
that read a long file, process it, and write it to another file are
limited by disk read and write performance. However, you should put the
application together as simply as possible, and then measure it. If it
runs fast enough, do nothing. If it is too slow, your actions should
depend on whether the problem is I/O or CPU.

For I/O, which is more likely to be the limiting factor, I would try to
keep the reads and write sequential if at all possible, to minimize disk
head movement and take advantage of OS optimizations. Also, put the
input and output files on different disks. If the files are really huge,
consider disk striping.

Multi-threading might help in a couple of cases. One is if the reads are
necessarily random, so that the disk drive has idle time while waiting
for your program to issue the next read. However, if that is the problem
I would begin by looking at java.nio.

Another case for multi-threading is if the CPU time for processing the
file is longer than the I/O time for reading it, an unusual situation.
If so, multi-threading on suitable hardware would let you spread the
work out over multiple processors, or multiple hardware threads in one
processor.

Patricia