Re: Reading from very large file

From:

=?ISO-8859-1?Q?Arne_Vajh=F8j?= <arne@vajhoej.dk>

Newsgroups:

comp.lang.java.programmer

Date:

Sun, 09 May 2010 12:46:33 -0400

Message-ID:

<4be6e6e8$0$282$14726298@news.sunsite.dk>

On 09-05-2010 02:57, Hakan wrote:

Sorry about the mistake, but the file is actually 13 GB.

That will take some time to process.

I can read to a
character array buffering about 30 million characters before the heap
space is overflowed.

You can increase heap space using -Xmx1g or other size, but buffers
larger than 30 Mchar will not improve performance significantly
(assuming sequential processing).

The sscce site is down and not accessible when I tried. What I have been
doing so far is something like this in rough code:

static int nchars=27000000;
int startpos=0;
File readfile="../x.txt";
FileReader frd=new File;
String searchs="20020701";
char[] arr=new char[nchars];

while (more dates to search for)
{
frd=new FileReader(readfile); /*reopen file
frd.skip(startpos); /*move to file pointer where final place of last
date was found
frd.read(arr,0,nchars); /*10
find number of date occurrences in arr with pattern matching
update searchs (first time to "20020702" and so on
startpos=startpos+(last place of pattern match)
output result for this date
}

This in all tends to use one to two minutes per run of the loop. What I
would like to do is to a) either preprocess the file such that I get an
input file where only numbers are present or b) change the read call at
label 10 so that it only reads numbers instead of all next characters.

The above code is not precise enough to that we can see what
could be the bottleneck.

As someone else stated then the skip trick looks very suspiciously.

Arne