Re: FFT: Problem with large dataset in memory
On 2008-09-24 21:28, Simon Lu wrote:
Hi,
I am working in a project dealing with a big many of data. Now I have
a problem with doing the FFT on a very long time trace, a signal with
over 300 million sampling points. After testing with my computer, I
realise that I can store only 2**27 points into memory, which will
need 2 GB RAM. With an array of 2**28 Double_t points the program
crash ("segmentation violation"). I tried the Root cern framework, gsl
and fftw3 library. They all need to load the data into memory. So the
question is: Are there some mechanisms or algorithms to manage the
array on a TTree or somewhere else on the hard disk? And then load the
the data step by step into cache? Something likes a FileArray. Or you
get a better idea?
This is really urgent. I am very grateful if I can hear something from
There are many ways to do what you want, and the best way depends on
your needs. If you need to operate on the whole dataset at once you have
somewhat of a problem, if you are able to work with just a few data-
points at a time you should be able to "stream" the data through your
algorithm and can probably get away with a fairly small memory footprint.
None of this is however directly topical in this group, is is on a to
high level and have no C++ specific parts, try asking in a general
programming group, such as comp.programming, instead.
If you are running on Windows you might also be able to increase the
virtual memory available for your application to 3GB, for more info:
http://www.microsoft.com/whdc/system/platform/server/PAE/PAEmem.mspx
--
Erik Wikstr??m
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]