Re: Creating a byte[] of long size
On 09-07-2010 12:21, Wayne wrote:
On 7/9/2010 12:31 AM, Patricia Shanahan wrote:
Wayne wrote:
On 7/8/2010 5:35 PM, Boris Punk wrote:
Integer.MAX_VALUE = 2147483647
I might need more items than that. I probably won't, but it's nice to have
extensibility.
To me, it is unlikely your system will run well if this one data structure
consumes 2G of memory. (You didn't really state the application or system;
certainly there are exceptions to the rule.) I would suggest you use a
more flexible system, where you keep the data on storage (disk) and use
memory as a cache. Perhaps an ArrayList of soft references would work well.
It might even be possible in your particular case to run a daemon thread
that pre-fetches items into the cache.
What's the difference between one data structure occupying over 2 GB and a set of
data structures that use that much space?
Certainly, given enough memory, Java can support total data structure sizes well over
2 GB without excessive paging.
A reduction in the number of page faults. There was an interesting article about
this topic in this month's Communications of the ACM, by Poul-Jenning Kamp,
Poul-Henning Kamp often abbreviated PHK.
who
was one of the lead developers of the FreeBSD kernel.
He still contributes to FreeBSD.
He applied his insight
to a web proxy replacement for Squid called Varnish, and was able to replace
12 Squid machines with 3 Varnish ones. It used a modified binary heap he called
a B-heap, which respected the page size of memory. The article was titled
"You're doing It Wrong". The message I came away with was, don't ignore the
fact that computers use paging when designing large data structures. I was
thinking that lesson might apply to the OP's situation.
I assume you are talking about this article:
http://queue.acm.org/detail.cfm?id=1814327
He is not suggesting any custom swap to disk or anything, but just
noting that it is beneficial to keep stuff together to minimize
paging.
In what may actually be an earlier version of the same story:
http://www.version2.dk/artikel/13201-poul-henning-kamp-ungdommen-kan-ikke-programmere-til-et-moderne-os
translated:
http://translate.google.com/translate?js=y&prev=_t&hl=en&ie=UTF-8&layout=1&eotf=1&u=http%3A%2F%2Fwww.version2.dk%2Fartikel%2F13201-poul-henning-kamp-ungdommen-kan-ikke-programmere-til-et-moderne-os&sl=da&tl=en
he explicit states:
'A classic example is that you stand and move things between disk and
memory at all times. Men det g?r operativsystemkernen jo selv. But it
makes the operating system kernel yourself. Man skal bare l?gge det et
sted i den virtuelle hukommelse, s? g?r den det selv, og det er den
meget bedre til. You just put it somewhere in the virtual memory, so it
makes it even and it is much better. Men det er folk ikke klar over,?
lyder vurderingen fra Poul-Henning Kamp. But people are not aware,
"reads the assessment from Poul-Henning Kamp.
Arne