Re: lots of little mallocs or one big one? (somewhat naive)
"Dave Stallard" <stallard@nospam.net> wrote in message
news:0OCdndpK28xQXc7bnZ2dnUVZ_h-vnZ2d@comcast.com...
So, I'm looking at some code that does 770K malloc calls in a row, of
varying size, paired with corresponding freads from a binary file to
initialize. In total, about 58 MB of data is allocated an initialized.
Unsurprisingly, this takes a good bit of time.
Question: Wouldn't it be a lot more efficient to allocate this 58 MB in
one big gulp, then fread in the 58 MB of data from the file to initialize
it, rather than a zillion small calls?
Maybe. It would be faster to allocate 58MB in one call then 770k calls for
smaller chucnks, quite a bit faster (maybe even 770k times faster). Also,
you will need 770k places to hold the pointers to the allocated memory with
smaller calls, just one pointer with the bigger call.
However, alignment can be an issue. Consider you want to read, say, a
character and an integer. On your system (for example) char has sizeof 1,
integer has sizeof 4. So you may allocate 5 bytes. Then read the character
then the integer. However, misaligned integers on some systems are slower
(mine) than an aligned one, and on some systems will cause a program crash.
Typically, a 4 byte integer needs to be aligned on a 4 byte boundary. That
is, an address evenly divisible by 4. On some systems the general rule
*may* apply, that any given built in type needs to be aligned on it's size
(I doubt this is true for all/most systems).
So, code like this may fail (untested code, and I don't use fread so kinda
guessing at it's syntax):
unsigned char* buffer = malloc( sizeof( char ) + sizeof( int ) );
unsigned char* bufferpos = buffer;
fread( bufferpos, sizeof( char ), 1, myfile );
bufferpos += sizeof( char );
fread( bufferpos, sizeof( int ), 1, myfile );
Well, now our buffer contains (supposedly) a character in the first bye, an
integer in the next 4 bytes. Now you get the fun of pulling out the integer
without crashing your system. Which generally means you're going to have to
allocate an integer and move the bytes over one by one with some method to
be safe. If you had allocated an integer in the first place and read into
it, you wouldn't have to do this extra step.
So, in the long run, yes, the malloc would be faster, but you're going to go
through extra steps to move your data into variables you can use, which you
are going ot have to malloc anyway.
Also, can it ever happen that a malloc of N bytes would fail, yet an
equivalent number of small mallocs summing to N would succeed?
Depending on how effecient the OS's mallocing is, yes. With padding and
everything else being an issue.
What about the fread?
What about it?