Does malloc reserve more space while allocating memory?Memory usage doesn't decrease when free() usedWill malloc implementations return free-ed memory back to the system?How do I free memory in C?Improve INSERT-per-second performance of SQLite?Some memory seems to be left allocated after malloc() and free()Does dynamic memory allocation differ in C and C++ in popular implementations?Does Malloc only use the heap if requested memory space is large?Allocating more memory than there exists using mallocWhat does “Memory allocated at compile time” really mean?Why Linux Free command is not showing less free memory when I run a process which keeps on allocating memoryWhat are the contents of the memory just allocated by `malloc()`?Memory allocation and process memory useWhy is virtual memory allocated with malloc not released?
Calculating the number of days between 2 dates in Excel
Visiting the UK as unmarried couple
Resetting two CD4017 counters simultaneously, only one resets
Simple image editor tool to draw a simple box/rectangle in an existing image
Female=gender counterpart?
How can I successfully establish a nationwide combat training program for a large country?
What should I use for Mishna study?
Blender - show edges angles “direction”
Is there a problem with hiding "forgot password" until it's needed?
Why are all the doors on Ferenginar (the Ferengi home world) far shorter than the average Ferengi?
What was required to accept "troll"?
Freedom of speech and where it applies
Hostile work environment after whistle-blowing on coworker and our boss. What do I do?
Is there an Impartial Brexit Deal comparison site?
Identify a stage play about a VR experience in which participants are encouraged to simulate performing horrific activities
Is a naturally all "male" species possible?
Is exact Kanji stroke length important?
How will losing mobility of one hand affect my career as a programmer?
Meta programming: Declare a new struct on the fly
Who must act to prevent Brexit on March 29th?
How do ultrasonic sensors differentiate between transmitted and received signals?
Proof of Lemma: Every integer can be written as a product of primes
Is it okay / does it make sense for another player to join a running game of Munchkin?
What will be the benefits of Brexit?
Does malloc reserve more space while allocating memory?
Memory usage doesn't decrease when free() usedWill malloc implementations return free-ed memory back to the system?How do I free memory in C?Improve INSERT-per-second performance of SQLite?Some memory seems to be left allocated after malloc() and free()Does dynamic memory allocation differ in C and C++ in popular implementations?Does Malloc only use the heap if requested memory space is large?Allocating more memory than there exists using mallocWhat does “Memory allocated at compile time” really mean?Why Linux Free command is not showing less free memory when I run a process which keeps on allocating memoryWhat are the contents of the memory just allocated by `malloc()`?Memory allocation and process memory useWhy is virtual memory allocated with malloc not released?
I am observing the following behavior in my test program:
I am doing malloc()
for 1 MB and then free()
it after sleep(10)
. I am doing this five times. I am observing memory consumption in top
while the program is running.
Once free()
-d, I am expecting the program's virtual memory (VIRT) consumption to be down by 1 MB. But actually it isn't. It stays stable. What is the explanation for this behavior? Does malloc()
do some reserve while allocating memory?
c malloc free dynamic-memory-allocation
add a comment |
I am observing the following behavior in my test program:
I am doing malloc()
for 1 MB and then free()
it after sleep(10)
. I am doing this five times. I am observing memory consumption in top
while the program is running.
Once free()
-d, I am expecting the program's virtual memory (VIRT) consumption to be down by 1 MB. But actually it isn't. It stays stable. What is the explanation for this behavior? Does malloc()
do some reserve while allocating memory?
c malloc free dynamic-memory-allocation
Related: How do I free memory in C?
– Lundin
Mar 22 at 13:51
2
Possible duplicate of Memory usage doesn't decrease when free() used
– Useless
Mar 22 at 16:29
2
@Useless This question has better answers than the older one so I've bucked convention and marked the old question a duplicate of this one.
– John Kugelman
Mar 22 at 18:32
I think nearly all malloc/free implementations use some internal management which does request larger chunks and free them opportunistically. This may usebrk(2)
or mmap. It also means that pages might not actually get used before touched (and sometimes even uncommitted on free, so the virtual or data segment size is not so important)
– eckes
Mar 22 at 19:42
add a comment |
I am observing the following behavior in my test program:
I am doing malloc()
for 1 MB and then free()
it after sleep(10)
. I am doing this five times. I am observing memory consumption in top
while the program is running.
Once free()
-d, I am expecting the program's virtual memory (VIRT) consumption to be down by 1 MB. But actually it isn't. It stays stable. What is the explanation for this behavior? Does malloc()
do some reserve while allocating memory?
c malloc free dynamic-memory-allocation
I am observing the following behavior in my test program:
I am doing malloc()
for 1 MB and then free()
it after sleep(10)
. I am doing this five times. I am observing memory consumption in top
while the program is running.
Once free()
-d, I am expecting the program's virtual memory (VIRT) consumption to be down by 1 MB. But actually it isn't. It stays stable. What is the explanation for this behavior? Does malloc()
do some reserve while allocating memory?
c malloc free dynamic-memory-allocation
c malloc free dynamic-memory-allocation
edited Mar 22 at 21:25
Peter Mortensen
13.8k1987113
13.8k1987113
asked Mar 22 at 7:45
user1228352user1228352
14011
14011
Related: How do I free memory in C?
– Lundin
Mar 22 at 13:51
2
Possible duplicate of Memory usage doesn't decrease when free() used
– Useless
Mar 22 at 16:29
2
@Useless This question has better answers than the older one so I've bucked convention and marked the old question a duplicate of this one.
– John Kugelman
Mar 22 at 18:32
I think nearly all malloc/free implementations use some internal management which does request larger chunks and free them opportunistically. This may usebrk(2)
or mmap. It also means that pages might not actually get used before touched (and sometimes even uncommitted on free, so the virtual or data segment size is not so important)
– eckes
Mar 22 at 19:42
add a comment |
Related: How do I free memory in C?
– Lundin
Mar 22 at 13:51
2
Possible duplicate of Memory usage doesn't decrease when free() used
– Useless
Mar 22 at 16:29
2
@Useless This question has better answers than the older one so I've bucked convention and marked the old question a duplicate of this one.
– John Kugelman
Mar 22 at 18:32
I think nearly all malloc/free implementations use some internal management which does request larger chunks and free them opportunistically. This may usebrk(2)
or mmap. It also means that pages might not actually get used before touched (and sometimes even uncommitted on free, so the virtual or data segment size is not so important)
– eckes
Mar 22 at 19:42
Related: How do I free memory in C?
– Lundin
Mar 22 at 13:51
Related: How do I free memory in C?
– Lundin
Mar 22 at 13:51
2
2
Possible duplicate of Memory usage doesn't decrease when free() used
– Useless
Mar 22 at 16:29
Possible duplicate of Memory usage doesn't decrease when free() used
– Useless
Mar 22 at 16:29
2
2
@Useless This question has better answers than the older one so I've bucked convention and marked the old question a duplicate of this one.
– John Kugelman
Mar 22 at 18:32
@Useless This question has better answers than the older one so I've bucked convention and marked the old question a duplicate of this one.
– John Kugelman
Mar 22 at 18:32
I think nearly all malloc/free implementations use some internal management which does request larger chunks and free them opportunistically. This may use
brk(2)
or mmap. It also means that pages might not actually get used before touched (and sometimes even uncommitted on free, so the virtual or data segment size is not so important)– eckes
Mar 22 at 19:42
I think nearly all malloc/free implementations use some internal management which does request larger chunks and free them opportunistically. This may use
brk(2)
or mmap. It also means that pages might not actually get used before touched (and sometimes even uncommitted on free, so the virtual or data segment size is not so important)– eckes
Mar 22 at 19:42
add a comment |
3 Answers
3
active
oldest
votes
Once
free()
-d, I am expecting program's virtual memory (VIRT) consumption to be down by 1MB.
Well, this is not guaranteed by the C standard. It only says, once you free()
the memory, you should not be accessing that any more.
Whether the memory block is actually returned to the available memory pool or kept aside for future allocations is decided by the memory manager.
1
Is it possible to release the free()'d memory block back to OS?
– user1228352
Mar 22 at 7:59
7
@user1228352 no, the C language doesn't allow this. If you want more control, you need to implement your own memory manager that relies on platform specific OS system calls.
– Jabberwocky
Mar 22 at 8:30
8
@user1228352 I understand the feeling after this, let's say trickery, however - you really don't want to go that way, nor it makes sense in the long-term approach because it's just a waste of time for you to figure out how to make your own memory manager (if allowed by the OS) and debug it. Go by the C standard and you'll have more comfortable experience, while the OS does the thing it's made for. Well, unless your target is to make your own OS, but then you probably wouldn't ask this question.
– KeyWeeUsr
Mar 22 at 14:15
@user1228352 Why would you want to? Virtual memory is effectively free.
– David Schwartz
Mar 22 at 23:27
2
Why would you want to reduce the unnecessary consumption of something that is not scarce? You should tell us a lot more about your environment if you want a helpful answer. Some unusual environments also have unusual implementations ofmalloc
andfree
. If you have a real issue (and this isn't just cosmetic) you could replace the allocator with one that never holds any extra virtual memory but there's about a 99% chance it will just make things worse due to issues like fragmentation.
– David Schwartz
2 days ago
|
show 2 more comments
The C standard doesn't force on the implementer of malloc
and free
to return the memory to the OS directly. So different C library implementations will behave differently. Some of them might give it back directly and some might not. In fact, the same implementation will also behave differently depending on the allocation sizes and patterns.
This behavior, of course, is for good reasons:
- It is not always possible. OS-level memory allocations usually are done in pages (4KB, 4MB, or ... sizes at once). And if a small part of the page is still being used after freeing another part then the page cannot be given back to the operating system until that part is also freed.
- Efficiency. It is very likely that an application will ask for memory again. So why give it back to the OS and ask for it again soon after. (of course, there is probably a limit on the size of the memory kept.)
In most cases, you are not accountable for the memory you free
if the implementation decided to keep it (assuming it is a good implementation). Sooner or later it will be reallocated or returned to the OS. Hence, optimizing for memory usage should be based on the amount you have malloc
-ed and you haven't free
-d. The case where you have to worry about this, is when your allocation patterns/sizes start causing memory fragmentation which is a very big topic on its own.
If you are, however, on an embedded system and the amount of memory available is limited and you need more control over when/how memory is allocated and freed then you need to ask for memory pages from the OS directly and manage it manually.
Edit: I did not explain why you are not accountable for memory you free.
The reason is, on a modern OS, allocated memory is virtual. Meaning if you allocate 512MB on 32-bit system or 10TB of 64-bit system, as long as you don't read or write to that memory, it will not reserve any physical space for it. Actually, it will only reserve physical memory for the pages you touch from that big block and not the entire block. And after "a while of not using that memory", its contents will be copied to disk and the underlying physical memory will be used for something else.
1
Note that some allocators may avoid the possibility of copying data to disk by using OS specific calls that say "these pages aren't in use, so feel free to drop their contents, even though I'm not releasing the virtual memory itself". Example would be using themadvise
call on Linux withMADV_DONTNEED
.
– ShadowRanger
2 days ago
@ShadowRanger very interesting to know! thank you.
– Ameen
2 days ago
add a comment |
This is very dependent on the actual malloc implementation in use.
Under Linux, there is a threshold (MMAP_THRESHOLD
) to decide where the memory for a given malloc()
request comes from.
If the requested amount is below or equal to MMAP_THRESHOLD
, the request is satisfied by either taking it from the so-called "free list", if any memory blocks have already been free()
d. Otherwise, the "break line" of the program (i. e. the end of the data segment) is increased and the memory made available to the program by this process is used for the request.
On free()
, the freed memory block is added to the free list. If there is enough free memory at the very end of the data segment, the break line (mentionned above) is moved again to shrink the data segment, returning the excess memory to the OS.
If the requested amount exceeds MMAP_THRESHOLD
, a separate memory block is requested by the OS and returned again during free()
.
See also https://linux.die.net/man/3/malloc for details.
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55294985%2fdoes-malloc-reserve-more-space-while-allocating-memory%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
Once
free()
-d, I am expecting program's virtual memory (VIRT) consumption to be down by 1MB.
Well, this is not guaranteed by the C standard. It only says, once you free()
the memory, you should not be accessing that any more.
Whether the memory block is actually returned to the available memory pool or kept aside for future allocations is decided by the memory manager.
1
Is it possible to release the free()'d memory block back to OS?
– user1228352
Mar 22 at 7:59
7
@user1228352 no, the C language doesn't allow this. If you want more control, you need to implement your own memory manager that relies on platform specific OS system calls.
– Jabberwocky
Mar 22 at 8:30
8
@user1228352 I understand the feeling after this, let's say trickery, however - you really don't want to go that way, nor it makes sense in the long-term approach because it's just a waste of time for you to figure out how to make your own memory manager (if allowed by the OS) and debug it. Go by the C standard and you'll have more comfortable experience, while the OS does the thing it's made for. Well, unless your target is to make your own OS, but then you probably wouldn't ask this question.
– KeyWeeUsr
Mar 22 at 14:15
@user1228352 Why would you want to? Virtual memory is effectively free.
– David Schwartz
Mar 22 at 23:27
2
Why would you want to reduce the unnecessary consumption of something that is not scarce? You should tell us a lot more about your environment if you want a helpful answer. Some unusual environments also have unusual implementations ofmalloc
andfree
. If you have a real issue (and this isn't just cosmetic) you could replace the allocator with one that never holds any extra virtual memory but there's about a 99% chance it will just make things worse due to issues like fragmentation.
– David Schwartz
2 days ago
|
show 2 more comments
Once
free()
-d, I am expecting program's virtual memory (VIRT) consumption to be down by 1MB.
Well, this is not guaranteed by the C standard. It only says, once you free()
the memory, you should not be accessing that any more.
Whether the memory block is actually returned to the available memory pool or kept aside for future allocations is decided by the memory manager.
1
Is it possible to release the free()'d memory block back to OS?
– user1228352
Mar 22 at 7:59
7
@user1228352 no, the C language doesn't allow this. If you want more control, you need to implement your own memory manager that relies on platform specific OS system calls.
– Jabberwocky
Mar 22 at 8:30
8
@user1228352 I understand the feeling after this, let's say trickery, however - you really don't want to go that way, nor it makes sense in the long-term approach because it's just a waste of time for you to figure out how to make your own memory manager (if allowed by the OS) and debug it. Go by the C standard and you'll have more comfortable experience, while the OS does the thing it's made for. Well, unless your target is to make your own OS, but then you probably wouldn't ask this question.
– KeyWeeUsr
Mar 22 at 14:15
@user1228352 Why would you want to? Virtual memory is effectively free.
– David Schwartz
Mar 22 at 23:27
2
Why would you want to reduce the unnecessary consumption of something that is not scarce? You should tell us a lot more about your environment if you want a helpful answer. Some unusual environments also have unusual implementations ofmalloc
andfree
. If you have a real issue (and this isn't just cosmetic) you could replace the allocator with one that never holds any extra virtual memory but there's about a 99% chance it will just make things worse due to issues like fragmentation.
– David Schwartz
2 days ago
|
show 2 more comments
Once
free()
-d, I am expecting program's virtual memory (VIRT) consumption to be down by 1MB.
Well, this is not guaranteed by the C standard. It only says, once you free()
the memory, you should not be accessing that any more.
Whether the memory block is actually returned to the available memory pool or kept aside for future allocations is decided by the memory manager.
Once
free()
-d, I am expecting program's virtual memory (VIRT) consumption to be down by 1MB.
Well, this is not guaranteed by the C standard. It only says, once you free()
the memory, you should not be accessing that any more.
Whether the memory block is actually returned to the available memory pool or kept aside for future allocations is decided by the memory manager.
edited 2 days ago
Brian McCutchon
4,77722136
4,77722136
answered Mar 22 at 7:48
Sourav GhoshSourav Ghosh
111k15132191
111k15132191
1
Is it possible to release the free()'d memory block back to OS?
– user1228352
Mar 22 at 7:59
7
@user1228352 no, the C language doesn't allow this. If you want more control, you need to implement your own memory manager that relies on platform specific OS system calls.
– Jabberwocky
Mar 22 at 8:30
8
@user1228352 I understand the feeling after this, let's say trickery, however - you really don't want to go that way, nor it makes sense in the long-term approach because it's just a waste of time for you to figure out how to make your own memory manager (if allowed by the OS) and debug it. Go by the C standard and you'll have more comfortable experience, while the OS does the thing it's made for. Well, unless your target is to make your own OS, but then you probably wouldn't ask this question.
– KeyWeeUsr
Mar 22 at 14:15
@user1228352 Why would you want to? Virtual memory is effectively free.
– David Schwartz
Mar 22 at 23:27
2
Why would you want to reduce the unnecessary consumption of something that is not scarce? You should tell us a lot more about your environment if you want a helpful answer. Some unusual environments also have unusual implementations ofmalloc
andfree
. If you have a real issue (and this isn't just cosmetic) you could replace the allocator with one that never holds any extra virtual memory but there's about a 99% chance it will just make things worse due to issues like fragmentation.
– David Schwartz
2 days ago
|
show 2 more comments
1
Is it possible to release the free()'d memory block back to OS?
– user1228352
Mar 22 at 7:59
7
@user1228352 no, the C language doesn't allow this. If you want more control, you need to implement your own memory manager that relies on platform specific OS system calls.
– Jabberwocky
Mar 22 at 8:30
8
@user1228352 I understand the feeling after this, let's say trickery, however - you really don't want to go that way, nor it makes sense in the long-term approach because it's just a waste of time for you to figure out how to make your own memory manager (if allowed by the OS) and debug it. Go by the C standard and you'll have more comfortable experience, while the OS does the thing it's made for. Well, unless your target is to make your own OS, but then you probably wouldn't ask this question.
– KeyWeeUsr
Mar 22 at 14:15
@user1228352 Why would you want to? Virtual memory is effectively free.
– David Schwartz
Mar 22 at 23:27
2
Why would you want to reduce the unnecessary consumption of something that is not scarce? You should tell us a lot more about your environment if you want a helpful answer. Some unusual environments also have unusual implementations ofmalloc
andfree
. If you have a real issue (and this isn't just cosmetic) you could replace the allocator with one that never holds any extra virtual memory but there's about a 99% chance it will just make things worse due to issues like fragmentation.
– David Schwartz
2 days ago
1
1
Is it possible to release the free()'d memory block back to OS?
– user1228352
Mar 22 at 7:59
Is it possible to release the free()'d memory block back to OS?
– user1228352
Mar 22 at 7:59
7
7
@user1228352 no, the C language doesn't allow this. If you want more control, you need to implement your own memory manager that relies on platform specific OS system calls.
– Jabberwocky
Mar 22 at 8:30
@user1228352 no, the C language doesn't allow this. If you want more control, you need to implement your own memory manager that relies on platform specific OS system calls.
– Jabberwocky
Mar 22 at 8:30
8
8
@user1228352 I understand the feeling after this, let's say trickery, however - you really don't want to go that way, nor it makes sense in the long-term approach because it's just a waste of time for you to figure out how to make your own memory manager (if allowed by the OS) and debug it. Go by the C standard and you'll have more comfortable experience, while the OS does the thing it's made for. Well, unless your target is to make your own OS, but then you probably wouldn't ask this question.
– KeyWeeUsr
Mar 22 at 14:15
@user1228352 I understand the feeling after this, let's say trickery, however - you really don't want to go that way, nor it makes sense in the long-term approach because it's just a waste of time for you to figure out how to make your own memory manager (if allowed by the OS) and debug it. Go by the C standard and you'll have more comfortable experience, while the OS does the thing it's made for. Well, unless your target is to make your own OS, but then you probably wouldn't ask this question.
– KeyWeeUsr
Mar 22 at 14:15
@user1228352 Why would you want to? Virtual memory is effectively free.
– David Schwartz
Mar 22 at 23:27
@user1228352 Why would you want to? Virtual memory is effectively free.
– David Schwartz
Mar 22 at 23:27
2
2
Why would you want to reduce the unnecessary consumption of something that is not scarce? You should tell us a lot more about your environment if you want a helpful answer. Some unusual environments also have unusual implementations of
malloc
and free
. If you have a real issue (and this isn't just cosmetic) you could replace the allocator with one that never holds any extra virtual memory but there's about a 99% chance it will just make things worse due to issues like fragmentation.– David Schwartz
2 days ago
Why would you want to reduce the unnecessary consumption of something that is not scarce? You should tell us a lot more about your environment if you want a helpful answer. Some unusual environments also have unusual implementations of
malloc
and free
. If you have a real issue (and this isn't just cosmetic) you could replace the allocator with one that never holds any extra virtual memory but there's about a 99% chance it will just make things worse due to issues like fragmentation.– David Schwartz
2 days ago
|
show 2 more comments
The C standard doesn't force on the implementer of malloc
and free
to return the memory to the OS directly. So different C library implementations will behave differently. Some of them might give it back directly and some might not. In fact, the same implementation will also behave differently depending on the allocation sizes and patterns.
This behavior, of course, is for good reasons:
- It is not always possible. OS-level memory allocations usually are done in pages (4KB, 4MB, or ... sizes at once). And if a small part of the page is still being used after freeing another part then the page cannot be given back to the operating system until that part is also freed.
- Efficiency. It is very likely that an application will ask for memory again. So why give it back to the OS and ask for it again soon after. (of course, there is probably a limit on the size of the memory kept.)
In most cases, you are not accountable for the memory you free
if the implementation decided to keep it (assuming it is a good implementation). Sooner or later it will be reallocated or returned to the OS. Hence, optimizing for memory usage should be based on the amount you have malloc
-ed and you haven't free
-d. The case where you have to worry about this, is when your allocation patterns/sizes start causing memory fragmentation which is a very big topic on its own.
If you are, however, on an embedded system and the amount of memory available is limited and you need more control over when/how memory is allocated and freed then you need to ask for memory pages from the OS directly and manage it manually.
Edit: I did not explain why you are not accountable for memory you free.
The reason is, on a modern OS, allocated memory is virtual. Meaning if you allocate 512MB on 32-bit system or 10TB of 64-bit system, as long as you don't read or write to that memory, it will not reserve any physical space for it. Actually, it will only reserve physical memory for the pages you touch from that big block and not the entire block. And after "a while of not using that memory", its contents will be copied to disk and the underlying physical memory will be used for something else.
1
Note that some allocators may avoid the possibility of copying data to disk by using OS specific calls that say "these pages aren't in use, so feel free to drop their contents, even though I'm not releasing the virtual memory itself". Example would be using themadvise
call on Linux withMADV_DONTNEED
.
– ShadowRanger
2 days ago
@ShadowRanger very interesting to know! thank you.
– Ameen
2 days ago
add a comment |
The C standard doesn't force on the implementer of malloc
and free
to return the memory to the OS directly. So different C library implementations will behave differently. Some of them might give it back directly and some might not. In fact, the same implementation will also behave differently depending on the allocation sizes and patterns.
This behavior, of course, is for good reasons:
- It is not always possible. OS-level memory allocations usually are done in pages (4KB, 4MB, or ... sizes at once). And if a small part of the page is still being used after freeing another part then the page cannot be given back to the operating system until that part is also freed.
- Efficiency. It is very likely that an application will ask for memory again. So why give it back to the OS and ask for it again soon after. (of course, there is probably a limit on the size of the memory kept.)
In most cases, you are not accountable for the memory you free
if the implementation decided to keep it (assuming it is a good implementation). Sooner or later it will be reallocated or returned to the OS. Hence, optimizing for memory usage should be based on the amount you have malloc
-ed and you haven't free
-d. The case where you have to worry about this, is when your allocation patterns/sizes start causing memory fragmentation which is a very big topic on its own.
If you are, however, on an embedded system and the amount of memory available is limited and you need more control over when/how memory is allocated and freed then you need to ask for memory pages from the OS directly and manage it manually.
Edit: I did not explain why you are not accountable for memory you free.
The reason is, on a modern OS, allocated memory is virtual. Meaning if you allocate 512MB on 32-bit system or 10TB of 64-bit system, as long as you don't read or write to that memory, it will not reserve any physical space for it. Actually, it will only reserve physical memory for the pages you touch from that big block and not the entire block. And after "a while of not using that memory", its contents will be copied to disk and the underlying physical memory will be used for something else.
1
Note that some allocators may avoid the possibility of copying data to disk by using OS specific calls that say "these pages aren't in use, so feel free to drop their contents, even though I'm not releasing the virtual memory itself". Example would be using themadvise
call on Linux withMADV_DONTNEED
.
– ShadowRanger
2 days ago
@ShadowRanger very interesting to know! thank you.
– Ameen
2 days ago
add a comment |
The C standard doesn't force on the implementer of malloc
and free
to return the memory to the OS directly. So different C library implementations will behave differently. Some of them might give it back directly and some might not. In fact, the same implementation will also behave differently depending on the allocation sizes and patterns.
This behavior, of course, is for good reasons:
- It is not always possible. OS-level memory allocations usually are done in pages (4KB, 4MB, or ... sizes at once). And if a small part of the page is still being used after freeing another part then the page cannot be given back to the operating system until that part is also freed.
- Efficiency. It is very likely that an application will ask for memory again. So why give it back to the OS and ask for it again soon after. (of course, there is probably a limit on the size of the memory kept.)
In most cases, you are not accountable for the memory you free
if the implementation decided to keep it (assuming it is a good implementation). Sooner or later it will be reallocated or returned to the OS. Hence, optimizing for memory usage should be based on the amount you have malloc
-ed and you haven't free
-d. The case where you have to worry about this, is when your allocation patterns/sizes start causing memory fragmentation which is a very big topic on its own.
If you are, however, on an embedded system and the amount of memory available is limited and you need more control over when/how memory is allocated and freed then you need to ask for memory pages from the OS directly and manage it manually.
Edit: I did not explain why you are not accountable for memory you free.
The reason is, on a modern OS, allocated memory is virtual. Meaning if you allocate 512MB on 32-bit system or 10TB of 64-bit system, as long as you don't read or write to that memory, it will not reserve any physical space for it. Actually, it will only reserve physical memory for the pages you touch from that big block and not the entire block. And after "a while of not using that memory", its contents will be copied to disk and the underlying physical memory will be used for something else.
The C standard doesn't force on the implementer of malloc
and free
to return the memory to the OS directly. So different C library implementations will behave differently. Some of them might give it back directly and some might not. In fact, the same implementation will also behave differently depending on the allocation sizes and patterns.
This behavior, of course, is for good reasons:
- It is not always possible. OS-level memory allocations usually are done in pages (4KB, 4MB, or ... sizes at once). And if a small part of the page is still being used after freeing another part then the page cannot be given back to the operating system until that part is also freed.
- Efficiency. It is very likely that an application will ask for memory again. So why give it back to the OS and ask for it again soon after. (of course, there is probably a limit on the size of the memory kept.)
In most cases, you are not accountable for the memory you free
if the implementation decided to keep it (assuming it is a good implementation). Sooner or later it will be reallocated or returned to the OS. Hence, optimizing for memory usage should be based on the amount you have malloc
-ed and you haven't free
-d. The case where you have to worry about this, is when your allocation patterns/sizes start causing memory fragmentation which is a very big topic on its own.
If you are, however, on an embedded system and the amount of memory available is limited and you need more control over when/how memory is allocated and freed then you need to ask for memory pages from the OS directly and manage it manually.
Edit: I did not explain why you are not accountable for memory you free.
The reason is, on a modern OS, allocated memory is virtual. Meaning if you allocate 512MB on 32-bit system or 10TB of 64-bit system, as long as you don't read or write to that memory, it will not reserve any physical space for it. Actually, it will only reserve physical memory for the pages you touch from that big block and not the entire block. And after "a while of not using that memory", its contents will be copied to disk and the underlying physical memory will be used for something else.
edited Mar 22 at 18:49
answered Mar 22 at 8:53
AmeenAmeen
7451519
7451519
1
Note that some allocators may avoid the possibility of copying data to disk by using OS specific calls that say "these pages aren't in use, so feel free to drop their contents, even though I'm not releasing the virtual memory itself". Example would be using themadvise
call on Linux withMADV_DONTNEED
.
– ShadowRanger
2 days ago
@ShadowRanger very interesting to know! thank you.
– Ameen
2 days ago
add a comment |
1
Note that some allocators may avoid the possibility of copying data to disk by using OS specific calls that say "these pages aren't in use, so feel free to drop their contents, even though I'm not releasing the virtual memory itself". Example would be using themadvise
call on Linux withMADV_DONTNEED
.
– ShadowRanger
2 days ago
@ShadowRanger very interesting to know! thank you.
– Ameen
2 days ago
1
1
Note that some allocators may avoid the possibility of copying data to disk by using OS specific calls that say "these pages aren't in use, so feel free to drop their contents, even though I'm not releasing the virtual memory itself". Example would be using the
madvise
call on Linux with MADV_DONTNEED
.– ShadowRanger
2 days ago
Note that some allocators may avoid the possibility of copying data to disk by using OS specific calls that say "these pages aren't in use, so feel free to drop their contents, even though I'm not releasing the virtual memory itself". Example would be using the
madvise
call on Linux with MADV_DONTNEED
.– ShadowRanger
2 days ago
@ShadowRanger very interesting to know! thank you.
– Ameen
2 days ago
@ShadowRanger very interesting to know! thank you.
– Ameen
2 days ago
add a comment |
This is very dependent on the actual malloc implementation in use.
Under Linux, there is a threshold (MMAP_THRESHOLD
) to decide where the memory for a given malloc()
request comes from.
If the requested amount is below or equal to MMAP_THRESHOLD
, the request is satisfied by either taking it from the so-called "free list", if any memory blocks have already been free()
d. Otherwise, the "break line" of the program (i. e. the end of the data segment) is increased and the memory made available to the program by this process is used for the request.
On free()
, the freed memory block is added to the free list. If there is enough free memory at the very end of the data segment, the break line (mentionned above) is moved again to shrink the data segment, returning the excess memory to the OS.
If the requested amount exceeds MMAP_THRESHOLD
, a separate memory block is requested by the OS and returned again during free()
.
See also https://linux.die.net/man/3/malloc for details.
add a comment |
This is very dependent on the actual malloc implementation in use.
Under Linux, there is a threshold (MMAP_THRESHOLD
) to decide where the memory for a given malloc()
request comes from.
If the requested amount is below or equal to MMAP_THRESHOLD
, the request is satisfied by either taking it from the so-called "free list", if any memory blocks have already been free()
d. Otherwise, the "break line" of the program (i. e. the end of the data segment) is increased and the memory made available to the program by this process is used for the request.
On free()
, the freed memory block is added to the free list. If there is enough free memory at the very end of the data segment, the break line (mentionned above) is moved again to shrink the data segment, returning the excess memory to the OS.
If the requested amount exceeds MMAP_THRESHOLD
, a separate memory block is requested by the OS and returned again during free()
.
See also https://linux.die.net/man/3/malloc for details.
add a comment |
This is very dependent on the actual malloc implementation in use.
Under Linux, there is a threshold (MMAP_THRESHOLD
) to decide where the memory for a given malloc()
request comes from.
If the requested amount is below or equal to MMAP_THRESHOLD
, the request is satisfied by either taking it from the so-called "free list", if any memory blocks have already been free()
d. Otherwise, the "break line" of the program (i. e. the end of the data segment) is increased and the memory made available to the program by this process is used for the request.
On free()
, the freed memory block is added to the free list. If there is enough free memory at the very end of the data segment, the break line (mentionned above) is moved again to shrink the data segment, returning the excess memory to the OS.
If the requested amount exceeds MMAP_THRESHOLD
, a separate memory block is requested by the OS and returned again during free()
.
See also https://linux.die.net/man/3/malloc for details.
This is very dependent on the actual malloc implementation in use.
Under Linux, there is a threshold (MMAP_THRESHOLD
) to decide where the memory for a given malloc()
request comes from.
If the requested amount is below or equal to MMAP_THRESHOLD
, the request is satisfied by either taking it from the so-called "free list", if any memory blocks have already been free()
d. Otherwise, the "break line" of the program (i. e. the end of the data segment) is increased and the memory made available to the program by this process is used for the request.
On free()
, the freed memory block is added to the free list. If there is enough free memory at the very end of the data segment, the break line (mentionned above) is moved again to shrink the data segment, returning the excess memory to the OS.
If the requested amount exceeds MMAP_THRESHOLD
, a separate memory block is requested by the OS and returned again during free()
.
See also https://linux.die.net/man/3/malloc for details.
answered Mar 22 at 14:06
glglglglglgl
67.9k796169
67.9k796169
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55294985%2fdoes-malloc-reserve-more-space-while-allocating-memory%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
-c++, dynamic-memory-allocation, free, malloc
Related: How do I free memory in C?
– Lundin
Mar 22 at 13:51
2
Possible duplicate of Memory usage doesn't decrease when free() used
– Useless
Mar 22 at 16:29
2
@Useless This question has better answers than the older one so I've bucked convention and marked the old question a duplicate of this one.
– John Kugelman
Mar 22 at 18:32
I think nearly all malloc/free implementations use some internal management which does request larger chunks and free them opportunistically. This may use
brk(2)
or mmap. It also means that pages might not actually get used before touched (and sometimes even uncommitted on free, so the virtual or data segment size is not so important)– eckes
Mar 22 at 19:42