Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
T
Tor
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Package registry
Container Registry
Model registry
Operate
Environments
Terraform modules
Monitor
Incidents
Service Desk
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
Benjamin J. Thompson
Tor
Commits
fb20618e
Commit
fb20618e
authored
5 years ago
by
Nick Mathewson
Browse files
Options
Downloads
Patches
Plain Diff
Divide 01a-memory.md into doxygen.
parent
4bf73dfa
No related branches found
Branches containing commit
No related tags found
Tags containing commit
No related merge requests found
Changes
3
Hide whitespace changes
Inline
Side-by-side
Showing
3 changed files
doc/HACKING/design/01a-memory.md
+0
-103
0 additions, 103 deletions
doc/HACKING/design/01a-memory.md
src/lib/malloc/lib_malloc.dox
+73
-0
73 additions, 0 deletions
src/lib/malloc/lib_malloc.dox
src/lib/memarea/lib_memarea.dox
+20
-0
20 additions, 0 deletions
src/lib/memarea/lib_memarea.dox
with
93 additions
and
103 deletions
doc/HACKING/design/01a-memory.md
deleted
100644 → 0
+
0
−
103
View file @
4bf73dfa
## Memory management
### Heap-allocation functions: lib/malloc/malloc.h
Tor imposes a few light wrappers over C's native malloc and free
functions, to improve convenience, and to allow wholescale replacement
of malloc and free as needed.
You should never use 'malloc', 'calloc', 'realloc, or 'free' on their
own; always use the variants prefixed with 'tor_'.
They are the same as the standard C functions, with the following
exceptions:
*
`tor_free(NULL)`
is a no-op.
*
`tor_free()`
is a macro that takes an lvalue as an argument and sets it to
NULL after freeing it. To avoid this behavior, you can use
`tor_free_()`
instead.
*
tor_malloc() and friends fail with an assertion if they are asked to
allocate a value so large that it is probably an underflow.
*
It is always safe to
`tor_malloc(0)`
, regardless of whether your libc
allows it.
*
`tor_malloc()`
,
`tor_realloc()`
, and friends are never allowed to fail.
Instead, Tor will die with an assertion. This means that you never
need to check their return values. See the next subsection for
information on why we think this is a good idea.
We define additional general-purpose memory allocation functions as well:
*
`tor_malloc_zero(x)`
behaves as
`calloc(1, x)`
, except the it makes clear
the intent to allocate a single zeroed-out value.
*
`tor_reallocarray(x,y)`
behaves as the OpenBSD reallocarray function.
Use it for cases when you need to realloc() in a multiplication-safe
way.
And specific-purpose functions as well:
*
`tor_strdup()`
and
`tor_strndup()`
behaves as the underlying libc
functions, but use
`tor_malloc()`
instead of the underlying function.
*
`tor_memdup()`
copies a chunk of memory of a given size.
*
`tor_memdup_nulterm()`
copies a chunk of memory of a given size, then
NUL-terminates it just to be safe.
#### Why assert on allocation failure?
Why don't we allow
`tor_malloc()`
and its allies to return NULL?
First, it's error-prone. Many programmers forget to check for NULL return
values, and testing for
`malloc()`
failures is a major pain.
Second, it's not necessarily a great way to handle OOM conditions. It's
probably better (we think) to have a memory target where we dynamically free
things ahead of time in order to stay under the target. Trying to respond to
an OOM at the point of
`tor_malloc()`
failure, on the other hand, would involve
a rare operation invoked from deep in the call stack. (Again, that's
error-prone and hard to debug.)
Third, thanks to the rise of Linux and other operating systems that allow
memory to be overcommitted, you can't actually ever rely on getting a NULL
from
`malloc()`
when you're out of memory; instead you have to use an approach
closer to tracking the total memory usage.
#### Conventions for your own allocation functions.
Whenever you create a new type, the convention is to give it a pair of
`x_new()`
and
`x_free_()`
functions, named after the type.
Calling
`x_free(NULL)`
should always be a no-op.
There should additionally be an
`x_free()`
macro, defined in terms of
`x_free_()`
. This macro should set its lvalue to NULL. You can define it
using the FREE_AND_NULL macro, as follows:
```
#define x_free(ptr) FREE_AND_NULL(x_t, x_free_, (ptr))
```
### Grow-only memory allocation: lib/memarea
It's often handy to allocate a large number of tiny objects, all of which
need to disappear at the same time. You can do this in tor using the
memarea.c abstraction, which uses a set of grow-only buffers for allocation,
and only supports a single "free" operation at the end.
Using memareas also helps you avoid memory fragmentation. You see, some libc
malloc implementations perform badly on the case where a large number of
small temporary objects are allocated at the same time as a few long-lived
objects of similar size. But if you use tor_malloc() for the long-lived ones
and a memarea for the temporary object, the malloc implementation is likelier
to do better.
To create a new memarea, use
`memarea_new()`
. To drop all the storage from a
memarea, and invalidate its pointers, use
`memarea_drop_all()`
.
The allocation functions
`memarea_alloc()`
,
`memarea_alloc_zero()`
,
`memarea_memdup()`
,
`memarea_strdup()`
, and
`memarea_strndup()`
are analogous
to the similarly-named malloc() functions. There is intentionally no
`memarea_free()`
or
`memarea_realloc()`
.
### Special allocation: lib/malloc/map_anon.h
TODO: WRITEME.
This diff is collapsed.
Click to expand it.
src/lib/malloc/lib_malloc.dox
+
73
−
0
View file @
fb20618e
...
@@ -2,4 +2,77 @@
...
@@ -2,4 +2,77 @@
@dir /lib/malloc
@dir /lib/malloc
@brief lib/malloc: Wrappers and utilities for memory management.
@brief lib/malloc: Wrappers and utilities for memory management.
Tor imposes a few light wrappers over C's native malloc and free
functions, to improve convenience, and to allow wholescale replacement
of malloc and free as needed.
You should never use 'malloc', 'calloc', 'realloc, or 'free' on their
own; always use the variants prefixed with 'tor_'.
They are the same as the standard C functions, with the following
exceptions:
* `tor_free(NULL)` is a no-op.
* `tor_free()` is a macro that takes an lvalue as an argument and sets it to
NULL after freeing it. To avoid this behavior, you can use `tor_free_()`
instead.
* tor_malloc() and friends fail with an assertion if they are asked to
allocate a value so large that it is probably an underflow.
* It is always safe to `tor_malloc(0)`, regardless of whether your libc
allows it.
* `tor_malloc()`, `tor_realloc()`, and friends are never allowed to fail.
Instead, Tor will die with an assertion. This means that you never
need to check their return values. See the next subsection for
information on why we think this is a good idea.
We define additional general-purpose memory allocation functions as well:
* `tor_malloc_zero(x)` behaves as `calloc(1, x)`, except the it makes clear
the intent to allocate a single zeroed-out value.
* `tor_reallocarray(x,y)` behaves as the OpenBSD reallocarray function.
Use it for cases when you need to realloc() in a multiplication-safe
way.
And specific-purpose functions as well:
* `tor_strdup()` and `tor_strndup()` behaves as the underlying libc
functions, but use `tor_malloc()` instead of the underlying function.
* `tor_memdup()` copies a chunk of memory of a given size.
* `tor_memdup_nulterm()` copies a chunk of memory of a given size, then
NUL-terminates it just to be safe.
#### Why assert on allocation failure?
Why don't we allow `tor_malloc()` and its allies to return NULL?
First, it's error-prone. Many programmers forget to check for NULL return
values, and testing for `malloc()` failures is a major pain.
Second, it's not necessarily a great way to handle OOM conditions. It's
probably better (we think) to have a memory target where we dynamically free
things ahead of time in order to stay under the target. Trying to respond to
an OOM at the point of `tor_malloc()` failure, on the other hand, would involve
a rare operation invoked from deep in the call stack. (Again, that's
error-prone and hard to debug.)
Third, thanks to the rise of Linux and other operating systems that allow
memory to be overcommitted, you can't actually ever rely on getting a NULL
from `malloc()` when you're out of memory; instead you have to use an approach
closer to tracking the total memory usage.
#### Conventions for your own allocation functions.
Whenever you create a new type, the convention is to give it a pair of
`x_new()` and `x_free_()` functions, named after the type.
Calling `x_free(NULL)` should always be a no-op.
There should additionally be an `x_free()` macro, defined in terms of
`x_free_()`. This macro should set its lvalue to NULL. You can define it
using the FREE_AND_NULL macro, as follows:
```
#define x_free(ptr) FREE_AND_NULL(x_t, x_free_, (ptr))
```
**/
**/
This diff is collapsed.
Click to expand it.
src/lib/memarea/lib_memarea.dox
+
20
−
0
View file @
fb20618e
...
@@ -7,4 +7,24 @@ once. This kind of allocation is very fast and avoids fragmentation, at the
...
@@ -7,4 +7,24 @@ once. This kind of allocation is very fast and avoids fragmentation, at the
expense of requiring all the data to be freed at the same time. We use this
expense of requiring all the data to be freed at the same time. We use this
for parsing and diff calculations.
for parsing and diff calculations.
It's often handy to allocate a large number of tiny objects, all of which
need to disappear at the same time. You can do this in tor using the
memarea.c abstraction, which uses a set of grow-only buffers for allocation,
and only supports a single "free" operation at the end.
Using memareas also helps you avoid memory fragmentation. You see, some libc
malloc implementations perform badly on the case where a large number of
small temporary objects are allocated at the same time as a few long-lived
objects of similar size. But if you use tor_malloc() for the long-lived ones
and a memarea for the temporary object, the malloc implementation is likelier
to do better.
To create a new memarea, use `memarea_new()`. To drop all the storage from a
memarea, and invalidate its pointers, use `memarea_drop_all()`.
The allocation functions `memarea_alloc()`, `memarea_alloc_zero()`,
`memarea_memdup()`, `memarea_strdup()`, and `memarea_strndup()` are analogous
to the similarly-named malloc() functions. There is intentionally no
`memarea_free()` or `memarea_realloc()`.
**/
**/
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment