WIP: libc.obj: Implementation of the libc memory allocator and internal mutex. #338

Draft
Egor00f wants to merge 21 commits from Egor00f/kolibrios:libc.obj--add-allocator into main
Owner

A small and simple implementation of an allocator instead of calling kernel syscalls that allocate memory by page.

  • allocator
  • free alocated memory in exit
  • multi-thread
    • mutex implementation
    • mutex init way

PS: for build malloc_test.c kolibrios must have 128Mb ram or more, else tcc has page fault. malloc_test require ≈90Mb

Original PR: #311

A small and simple implementation of an allocator instead of calling kernel syscalls that allocate memory by page. - [x] allocator - [x] free alocated memory in exit - [x] multi-thread - [x] mutex implementation - [x] mutex init way PS: for build `malloc_test.c` kolibrios must have 128Mb ram or more, else tcc has page fault. `malloc_test` require ≈90Mb Original PR: #311
Egor00f requested review from mxlgv 2026-02-22 08:00:42 +00:00
Egor00f reviewed 2026-02-22 08:07:05 +00:00
@@ -10,1 +5,3 @@
return NULL;
void* ptr = malloc(num * size);
if (ptr) {
memset(ptr, 0, num * size);
Author
Owner

Should calloc really call memset?

Should `calloc` really call `memset`?
Owner

Yes, the standard definitely guarantees that the memory allocated by calloc() is zeroed out.

Yes, the standard definitely guarantees that the memory allocated by `calloc()` is zeroed out.
mxlgv marked this conversation as resolved
Owner

@Egor00f If you wrote this allocator yourself, you can add your authorship by analogy:

/*
* SPDX-License-Identifier: GPL-2.0-only
* Copyright (C) 2026 KolibriOS team
* Author: Maxim Logaev <maxlogaev@proton.me>
*/

If you got this from somewhere, it’s better to keep the original copyrights

@Egor00f If you wrote this allocator yourself, you can add your authorship by analogy: https://git.kolibrios.org/KolibriOS/ports/src/commit/a797e4b21de0f4e8bfc6f6797dbfff3b71b086ce/libraries/newlib/kossup/src/posix/threads/pthread_create.c#L1-L5 If you got this from somewhere, it’s better to keep the original copyrights
mxlgv added the Lang/C
Kind
Enhancement
Priority
Low
4
PR
Request changes
labels 2026-02-23 23:37:42 +00:00
mxlgv changed title from libc.obj: Add allocator to libc.obj: Impl allocator (fix #205) 2026-02-23 23:50:30 +00:00
mxlgv changed title from libc.obj: Impl allocator (fix #205) to libc.obj: Impl allocator 2026-02-24 00:02:41 +00:00
mxlgv requested changes 2026-02-24 00:02:55 +00:00
@@ -0,0 +136,4 @@
unsigned char* byte_ptr = (unsigned char*)ptr;
for (size_t i = 0; i < size; ++i) {
if (byte_ptr[i] != pattern) {
fprintf(stderr, "Error: Byte %zu does not match pattern. Expected %02X, got %02X\n",
Owner

Make sure libc.obj printf() supports %zu

Make sure libc.obj printf() supports %zu
Egor00f marked this conversation as resolved
@@ -5,3 +39,2 @@
return _ksys_alloc(size);
}
char b[32];
Owner

Unnecessary code?

Unnecessary code?
Author
Owner

yes

yes
Egor00f marked this conversation as resolved
Owner

@Egor00f сделай почту по приличнее если решил авторство добавить

@Egor00f сделай почту по приличнее если решил авторство добавить
Egor00f marked the pull request as work in progress 2026-03-08 15:06:56 +00:00
Author
Owner

так и нужно будет сделать ещё мьютексы

так и нужно будет сделать ещё мьютексы
Egor00f force-pushed libc.obj--add-allocator from 170fa3e1b0 to 7956ab38e7 2026-03-15 06:49:55 +00:00 Compare
Egor00f force-pushed libc.obj--add-allocator from 3dec2b2093 to 3a7c70ee47 2026-05-24 16:45:06 +00:00 Compare
Egor00f force-pushed libc.obj--add-allocator from ea01d69852 to 86cf2d77b4 2026-05-28 08:59:14 +00:00 Compare
Egor00f added a new dependency 2026-06-22 07:18:18 +00:00
Egor00f force-pushed libc.obj--add-allocator from 86cf2d77b4 to 488786a5c9 2026-06-22 07:21:03 +00:00 Compare
Egor00f changed title from WIP: libc.obj: Impl allocator to WIP: libc.obj: Implementation of the libc memory allocator and internal mutex. 2026-06-22 08:52:26 +00:00
Egor00f marked the pull request as ready for review 2026-06-22 17:12:36 +00:00
Egor00f requested review from mxlgv 2026-06-22 17:12:58 +00:00
Egor00f requested review from ace-dent 2026-06-22 17:12:58 +00:00
Egor00f requested review from bad_Dr3dd0x 2026-06-22 17:12:58 +00:00
Egor00f requested review from Burer 2026-06-22 17:12:58 +00:00
Egor00f requested review from Doczom 2026-06-22 17:12:59 +00:00
Egor00f requested review from dunkaist 2026-06-22 17:12:59 +00:00
Egor00f requested review from hidnplayr 2026-06-22 17:12:59 +00:00
Egor00f requested review from IgorA 2026-06-22 17:12:59 +00:00
Egor00f requested review from Leency 2026-06-22 17:12:59 +00:00
Egor00f requested review from ramenu 2026-06-22 17:13:00 +00:00
Egor00f requested review from rgimad 2026-06-22 17:13:00 +00:00
Egor00f requested review from sdongles 2026-06-22 17:13:00 +00:00
Egor00f requested review from Sweetbread 2026-06-22 17:13:00 +00:00
Doczom reviewed 2026-06-22 18:16:37 +00:00
@@ -7,6 +7,7 @@
#include "sys/closedir.c"
#include "sys/dir.c"
#include "sys/ksys.h"
Owner

необходимо ли добавление включения данного файла ?

необходимо ли добавление включения данного файла ?
Author
Owner

ща бы вспомнить почему оно вообще сдесь

ща бы вспомнить почему оно вообще сдесь
Author
Owner

кароч убрал

кароч убрал
Egor00f marked this conversation as resolved
Doczom requested changes 2026-06-23 17:38:03 +00:00
Dismissed
@@ -45,0 +47,4 @@
{ "libc.o", extra_inputs = { "libc.lds" } },
"kos32-ld -r -Tlibc.lds %f -o %o" .. tup.getconfig("KPACK_CMD"),
"%B.obj"
)
Owner

нет ли способа обойтись без линкера?

нет ли способа обойтись без линкера?
Author
Owner

Не, смысл в том чтобы автоматически собирать указатели на мьютексы в один массив.
tcc не умеет в скрипты, а как тут ещё сделать так чтобы массив указателей собирался сам я хз. Гугл ничего лучше скриптов не выдаёт.

Кароч можно, но тогда всё ручками.

Можно было ещё добавить мьютексу поле is_init и каждый раз проверять что он инициализирован, но терять время на это тоже вариант говна как мне кажется

Не, смысл в том чтобы автоматически собирать указатели на мьютексы в один массив. tcc не умеет в скрипты, а как тут ещё сделать так чтобы массив указателей собирался сам я хз. Гугл ничего лучше скриптов не выдаёт. Кароч можно, но тогда всё ручками. Можно было ещё добавить мьютексу поле `is_init` и каждый раз проверять что он инициализирован, но терять время на это тоже вариант говна как мне кажется
Owner

хорошо, если так реализовать проще, то принято

хорошо, если так реализовать проще, то принято
Egor00f marked this conversation as resolved
@@ -131,0 +132,4 @@
void __libc_init_all_mutexes(void)
{
size_t count = __mutex_init_end - __mutex_init_start;
Owner

Возможно имеется ошибка. Разве нет необходимости для вычисления кол-ва элементов массива разделить это значение на sizeof(__libc_mutex_t)?

Возможно имеется ошибка. Разве нет необходимости для вычисления кол-ва элементов массива разделить это значение на sizeof(__libc_mutex_t)?
Author
Owner

__mutex_init_start и __mutex_init_end это объявленны как указатели на мьютекс. Оно так и должно работать. Или можно приводить к char*.
Вместо моих скомканых объяснений: https://www.google.com/search?q=%D0%B2%D1%8B%D1%87%D0%B8%D1%82%D0%B0%D0%BD%D0%B8%D0%B5%20%D1%83%D0%BA%D0%B0%D0%B7%D0%B0%D1%82%D0%B5%D0%BB%D0%B5%D0%B9%20%D0%B2%20%D1%81%D0%B8#sbfbu=1&pi=%D0%B2%D1%8B%D1%87%D0%B8%D1%82%D0%B0%D0%BD%D0%B8%D0%B5%20%D1%83%D0%BA%D0%B0%D0%B7%D0%B0%D1%82%D0%B5%D0%BB%D0%B5%D0%B9%20%D0%B2%20%D1%81%D0%B8

`__mutex_init_start` и `__mutex_init_end` это объявленны как указатели на мьютекс. Оно так и должно работать. Или можно приводить к `char*`. Вместо моих скомканых объяснений: https://www.google.com/search?q=%D0%B2%D1%8B%D1%87%D0%B8%D1%82%D0%B0%D0%BD%D0%B8%D0%B5%20%D1%83%D0%BA%D0%B0%D0%B7%D0%B0%D1%82%D0%B5%D0%BB%D0%B5%D0%B9%20%D0%B2%20%D1%81%D0%B8#sbfbu=1&pi=%D0%B2%D1%8B%D1%87%D0%B8%D1%82%D0%B0%D0%BD%D0%B8%D0%B5%20%D1%83%D0%BA%D0%B0%D0%B7%D0%B0%D1%82%D0%B5%D0%BB%D0%B5%D0%B9%20%D0%B2%20%D1%81%D0%B8
Doczom marked this conversation as resolved
Egor00f force-pushed libc.obj--add-allocator from 4029b83ceb to 404d3d94fc 2026-06-23 19:35:43 +00:00 Compare
Doczom approved these changes 2026-06-23 21:47:55 +00:00
Egor00f added a new dependency 2026-06-24 17:25:10 +00:00
Sweetbread reviewed 2026-06-26 00:18:21 +00:00
@@ -3,0 +109,4 @@
if (current_node->next != NULL) {
current_node->next->last = new_node;
}
current_node->next = new_node;
Owner

What if new_node was created from __last_biggest_mem_node (🔗), but current_node is still equal to __mem_node (🔗) and points to an earlier node in the list?

What if `new_node` was created from `__last_biggest_mem_node` ([&#128279;](https://git.kolibrios.org/KolibriOS/kolibrios/src/commit/404d3d94fc32cfdf9b2a6c85e01ca67ea9ce0f91/programs/develop/ktcc/libc.obj/source/stdlib/malloc.c#L59)), but `current_node` is still equal to `__mem_node` ([&#128279;](https://git.kolibrios.org/KolibriOS/kolibrios/src/commit/404d3d94fc32cfdf9b2a6c85e01ca67ea9ce0f91/programs/develop/ktcc/libc.obj/source/stdlib/malloc.c#L54)) and points to an earlier node in the list?
Author
Owner

уххххххх ща бы вспомнить как аллокатор в принципе работает

аааа реально пропустил.

~~уххххххх ща бы вспомнить как аллокатор в принципе работает~~ аааа реально пропустил.
Sweetbread marked this conversation as resolved
Egor00f force-pushed libc.obj--add-allocator from 404d3d94fc to 98ecdce5cc 2026-06-26 09:42:46 +00:00 Compare
Sweetbread reviewed 2026-06-27 05:57:01 +00:00
@@ -3,0 +70,4 @@
}
}
else // new_node from __last_biggest_mem_node
{
Owner

Mixed bracket style is hard to read

Use

if (...) {
  ...
} else {
  ...
}

or

if (...)
{
  ...
}
else
{
  ...
}
Mixed bracket style is hard to read Use ```c if (...) { ... } else { ... } ``` or ```c if (...) { ... } else { ... } ```
Egor00f marked this conversation as resolved
@@ -3,0 +102,4 @@
if (!from_empty_node) {
// Set the last pointer of the new node to the current node.
new_node->last = current_node;
Owner

How the new_node is attached to the list if current_node was defined at L69 and equals NULL (due to the while condition, w/o break) and that's why

new_node->last = NULL;
new_node->next = NULL;

?
Btw, why last, not prev?

How the `new_node` is attached to the list if `current_node` was defined at [L69](https://git.kolibrios.org/KolibriOS/kolibrios/src/commit/4c82a17dc0110a30a79f964d018f10bc6f18c930/programs/develop/ktcc/libc.obj/source/stdlib/malloc.c#L69) and equals NULL (due to the while condition, w/o break) and that's why ```c new_node->last = NULL; new_node->next = NULL; ``` ? Btw, why `last`, not `prev`?
Author
Owner

Btw, why last, not prev?

because my English is bad

How the new_node is attached to the list if current_node was defined at L69 and equals NULL (due to the while condition, w/o break) and that's why

No way. If __mem_node is NULL, then there are no nodes yet except the one being created. So, last and next. The current new_node will be the root of the list. __mem_node is just a pointer to the root of the list.

> Btw, why last, not prev? because my English is bad > How the new_node is attached to the list if current_node was defined at [L69](https://git.kolibrios.org/KolibriOS/kolibrios/src/commit/4c82a17dc0110a30a79f964d018f10bc6f18c930/programs/develop/ktcc/libc.obj/source/stdlib/malloc.c#L69) and equals NULL (due to the while condition, w/o break) and that's why No way. If `__mem_node` is NULL, then there are no nodes yet except the one being created. So, `last` and `next`. The current `new_node` will be the root of the list. `__mem_node` is just a pointer to the root of the list.
Owner

If __mem_node is NULL

I said nothing about __mem_node, did I?

> If `__mem_node` is NULL I said nothing about `__mem_node`, did I?
Author
Owner

struct mem_node* current_node = __mem_node;

https://git.kolibrios.org/KolibriOS/kolibrios/src/commit/855b233462d26795c42e47b26661c08b227bf0c2/programs/develop/ktcc/libc.obj/source/stdlib/malloc.c#L56
Author
Owner

new_node can become a root of list.

if (__mem_node == NULL) {
__mem_node = new_node;
}

`new_node` can become a root of list. https://git.kolibrios.org/KolibriOS/kolibrios/src/commit/855b233462d26795c42e47b26661c08b227bf0c2/programs/develop/ktcc/libc.obj/source/stdlib/malloc.c#L128-L130
Author
Owner

idk

idk
Author
Owner

кароч похуй что NULL оно так и должно работать

кароч похуй что `NULL` оно так и должно работать
Egor00f marked this conversation as resolved
Egor00f force-pushed libc.obj--add-allocator from 3051239d3e to 4a5e8c2cb2 2026-06-28 08:25:22 +00:00 Compare
Author
Owner

не чет косячит
Косячит в free кажется
a я тесты криво написал. хахахахахахаххахахаххахахахахаххахахахаха на осознание этого ушло около суток

не чет косячит Косячит в `free` кажется a я тесты криво написал. хахахахахахаххахахаххахахахахаххахахахаха на осознание этого ушло около суток
Egor00f added
Category
Libraries
PR
Ready to merge
and removed
PR
Request changes
labels 2026-06-29 05:08:10 +00:00
Egor00f force-pushed libc.obj--add-allocator from e894a5b891 to aa56e9a482 2026-07-03 09:33:31 +00:00 Compare
Egor00f force-pushed libc.obj--add-allocator from dfa5bf62bf to 2ae5e5464b 2026-07-03 18:15:58 +00:00 Compare
Egor00f force-pushed libc.obj--add-allocator from 0290c64f5e to f9855a8ab6 2026-07-03 18:25:49 +00:00 Compare
dunkaist approved these changes 2026-07-03 18:45:26 +00:00
Dismissed
dunkaist approved these changes 2026-07-03 18:45:31 +00:00
bad_Dr3dd0x approved these changes 2026-07-03 18:47:55 +00:00
Egor00f force-pushed libc.obj--add-allocator from 37a584d50f to 855b233462 2026-07-06 14:43:52 +00:00 Compare
Sweetbread requested changes 2026-07-07 14:53:00 +00:00
Sweetbread left a comment
Owner
https://git.kolibrios.org/KolibriOS/kolibrios/pulls/338#issuecomment-8905
Egor00f force-pushed libc.obj--add-allocator from 855b233462 to eb9c181b1c 2026-07-10 13:10:46 +00:00 Compare
Egor00f force-pushed libc.obj--add-allocator from 06aa42afad to eb9c181b1c 2026-07-10 16:39:26 +00:00 Compare
Egor00f added a new dependency 2026-07-10 16:48:44 +00:00
Burer requested changes 2026-07-12 08:45:29 +00:00
Burer left a comment
Owner

Critical:

free.c:36-46: merged block is never returned to the kernel.
__mem_MERGE_MEM_NODES returns NULL whenever the neighbour isn't free+adjacent (i.e. any node with an in-use next). Then node is NULL, node->last reads addr 8, the backward merge and the whole if (node) block (Step 4/5 = _ksys_free) are skipped. Backward coalescing and block release are dead for any non-tail node. Fix: keep the merge result in a temp, advance node only on success.

malloc.c:107-130: a fresh block is orphaned from the list.
After walking a non-empty list without a hit, current_node == NULL (not because the list is empty). The new node gets last=next=NULL and if (__mem_node==NULL) is false, so nothing links it. Free space in orphaned blocks becomes unreachable (heap only grows) and exit.c can't see them. If the loop instead stops on the counter limit, the fresh node is spliced into the middle of another block's run, breaking the invariant GET_MEM_BLOCK relies on. Fix: actually link the new node in. (This is the open thread - Sweetbread is right.)

_mem.h:91-123: merge doesn't invalidate __last_biggest_mem_node.
The merge zeroes addition but never updates the tracker. If __last_biggest == addition, it now points at a zeroed header inside the merged block's free area — memory that will be handed to a user; a later malloc reads ->free off user data → heap corruption. Fix: if (__last_biggest_mem_node == addition) __last_biggest_mem_node = base;.

Medium:

_mem.h:59-61 MEM_NODE_IS_ONE_IN_BLOCK: NULL-guards are crossed vs the pairs they protect → NULL deref of a low address when last==NULL, next!=NULL; only saved by the readable page + the exact block->size check.

calloc.c:5: malloc(num * size) overflows. Add if (size && num > SIZE_MAX/size) return NULL;.

malloc.c:136: __total_num_of_mem_nodes++ is unconditional (bumps even when reusing a fully-free node) → counter drifts → perturbs the min(512,total) search bound.

malloc.c:56-77: search is forward-only from __last_biggest_mem_node; free space before the tracker is never reused → heap grows.

Minor:

realloc.c:30-41: dead branch - merge needs a free addition, but the realloc'd node is in use, so it never runs (realloc still works via the malloc fallback). Also reads used-size after the merge would zero it. Remove or fix.

free.c:98-111: unreachable (tracker already set by the preceding chain).

mutex.h: mutex never destroyed → one leaked futex per process.

Headers use inline without static/extern - relies on tcc leniency.

malloc.c:150 #undef __mem_align: no-op (it's an inline fn).

Build dropped strip --strip-unneeded.

Need verification:

Mutex init via libc.lds. __mutex_init_start/end must be relocated by the KolibriOS dll.obj loader; if not, __mem_alloc_mutex stays uninitialized and the first malloc spins in _ksys_futex_wait on id 0. lib_init wiring itself is correct (dll.inc:120).

Tests don't cover coalescing / block-return (dead due to C1), so "tests pass" proves little there.

**Critical:** **`free.c:36-46`: merged block is never returned to the kernel.** `__mem_MERGE_MEM_NODES` returns `NULL` whenever the neighbour isn't free+adjacent (i.e. any node with an in-use `next`). Then `node` is `NULL`, `node->last` reads addr 8, the backward merge and the whole `if (node)` block (Step 4/5 = `_ksys_free`) are skipped. Backward coalescing and block release are dead for any non-tail node. Fix: keep the merge result in a temp, advance `node` only on success. **`malloc.c:107-130`: a fresh block is orphaned from the list.** After walking a non-empty list without a hit, `current_node == NULL` (not because the list is empty). The new node gets `last=next=NULL` and `if (__mem_node==NULL)` is false, so nothing links it. Free space in orphaned blocks becomes unreachable (heap only grows) and `exit.c` can't see them. If the loop instead stops on the counter limit, the fresh node is spliced into the middle of another block's run, breaking the invariant `GET_MEM_BLOCK` relies on. Fix: actually link the new node in. (This is the open thread - Sweetbread is right.) **`_mem.h:91-123`: merge doesn't invalidate `__last_biggest_mem_node`.** The merge zeroes `addition` but never updates the tracker. If `__last_biggest == addition`, it now points at a zeroed header inside the merged block's free area — memory that will be handed to a user; a later `malloc` reads `->free` off user data → heap corruption. Fix: `if (__last_biggest_mem_node == addition) __last_biggest_mem_node = base;`. **Medium:** **`_mem.h:59-61` `MEM_NODE_IS_ONE_IN_BLOCK`:** NULL-guards are crossed vs the pairs they protect → NULL deref of a low address when `last==NULL, next!=NULL`; only saved by the readable page + the exact `block->size` check. **`calloc.c:5`:** `malloc(num * size)` overflows. Add `if (size && num > SIZE_MAX/size) return NULL;`. **`malloc.c:136`:** `__total_num_of_mem_nodes++` is unconditional (bumps even when reusing a fully-free node) → counter drifts → perturbs the `min(512,total)` search bound. **`malloc.c:56-77`:** search is forward-only from `__last_biggest_mem_node`; free space before the tracker is never reused → heap grows. **Minor:** **`realloc.c:30-41`:** dead branch - merge needs a free `addition`, but the realloc'd node is in use, so it never runs (realloc still works via the malloc fallback). Also reads used-size after the merge would zero it. Remove or fix. `free.c:98-111`: unreachable (tracker already set by the preceding chain). `mutex.h`: mutex never destroyed → one leaked futex per process. Headers use `inline` without `static`/`extern` - relies on tcc leniency. `malloc.c:150` `#undef __mem_align`: no-op (it's an inline fn). Build dropped `strip --strip-unneeded`. **Need verification:** **Mutex init via `libc.lds`.** `__mutex_init_start/end` must be relocated by the KolibriOS `dll.obj` loader; if not, `__mem_alloc_mutex` stays uninitialized and the first `malloc` spins in `_ksys_futex_wait` on id 0. `lib_init` wiring itself is correct (`dll.inc:120`). Tests don't cover coalescing / block-return (dead due to C1), so "tests pass" proves little there.
Egor00f removed the
PR
Ready to merge
label 2026-08-02 21:45:48 +00:00
Egor00f added 21 commits 2026-08-02 21:46:14 +00:00
Egor00f force-pushed libc.obj--add-allocator from eb9c181b1c to 323e07d91e 2026-08-02 21:46:14 +00:00 Compare
Egor00f marked the pull request as work in progress 2026-08-03 11:22:29 +00:00
Egor00f removed a dependency 2026-08-03 13:38:10 +00:00
Some required checks failed
Test PR / Build (ru_RU) (pull_request) Failing after 1m6s
Required
Details
Test PR / Build (es_ES) (pull_request) Successful in 1m47s
Required
Details
Test PR / Build (en_US) (pull_request) Successful in 1m50s
Required
Details
This pull request has changes conflicting with the target branch.
  • programs/develop/ktcc/libc.obj/samples/Makefile
  • programs/develop/ktcc/libc.obj/samples/build_all.sh
  • programs/develop/ktcc/libc.obj/source/libc.c
  • programs/develop/ktcc/libc.obj/source/libc.lds
  • programs/develop/ktcc/libc.obj/source/sys/mutex.h
This pull request is marked as a work in progress.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u http://git.kolibrios.org/Egor00f/kolibrios libc.obj--add-allocator:Egor00f-libc.obj--add-allocator
git checkout Egor00f-libc.obj--add-allocator
Sign in to join this conversation.
7 Participants
Notifications
Due Date
No due date set.
Reference: KolibriOS/kolibrios#338