WIP: libc.obj: Implementation of the libc memory allocator and internal mutex. #338
Draft
Egor00f
wants to merge 21 commits from
Egor00f/kolibrios:libc.obj--add-allocator into main
pull from: Egor00f/kolibrios:libc.obj--add-allocator
merge into: :main
:main
:chore/update-kterm
:krn_refactoring_curr_slot_idx_1
:dino-rewrite
:fat-fix-stack-imbalance
:kernel-tcp-socket-buffer-128k
:fix_602
:kernel-tcp-socket-list-locking
:network/getsockname
:opendial-update
:kernel-tcp-reassembly-queue
:kernel-tcp-connrefused-uaf
:rewrite_ide_drv
:libimg-png-8-bit
:apps/table-msvc-to-tcc
:netsurf-4
:vidmode-s3ide-clgd54xx-kms-etc
:updf-1.5
:pr-fs-unhardcode
:kbd-busoff
:webview-4
:workflow-fuse
:add-license-file-header-to-guide
:shell-improve-cpuid
:qrcodegen
:ci/update
:laser-tank-fix-win-height
:improvement/commit-and-branch-styles
:docs/libs
No Reviewers
Dismiss Review
Are you sure you want to dismiss this review?
Labels
Clear labels
AI
Eolite
FS
Good First PR
GSoC
HardwareTested
HLL
Influence/Settings
Influence/Text/TYPO
IRCC
Lang/C
Lang/FASM
Pay for the code
Subsystem/API
Subsystem/Audio
Subsystem/Graphics
Subsystem/IPC and events
Subsystem/Memory
Subsystem/Network
Subsystem/Services(daemon)
Subsystem/Taskmanager
Subsystem/VFS
Subsystem/Window
Category
Applications
Category
Drivers
Category
General
Category
Kernel
Category
Libraries
The issue is suitable to beginners
This issue or PR in the Google Source of Code program
Kind
Breaking
Breaking change that won't be backward compatible
Kind
Bug
Something is not working
Kind
Build
Kind
Documentation
Documentation changes
Kind
Enhancement
Improve existing functionality
Kind
Feature
New functionality
Kind
Security
This is security issue
Kind
Testing
Issue or pull request related to testing
Paid task
PR
Conflicts
PR conflicts with main
PR
Dependent
This PR is dependent on another PR
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
PR
Ready to merge
Pull request is ready for merge
PR
Request changes
Changes requested in pull request
PR
Review required
Reviewed
Confirmed
Issue has been confirmed
Reviewed
Duplicate
This issue or pull request already exists
Reviewed
Invalid
Invalid issue
Reviewed
Won't Fix
This issue won't be fixed
Status
Abandoned
Somebody has started to work on this but abandoned work
Status
Blocked
Something is blocking this issue or pull request
Status
Need More Info
Feedback is required to reproduce issue or to continue work
infinity service, audio drivers, midi, speacker, audio programs
vesa, vga, framebuffer, cursors, blitter, and video drivers
pipes, signals, events, shared memory
virt and phys memory allocators, malloc and other
userspace and kernel(for example: serial) services
process, threads, run apps, scheduler
drivers from filesystem, fs api, blkdev, programs that work with the file system
windows, skins, buttons, mouse and keyboard code for windows (not the base code)
Milestone
No items
No Milestone
Projects
Clear projects
No projects
No Assignees
Notifications
Due Date
No due date set.
Blocks
#205 malloc() in libc.obj gets at least 4k every time
KolibriOS/kolibrios
#563 WIP: `libc.obj`: add gcc support
KolibriOS/kolibrios
Reference: KolibriOS/kolibrios#338
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
A small and simple implementation of an allocator instead of calling kernel syscalls that allocate memory by page.
PS: for build
malloc_test.ckolibrios must have 128Mb ram or more, else tcc has page fault.malloc_testrequire ≈90MbOriginal PR: #311
@@ -10,1 +5,3 @@return NULL;void* ptr = malloc(num * size);if (ptr) {memset(ptr, 0, num * size);Should
callocreally callmemset?Yes, the standard definitely guarantees that the memory allocated by
calloc()is zeroed out.@Egor00f If you wrote this allocator yourself, you can add your authorship by analogy:
If you got this from somewhere, it’s better to keep the original copyrights
libc.obj: Add allocatorto libc.obj: Impl allocator (fix #205)libc.obj: Impl allocator (fix #205)to libc.obj: Impl allocator@@ -0,0 +136,4 @@unsigned char* byte_ptr = (unsigned char*)ptr;for (size_t i = 0; i < size; ++i) {if (byte_ptr[i] != pattern) {fprintf(stderr, "Error: Byte %zu does not match pattern. Expected %02X, got %02X\n",Make sure libc.obj printf() supports %zu
@@ -5,3 +39,2 @@return _ksys_alloc(size);}char b[32];Unnecessary code?
yes
@Egor00f сделай почту по приличнее если решил авторство добавить
так и нужно будет сделать ещё мьютексы
170fa3e1b0to7956ab38e73dec2b2093to3a7c70ee47ea01d69852to86cf2d77b486cf2d77b4to488786a5c9WIP: libc.obj: Impl allocatorto WIP: libc.obj: Implementation of the libc memory allocator and internal mutex.@@ -7,6 +7,7 @@#include "sys/closedir.c"#include "sys/dir.c"#include "sys/ksys.h"необходимо ли добавление включения данного файла ?
ща бы вспомнить почему оно вообще сдесь
кароч убрал
@@ -45,0 +47,4 @@{ "libc.o", extra_inputs = { "libc.lds" } },"kos32-ld -r -Tlibc.lds %f -o %o" .. tup.getconfig("KPACK_CMD"),"%B.obj")нет ли способа обойтись без линкера?
Не, смысл в том чтобы автоматически собирать указатели на мьютексы в один массив.
tcc не умеет в скрипты, а как тут ещё сделать так чтобы массив указателей собирался сам я хз. Гугл ничего лучше скриптов не выдаёт.
Кароч можно, но тогда всё ручками.
Можно было ещё добавить мьютексу поле
is_initи каждый раз проверять что он инициализирован, но терять время на это тоже вариант говна как мне кажетсяхорошо, если так реализовать проще, то принято
@@ -131,0 +132,4 @@void __libc_init_all_mutexes(void){size_t count = __mutex_init_end - __mutex_init_start;Возможно имеется ошибка. Разве нет необходимости для вычисления кол-ва элементов массива разделить это значение на sizeof(__libc_mutex_t)?
__mutex_init_startи__mutex_init_endэто объявленны как указатели на мьютекс. Оно так и должно работать. Или можно приводить кchar*.Вместо моих скомканых объяснений: https://www.google.com/search?q=%D0%B2%D1%8B%D1%87%D0%B8%D1%82%D0%B0%D0%BD%D0%B8%D0%B5%20%D1%83%D0%BA%D0%B0%D0%B7%D0%B0%D1%82%D0%B5%D0%BB%D0%B5%D0%B9%20%D0%B2%20%D1%81%D0%B8#sbfbu=1&pi=%D0%B2%D1%8B%D1%87%D0%B8%D1%82%D0%B0%D0%BD%D0%B8%D0%B5%20%D1%83%D0%BA%D0%B0%D0%B7%D0%B0%D1%82%D0%B5%D0%BB%D0%B5%D0%B9%20%D0%B2%20%D1%81%D0%B8
4029b83cebto404d3d94fc@@ -3,0 +109,4 @@if (current_node->next != NULL) {current_node->next->last = new_node;}current_node->next = new_node;What if
new_nodewas created from__last_biggest_mem_node(🔗), butcurrent_nodeis still equal to__mem_node(🔗) and points to an earlier node in the list?уххххххх ща бы вспомнить как аллокатор в принципе работаетаааа реально пропустил.
404d3d94fcto98ecdce5cc@@ -3,0 +70,4 @@}}else // new_node from __last_biggest_mem_node{Mixed bracket style is hard to read
Use
or
@@ -3,0 +102,4 @@if (!from_empty_node) {// Set the last pointer of the new node to the current node.new_node->last = current_node;How the
new_nodeis attached to the list ifcurrent_nodewas defined at L69 and equals NULL (due to the while condition, w/o break) and that's why?
Btw, why
last, notprev?because my English is bad
No way. If
__mem_nodeis NULL, then there are no nodes yet except the one being created. So,lastandnext. The currentnew_nodewill be the root of the list.__mem_nodeis just a pointer to the root of the list.I said nothing about
__mem_node, did I?new_nodecan become a root of list.idk
кароч похуй что
NULLоно так и должно работать3051239d3eto4a5e8c2cb2не чет косячит
Косячит в
freeкажетсяa я тесты криво написал. хахахахахахаххахахаххахахахахаххахахахаха на осознание этого ушло около суток
e894a5b891toaa56e9a482dfa5bf62bfto2ae5e5464b0290c64f5etof9855a8ab637a584d50fto855b233462#338 (comment)
855b233462toeb9c181b1c06aa42afadtoeb9c181b1cCritical:
free.c:36-46: merged block is never returned to the kernel.__mem_MERGE_MEM_NODESreturnsNULLwhenever the neighbour isn't free+adjacent (i.e. any node with an in-usenext). ThennodeisNULL,node->lastreads addr 8, the backward merge and the wholeif (node)block (Step 4/5 =_ksys_free) are skipped. Backward coalescing and block release are dead for any non-tail node. Fix: keep the merge result in a temp, advancenodeonly on success.malloc.c:107-130: a fresh block is orphaned from the list.After walking a non-empty list without a hit,
current_node == NULL(not because the list is empty). The new node getslast=next=NULLandif (__mem_node==NULL)is false, so nothing links it. Free space in orphaned blocks becomes unreachable (heap only grows) andexit.ccan't see them. If the loop instead stops on the counter limit, the fresh node is spliced into the middle of another block's run, breaking the invariantGET_MEM_BLOCKrelies on. Fix: actually link the new node in. (This is the open thread - Sweetbread is right.)_mem.h:91-123: merge doesn't invalidate__last_biggest_mem_node.The merge zeroes
additionbut never updates the tracker. If__last_biggest == addition, it now points at a zeroed header inside the merged block's free area — memory that will be handed to a user; a latermallocreads->freeoff user data → heap corruption. Fix:if (__last_biggest_mem_node == addition) __last_biggest_mem_node = base;.Medium:
_mem.h:59-61MEM_NODE_IS_ONE_IN_BLOCK: NULL-guards are crossed vs the pairs they protect → NULL deref of a low address whenlast==NULL, next!=NULL; only saved by the readable page + the exactblock->sizecheck.calloc.c:5:malloc(num * size)overflows. Addif (size && num > SIZE_MAX/size) return NULL;.malloc.c:136:__total_num_of_mem_nodes++is unconditional (bumps even when reusing a fully-free node) → counter drifts → perturbs themin(512,total)search bound.malloc.c:56-77: search is forward-only from__last_biggest_mem_node; free space before the tracker is never reused → heap grows.Minor:
realloc.c:30-41: dead branch - merge needs a freeaddition, but the realloc'd node is in use, so it never runs (realloc still works via the malloc fallback). Also reads used-size after the merge would zero it. Remove or fix.free.c:98-111: unreachable (tracker already set by the preceding chain).mutex.h: mutex never destroyed → one leaked futex per process.Headers use
inlinewithoutstatic/extern- relies on tcc leniency.malloc.c:150#undef __mem_align: no-op (it's an inline fn).Build dropped
strip --strip-unneeded.Need verification:
Mutex init via
libc.lds.__mutex_init_start/endmust be relocated by the KolibriOSdll.objloader; if not,__mem_alloc_mutexstays uninitialized and the firstmallocspins in_ksys_futex_waiton id 0.lib_initwiring itself is correct (dll.inc:120).Tests don't cover coalescing / block-return (dead due to C1), so "tests pass" proves little there.
EXIT_SUCCESS/FAILUREinstdlib.h5aa88b1c16current_nodestate whennow_nodecreating from__last_biggest_mem_node2f08e867e2allocator_maptest 1ffea408b8test_malloc_random_multiple_allocations&& add more comments a1b441a55dexit&& update tests 80b3e5791dallocator_map.c: add output to file 6399065c76srand(time(NULL));to test af6c0d3fa1eb9c181b1cto323e07d91eView command line instructions
Checkout
From your project repository, check out a new branch and test the changes.