kernel/taskman: free leaked handle-table objects on process exit (#567)

I wrote an autotest for Netsurf. After 130 test run the system could not
create any new process anymore. Now this is fixed. Result:
916 OK, 0 CRASH, 0 HANG

Futexes (sysfn 77) live in the per-process handle table (PROC.htab)
and are allocated from the kernel's small-object heap via
create_object. The only code path that ever frees one is
destroy_object, which runs solely on an explicit FUTEX_DESTROY call --
and real programs never make it: newlib-based applications create a
dozen or so futexes at startup for their internal locks (malloc,
stdio) and simply exit, expecting the kernel to clean up. Nothing
does: destroy_process tears down HDLLs and page tables but never
walks the handle table.

Each leaked futex pins 48 bytes of the kernel malloc arena, and that
arena is a single fixed 128 KB block (init_malloc has no grow path).
After roughly two thousand leaked handles -- a few hundred to a few
thousand application launches within one uptime -- kernel malloc()
starts failing system-wide. The first visible casualty is
load_library: it can no longer allocate a DLLDESCR, so every dll.obj
load in every new process fails from that point on ("cannot load
library"), which is easy to mistake for a userland problem.

Reproduced by repeatedly launching a newlib application: the arena
filled with ~1750 48-byte chunks carrying the 'FUTX' magic (confirmed
by dumping the live arena; mst.topsize had dropped to 32 bytes with
free physical memory and kernel heap space still abundant), and dll
loading died after ~130 launches.

Fix: in destroy_process, after destroy_all_hdlls, walk the process's
handle table and free every live object. Free slots hold small
free-list indices and the reserved stdin/stdout/stderr handles hold
small values, so a bounds check against OS_BASE skips them; live
slots hold kernel pointers and are additionally verified by the
'FUTX' magic. If other object kinds are ever added to the handle
table, they will need their own destructors here -- the magic check
makes this walk skip them safely rather than crash.

---------

Co-authored-by: Burer <burer@kolibrios.org>
Reviewed-on: #567
Reviewed-by: hidnplayr <hidnplayr@gmail.com>
Reviewed-by: Burer <burer@kolibrios.org>
Co-authored-by: leency <lipatov.kiril@gmail.com>
Co-committed-by: leency <lipatov.kiril@gmail.com>

kernel/trunk/core/taskman.inc

@@ -423,6 +423,51 @@ align 4
         mov     esi, [esi + PROC.dlls_list_ptr]
         call    destroy_all_hdlls
 
+; Free the process's live handle-table objects (sysfn 77): futexes and pipe
+; FILEDs. Nothing else frees them on exit: destroy_object runs only on an
+; explicit FUTEX_DESTROY call, which real programs (newlib mutexes etc.)
+; never make, and close() (77.9) with pipe_close are unwired stubs. Each
+; leaked object pins part of the FIXED 128 KB kernel malloc arena
+; (init_malloc has no grow path), so after a few thousand leaked handles
+; malloc() dies system-wide (first visible casualty: load_library can no
+; longer allocate a DLLDESCR -> every dll.obj load fails). Live htab slots
+; hold kernel pointers, free slots hold small free-list indices; the magic
+; checks keep the walk safe either way.
+        mov     esi, [esp]
+        add     esi, PROC.htab
+        mov     edi, (PROC.pdt_0 - PROC.htab)/4
+.free_handles:
+        mov     eax, [esi]
+        cmp     eax, OS_BASE
+        jb      .next_handle
+        push    esi edi
+        cmp     [eax + FUTEX.magic], 'FUTX'
+        je      .free_object
+        cmp     [eax + FILED.magic], 'PIPE'
+        jne     .keep_object
+; Pipes are process-local and all threads are gone: no waiters, no locking.
+; The last reference frees the PIPE and its ring buffer (kernel_free is
+; safe on the mirrored mapping: the page bitmap frees each frame once).
+        mov     ebx, [eax + FILED.file]
+        mov     ecx, [eax + FILED.mode]        ;F_READ=1 readers, F_WRITE=2 writers
+        dec     dword [ebx + PIPE.readers + ecx*4 - 4]
+        mov     ecx, [ebx + PIPE.readers]
+        or      ecx, [ebx + PIPE.writers]
+        jnz     .free_object
+        push    eax
+        stdcall kernel_free, [ebx + PIPE.buffer]
+        mov     eax, ebx
+        call    free
+        pop     eax
+.free_object:
+        call    free
+.keep_object:
+        pop     edi esi
+.next_handle:
+        add     esi, 4
+        dec     edi
+        jnz     .free_handles
+
         mov     esi, [esp]
         add     esi, PROC.pdt_0
         mov     edi, (0x80000000 shr 20)/4