On Windows, symbolic links actually have a type depending on the target:
it can be a file or a directory.
In certain circumstances, this poses problems, e.g. when a symbolic link
is supposed to point into a submodule that is not checked out, so there
is no way for Git to auto-detect the type.
To help with that, we will add support over the course of the next
commits to specify that symlink type via the Git attributes. This
requires an index_state, though, something that Git for Windows'
`symlink()` replacement cannot know about because the function signature
is defined by the POSIX standard and not ours to change.
So let's introduce a helper function to create symbolic links that
*does* know about the index_state.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The reftable library goes out of its way to use its own set of allocator
functions that can be configured using `reftable_set_alloc()`. However,
Git does not configure this.
That is not typically a problem, except when Git uses a custom allocator
via some definitions in `git-compat-util.h`, as is the case in Git for
Windows (which switched away from the long-unmaintained nedmalloc to
mimalloc).
Then, it is quite possible that Git assigns a `strbuf` (allocated via
the custom allocator) to, say, the `refname` field of a
`reftable_log_record` in `write_transaction_table()`, and later on asks
the reftable library function `reftable_log_record_release()` to release
it, but that function was compiled without using `git-compat-util.h` and
hence calls regular `free()` (i.e. _not_ the custom allocator's own
function).
This has been a problem for a long time and it was a matter of some sort
of "luck" that 1) reftables are not commonly used on Windows, and 2)
mimalloc can often ignore gracefully when it is asked to release memory
that it has not allocated.
However, a recent update to `seen` brought this problem to the
forefront, letting t1460 fail in Git for Windows, with symptoms much in
the same way as the problem I had to address in d02c37c3e6
(t-reftable-basics: allow for `malloc` to be `#define`d, 2025-01-08)
where exit code 127 was also produced in lieu of
`STATUS_HEAP_CORRUPTION` (C0000374) because exit codes are only 7 bits
wide.
It was not possible to figure out what change in particular caused these
new failures within a reasonable time frame, as there are too many
changes in `seen` that conflict with Git for Windows' patches, I had to
stop the investigation after spending four hours on it fruitlessly.
To verify that this patch fixes the issue, I avoided using mimalloc and
temporarily patched in a "custom allocator" that would more reliably
point out problems, like this:
diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c
index 68f38291f84c..9421d630b9f5 100644
--- a/refs/reftable-backend.c
+++ b/refs/reftable-backend.c
@@ -353,6 +353,69 @@ static int reftable_be_fsync(int fd)
return fsync_component(FSYNC_COMPONENT_REFERENCE, fd);
}
+#define DEBUG_REFTABLE_ALLOC
+#ifdef DEBUG_REFTABLE_ALLOC
+#include "khash.h"
+
+static inline khint_t __ac_X31_hash_ptr(void *ptr)
+{
+ union {
+ void *ptr;
+ char s[sizeof(void *)];
+ } u;
+ size_t i;
+ khint_t h;
+
+ u.ptr = ptr;
+ h = (khint_t)*u.s;
+ for (i = 0; i < sizeof(void *); i++)
+ h = (h << 5) - h + (khint_t)u.s[i];
+ return h;
+}
+
+#define kh_ptr_hash_func(key) __ac_X31_hash_ptr(key)
+#define kh_ptr_hash_equal(a, b) ((a) == (b))
+
+KHASH_INIT(ptr, void *, int, 0, kh_ptr_hash_func, kh_ptr_hash_equal)
+
+static kh_ptr_t *my_malloced;
+
+static void *my_malloc(size_t sz)
+{
+ int dummy;
+ void *ptr = malloc(sz);
+ if (ptr)
+ kh_put_ptr(my_malloced, ptr, &dummy);
+ return ptr;
+}
+
+static void *my_realloc(void *ptr, size_t sz)
+{
+ int dummy;
+ if (ptr) {
+ khiter_t pos = kh_get_ptr(my_malloced, ptr);
+ if (pos >= kh_end(my_malloced))
+ die("Was not my_malloc()ed: %p", ptr);
+ kh_del_ptr(my_malloced, pos);
+ }
+ ptr = realloc(ptr, sz);
+ if (ptr)
+ kh_put_ptr(my_malloced, ptr, &dummy);
+ return ptr;
+}
+
+static void my_free(void *ptr)
+{
+ if (ptr) {
+ khiter_t pos = kh_get_ptr(my_malloced, ptr);
+ if (pos >= kh_end(my_malloced))
+ die("Was not my_malloc()ed: %p", ptr);
+ kh_del_ptr(my_malloced, pos);
+ }
+ free(ptr);
+}
+#endif
+
static struct ref_store *reftable_be_init(struct repository *repo,
const char *gitdir,
unsigned int store_flags)
@@ -362,6 +425,11 @@ static struct ref_store *reftable_be_init(struct repository *repo,
int is_worktree;
mode_t mask;
+#ifdef DEBUG_REFTABLE_ALLOC
+ my_malloced = kh_init_ptr();
+ reftable_set_alloc(my_malloc, my_realloc, my_free);
+#endif
+
mask = umask(0);
umask(mask);
I briefly considered contributing this "custom allocator" patch, too,
but it is unwieldy (for example, it would not work at all when compiling
with mimalloc support) and it would only waste space (or even time, if a
compile flag was introduced and exercised as part of the CI builds).
Given that it is highly unlikely that Git will lose the new
`reftable_set_alloc()` call by mistake, I rejected that idea as simply
too wasteful.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The source files for libgit.a have been moved into a new "lib/"
directory to clean up the top-level directory and clearly separate
library code.
* ps/libgit-in-subdir:
Move libgit.a sources into separate "lib/" directory
t/helper: prepare "test-example-tap.c" for introduction of "lib/"
The Git project is not exactly the easiest project to get started in:
it's written in C and POSIX shell, with bits of Perl, Rust and other
languages sprinkled into it. On top of that, the project has grown
somewhat organically over time, making the codebase hard to navigate.
These are problems that we're aware of, and there have been and still
are efforts to clean up some of the technical debt that is natural to
exist an a project that is more than 20 years old. Furthermore, we
provide resources to newcomers that help them out like our coding
guidelines, code of conduct or "MyFirstContribution.adoc".
But there is a rather practical problem: finding your way around in our
project's tree is not easy. Doing a directory listing in the top-level
directory will present you with more than 550 files, which makes it
extremely hard for a newcomer to figure out what files they are even
supposed to look at. This makes the onboarding experience somewhat
harder than it really needs to be. This isn't only a problem for
newcomers though, as I myself struggle to find the files I am looking
for because of the sheer number of files.
Besides the problem of discoverability it also creates a problem of
structure. It is not obvious at all which files are part of "libgit.a"
and which files are only linked into our final executables. So while we
have this split in our build systems, that split is not evident at all
in our tree.
Introduce a new "lib/" directory and move all of our sources for
"libgit.a" into it to fix these issues. It makes the split we have
evident and reduces the number of files in our top-level tree from 550
files to ~80 files.
This is still a lot of files, but it's significantly easier to navigate
already. Furthermore, we can further iterate after this step and think
about introducing a better structure for remaining files, as well.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
