This introduces `git survey` to Git for Windows ahead of upstream for
the express purpose of getting the path-based analysis in the hands of
more folks.
The inspiration of this builtin is
[`git-sizer`](https://github.com/github/git-sizer), but since that
command relies on `git cat-file --batch` to get the contents of objects,
it has limits to how much information it can provide.
This is mostly a rewrite of the `git survey` builtin that was introduced
into the `microsoft/git` fork in microsoft/git#667. That version had a
lot more bells and whistles, including an analysis much closer to what
`git-sizer` provides.
The biggest difference in this version is that this one is focused on
using the path-walk API in order to visit batches of objects based on a
common path. This allows identifying, for instance, the path that is
contributing the most to the on-disk size across all versions at that
path.
For example, here are the top ten paths contributing to my local Git
repository (which includes `microsoft/git` and `gitster/git`):
```
TOP FILES BY DISK SIZE
============================================================================
Path | Count | Disk Size | Inflated Size
-----------------------------------------+-------+-----------+--------------
whats-cooking.txt | 1373 | 11637459 | 37226854
t/helper/test-gvfs-protocol | 2 | 6847105 | 17233072
git-rebase--helper | 1 | 6027849 | 15269664
compat/mingw.c | 6111 | 5194453 | 463466970
t/helper/test-parse-options | 1 | 3420385 | 8807968
t/helper/test-pkt-line | 1 | 3408661 | 8778960
t/helper/test-dump-untracked-cache | 1 | 3408645 | 8780816
t/helper/test-dump-fsmonitor | 1 | 3406639 | 8776656
po/vi.po | 104 | 1376337 | 51441603
po/de.po | 210 | 1360112 | 71198603
```
This kind of analysis has been helpful in identifying the reasons for
growth in a few internal monorepos. Those findings motivated the changes
in #5157 and #5171.
With this early version in Git for Windows, we can expand the reach of
the experimental tool in advance of it being contributed to the upstream
project.
Unfortunately, this will mean that in the next `microsoft/git` rebase,
Jeff Hostetler's version will need to be pulled out since there are
enough conflicts. These conflicts include how tables are stored and
generated, as the version in this PR is slightly more general to allow
for different kinds of data.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This is a follow up to #5157 as well as motivated by the RFC in
gitgitgadget/git#1786.
We have ways of walking all objects, but it is focused on visiting a
single commit and then expanding the new trees and blobs reachable from
that commit that have not been visited yet. This means that objects
arrive without any locality based on their path.
Add a new "path walk API" that focuses on walking objects in batches
according to their type and path. This will walk all annotated tags, all
commits, all root trees, and then start a depth-first search among all
paths in the repo to collect trees and blobs in batches.
The most important application for this is being fast-tracked to Git for
Windows: `git pack-objects --path-walk`. This application of the path
walk API discovers the objects to pack via this batched walk, and
automatically groups objects that appear at a common path so they can be
checked for delta comparisons.
This use completely avoids any name-hash collisions (even the collisions
that sometimes occur with the new `--full-name-hash` option) and can be
much faster to compute since the first pass of delta calculations does
not waste time on objects that are unlikely to be diffable.
Some statistics are available in the commit messages.
Since Git LFS v3.5.x implicitly dropped Windows 7 support, we now want
users to be advised _what_ is going wrong on that Windows version. This
topic branch goes out of its way to provide users with such guidance.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This topic branch adds support for a more Windows-native user-wide
config file than `XDG_CONFIG_HOME` (or `~/.config/`) will ever be.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This topic branch contains a patch that made it into Git for Windows
v2.45.1 but not into Git v2.45.1 (because the latter does not come with
symlink support on Windows).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Another (hopefully clean) PR for showing the error warning about atomic
append on windows after failure on APFS, which returns EBADF not EINVAL.
Signed-off-by: David Lomas <dl3@pale-eds.co.uk>
Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
This patch adds a GitHub workflow (to be triggered manually) to allow
for conveniently verifying that Git and Scalar still work as intended in
Windows Nano Server (a relatively small container base image that is
frequently used where a "small Windows" is needed, e.g. in automation
;-))
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
As per
https://github.com/git-for-windows/git/issues/4350#issuecomment-1485041503,
the major block for upgrading Git for Windows' OpenSSL from v1.1 to v3
is the tricky part where such an upgrade would break `git fetch`/`git
clone` and `git push` because the libcurl depends on the OpenSSL DLL,
and the major version bump will _change_ the file name of said DLL.
To overcome that, the plan is to build libcurl flavors for each
supported SSL/TLS backend, aligning with the way MSYS2 builds libcurl,
then switch Git for Windows' SDK to the Secure Channel-flavored libcurl,
and teach Git to look for the specific flavor of libcurl corresponding
to the `http.sslBackend` setting (if that was configured).
Here is the PR to teach Git that trick.
Every once in a while, there are bug reports in Git for Windows' bug
tracker that describe an issue running [inside MSYS2
proper](https://gitforwindows.org/install-inside-msys2-proper), totally
ignoring the big, honking warning on top of [the
page](https://gitforwindows.org/install-inside-msys2-proper) that spells
out clearly that this is an unsupported use case.
At the same time, we cannot easily deflect and say "just use MSYS2
directly" (and leave the "and stop pestering us" out). We cannot do that
because there is only an _MSYS_ `git` package in MSYS2 (i.e. a Git that
uses the quite slow POSIX emulation layer provided by the MSYS2
runtime), but no `mingw-w64-git` package (which would be equivalent in
speed to Git for Windows).
In https://github.com/msys2/MINGW-packages/pull/26470, I am preparing to
change that. As part of that PR, I noticed and fixed a couple of issues
_in `git-for-windows/git` that prevented full support for
`mingw-w64-git` in MSYS2, such as problems with CLANG64 and UCRT64.
While at it, I simplified the entire setup to trust MSYS2's
`MINGW_PREFIX` & related environment variables instead of hard-coding
values like the installation prefix and what `MSYSTEM` to fall back on
if it is unset.
It is merely a historical wart that, say, `git-commit` exists in the
`libexec/git-core/` directory, a tribute to the original idea to let Git
be essentially a bunch of Unix shell scripts revolving around very few
"plumbing" (AKA low-level) commands.
Git has evolved a lot from there. These days, most of Git's
functionality is contained within the `git` executable, in the form of
"built-in" commands.
To accommodate for scripts that use the "dashed" form of Git commands,
even today, Git provides hard-links that make the `git` executable
available as, say, `git-commit`, just in case that an old script has not
been updated to invoke `git commit`.
Those hard-links do not come cheap: they take about half a minute for
every build of Git on Windows, they are mistaken for taking up huge
amounts of space by some Windows Explorer versions that do not
understand hard-links, and therefore many a "bug" report had to be
addressed.
The "dashed form" has been officially deprecated in Git version 1.5.4,
which was released on February 2nd, 2008, i.e. a very long time ago.
This deprecation was never finalized by skipping these hard-links, but
we can start the process now, in Git for Windows.
This addresses the concern raised in
https://github.com/git-for-windows/git/pull/4185#discussion_r1051661894
It is checked for w.r.t. global repository struct down in the callstack
in compatibility layer for MinGW before being assigned in the function
that `free()`'d it.
Add `FileVersion`, which is a required string ([Microsoft
documentation](https://learn.microsoft.com/en-us/windows/win32/menurc/versioninfo-resource))
in the `StringFileInfo` block.
As not all required strings were present in the block, none were being
included.
Fixes#4090
After including the `FileVersion` string, all other defined strings are
now being included on executables.
File version information for `git.exe` has changed from:
```
PS C:\Program Files\Git\bin> [System.Diagnostics.FileVersionInfo]::GetVersionInfo("C:\Data\git-sdk-64\usr\src\git\git.exe") | Select-Object *
FileVersionRaw : 2.38.1.1
ProductVersionRaw : 2.38.1.1
Comments :
CompanyName :
FileBuildPart : 1
FileDescription :
FileMajorPart : 2
FileMinorPart : 38
FileName : C:\Data\git-sdk-64\usr\src\git\git.exe
FilePrivatePart : 1
FileVersion :
InternalName :
IsDebug : False
IsPatched : False
IsPrivateBuild : False
IsPreRelease : False
IsSpecialBuild : False
Language : English (United States)
LegalCopyright :
LegalTrademarks :
OriginalFilename :
PrivateBuild :
ProductBuildPart : 1
ProductMajorPart : 2
ProductMinorPart : 38
ProductName :
ProductPrivatePart : 1
ProductVersion :
SpecialBuild :
```
To the following:
```
PS C:\Program Files\Git\bin> [System.Diagnostics.FileVersionInfo]::GetVersionInfo("C:\Data\git-sdk-64\usr\src\git\git.exe") | Select-Object *
FileVersionRaw : 2.38.1.1
ProductVersionRaw : 2.38.1.1
Comments :
CompanyName : The Git Development Community
FileBuildPart : 1
FileDescription : Git for Windows
FileMajorPart : 2
FileMinorPart : 38
FileName : C:\Data\git-sdk-64\usr\src\git\git.exe
FilePrivatePart : 1
FileVersion : 2.38.1.windows.1.10.g6ed65a6fab
InternalName : git
IsDebug : False
IsPatched : False
IsPrivateBuild : False
IsPreRelease : False
IsSpecialBuild : False
Language : English (United States)
LegalCopyright :
LegalTrademarks :
OriginalFilename : git.exe
PrivateBuild :
ProductBuildPart : 1
ProductMajorPart : 2
ProductMinorPart : 38
ProductName : Git
ProductPrivatePart : 1
ProductVersion : 2.38.1.windows.1.10.g6ed65a6fab
SpecialBuild :
```
I wasn't really expecting `GIT_VERSION` to contain the Git commit, I was
hoping for just `2.38.1` or `2.38.1.1`, at least for the `FileVersion`
string.
Anybody know if it's possible to concatenate the `MAJOR`, `MINOR`,
`MICRO`, and `PATCHLEVEL` fields with dots, or if there's another
variable that can be used (with or without `PATCHLEVEL`)?
Alternatively, use the complete `GIT_VERSION` for both `FileVersion` and
`ProductVersion`.
The first three commits are rebased versions of those in gitgitgadget/git#1215. These allow the following:
1. Fix `git config --global foo.bar <path>` from allowing the `<path>`. As a bonus, users with a config value starting with `/` will not get a warning about "old-style" paths needing a "`%(prefix)/`".
2. When in WSL, the path starts with `/` so it needs to be interpolated properly. Update the warning to include `%(prefix)/` to get the right value for WSL users. (This is specifically for using Git for Windows from Git Bash, but in a WSL directory.)
3. When using WSL, the ownership check fails and reports an error message. This is noisy, and happens even if the user has marked the path with `safe.directory`. Remove that error message.
This merges the current version of the patch that tries to address Git
GUI's problems with intent-to-add files.
This patch will likely be improved substantially before it is merged
into Git GUI's main branch, but we want to have _something_ resembling a
fix already in Git for Windows v2.29.0.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This topic branch allows `add -p` and `add -i` with a large number of
files. It is kind of a hack that was never really meant to be
upstreamed. Let's see if we can do better in the built-in `add -p`.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
In MSYS2, we have two Python interpreters at our disposal, so we can
include the Python stuff in the build.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This topic branch allows us to specify absolute paths without the drive
prefix e.g. when cloning.
Example:
C:\Users\me> git clone https://github.com/git/git \upstream-git
This will clone into a new directory C:\upstream-git, in line with how
Windows interprets absolute paths.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
These fixes were necessary for Sverre Rabbelier's remote-hg to work,
but for some magic reason they are not necessary for the current
remote-hg. Makes you wonder how that one gets away with it.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The reftable library goes out of its way to use its own set of allocator
functions that can be configured using `reftable_set_alloc()`. However,
Git does not configure this.
That is not typically a problem, except when Git uses a custom allocator
via some definitions in `git-compat-util.h`, as is the case in Git for
Windows (which switched away from the long-unmaintained nedmalloc to
mimalloc).
Then, it is quite possible that Git assigns a `strbuf` (allocated via
the custom allocator) to, say, the `refname` field of a
`reftable_log_record` in `write_transaction_table()`, and later on asks
the reftable library function `reftable_log_record_release()` to release
it, but that function was compiled without using `git-compat-util.h` and
hence calls regular `free()` (i.e. _not_ the custom allocator's own
function).
This has been a problem for a long time and it was a matter of some sort
of "luck" that 1) reftables are not commonly used on Windows, and 2)
mimalloc can often ignore gracefully when it is asked to release memory
that it has not allocated.
However, a recent update to `seen` brought this problem to the
forefront, letting t1460 fail in Git for Windows, with symptoms much in
the same way as the problem I had to address in d02c37c3e6
(t-reftable-basics: allow for `malloc` to be `#define`d, 2025-01-08)
where exit code 127 was also produced in lieu of
`STATUS_HEAP_CORRUPTION` (C0000374) because exit codes are only 7 bits
wide.
It was not possible to figure out what change in particular caused these
new failures within a reasonable time frame, as there are too many
changes in `seen` that conflict with Git for Windows' patches, I had to
stop the investigation after spending four hours on it fruitlessly.
To verify that this patch fixes the issue, I avoided using mimalloc and
temporarily patched in a "custom allocator" that would more reliably
point out problems, like this:
diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c
index 68f38291f84c..9421d630b9f5 100644
--- a/refs/reftable-backend.c
+++ b/refs/reftable-backend.c
@@ -353,6 +353,69 @@ static int reftable_be_fsync(int fd)
return fsync_component(FSYNC_COMPONENT_REFERENCE, fd);
}
+#define DEBUG_REFTABLE_ALLOC
+#ifdef DEBUG_REFTABLE_ALLOC
+#include "khash.h"
+
+static inline khint_t __ac_X31_hash_ptr(void *ptr)
+{
+ union {
+ void *ptr;
+ char s[sizeof(void *)];
+ } u;
+ size_t i;
+ khint_t h;
+
+ u.ptr = ptr;
+ h = (khint_t)*u.s;
+ for (i = 0; i < sizeof(void *); i++)
+ h = (h << 5) - h + (khint_t)u.s[i];
+ return h;
+}
+
+#define kh_ptr_hash_func(key) __ac_X31_hash_ptr(key)
+#define kh_ptr_hash_equal(a, b) ((a) == (b))
+
+KHASH_INIT(ptr, void *, int, 0, kh_ptr_hash_func, kh_ptr_hash_equal)
+
+static kh_ptr_t *my_malloced;
+
+static void *my_malloc(size_t sz)
+{
+ int dummy;
+ void *ptr = malloc(sz);
+ if (ptr)
+ kh_put_ptr(my_malloced, ptr, &dummy);
+ return ptr;
+}
+
+static void *my_realloc(void *ptr, size_t sz)
+{
+ int dummy;
+ if (ptr) {
+ khiter_t pos = kh_get_ptr(my_malloced, ptr);
+ if (pos >= kh_end(my_malloced))
+ die("Was not my_malloc()ed: %p", ptr);
+ kh_del_ptr(my_malloced, pos);
+ }
+ ptr = realloc(ptr, sz);
+ if (ptr)
+ kh_put_ptr(my_malloced, ptr, &dummy);
+ return ptr;
+}
+
+static void my_free(void *ptr)
+{
+ if (ptr) {
+ khiter_t pos = kh_get_ptr(my_malloced, ptr);
+ if (pos >= kh_end(my_malloced))
+ die("Was not my_malloc()ed: %p", ptr);
+ kh_del_ptr(my_malloced, pos);
+ }
+ free(ptr);
+}
+#endif
+
static struct ref_store *reftable_be_init(struct repository *repo,
const char *gitdir,
unsigned int store_flags)
@@ -362,6 +425,11 @@ static struct ref_store *reftable_be_init(struct repository *repo,
int is_worktree;
mode_t mask;
+#ifdef DEBUG_REFTABLE_ALLOC
+ my_malloced = kh_init_ptr();
+ reftable_set_alloc(my_malloc, my_realloc, my_free);
+#endif
+
mask = umask(0);
umask(mask);
I briefly considered contributing this "custom allocator" patch, too,
but it is unwieldy (for example, it would not work at all when compiling
with mimalloc support) and it would only waste space (or even time, if a
compile flag was introduced and exercised as part of the CI builds).
Given that it is highly unlikely that Git will lose the new
`reftable_set_alloc()` call by mistake, I rejected that idea as simply
too wasteful.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
