This merges the current version of the patch that tries to address Git
GUI's problems with intent-to-add files.
This patch will likely be improved substantially before it is merged
into Git GUI's main branch, but we want to have _something_ resembling a
fix already in Git for Windows v2.29.0.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This topic branch allows `add -p` and `add -i` with a large number of
files. It is kind of a hack that was never really meant to be
upstreamed. Let's see if we can do better in the built-in `add -p`.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
After releasing v2.2.6, they followed up really quickly with a v2.2.7,
which indicates some urgency in taking this update. The diff of this PR
is pretty informative, too.
This closes https://github.com/git-for-windows/git/issues/6046
In MSYS2, we have two Python interpreters at our disposal, so we can
include the Python stuff in the build.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This topic branch teaches `git clean` to respect NTFS junctions and Unix
bind mounts: it will now stop at those boundaries.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This topic branch allows us to specify absolute paths without the drive
prefix e.g. when cloning.
Example:
C:\Users\me> git clone https://github.com/git/git \upstream-git
This will clone into a new directory C:\upstream-git, in line with how
Windows interprets absolute paths.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
These fixes were necessary for Sverre Rabbelier's remote-hg to work,
but for some magic reason they are not necessary for the current
remote-hg. Makes you wonder how that one gets away with it.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
It was already decided in ef22148 (clean: do not traverse mount points,
2018-12-07) that we shouldn't traverse NTFS junctions/bind mounts when
using `git clean`, partly because they're sometimes used in worktrees.
But the same check wasn't applied to `remove_dir_recurse()` in `dir.c`,
which `git worktree remove` uses. So removing a worktree suffers the
same problem we had previously with `git clean`.
Let's add the same guard from ef22148.
Signed-off-by: Maks Kuznia <makskuznia244@gmail.com>
The previous commits disabled NTLM authentication by default due to its
cryptographic weaknesses. Users can re-enable it via the config setting
http.<url>.allowNTLMAuth, but this requires manual intervention.
Credential helpers may have knowledge about which servers are trusted
for NTLM authentication (e.g., known on-prem Azure DevOps instances).
To allow them to signal this trust, introduce a simple negotiation:
when NTLM is suppressed and the server offered it, Git advertises
ntlm=suppressed to the credential helper. The helper can respond with
ntlm=allow to re-enable NTLM for this request.
This happens precisely at the point where we would otherwise warn the
user about NTLM being suppressed, ensuring the capability is only
advertised when relevant.
Helped-by: Matthew John Cheetham <mjcheetham@outlook.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The 2.53.0.rc0.windows release candidate had a regression where
writing to stderr from a pre-push hook would error out.
The regression was fixed in 2.53.0.rc1.windows and the test here ensures
that this stays fixed.
Signed-off-by: Thomas Braun <thomas.braun@virtuell-zuhause.de>
Every once in a while, whitespace errors are introduced in Git for
Windows' rebases to newer Git versions, simply by virtue of integrating
upstream commits that do not follow upstream Git's own whitespace rule.
In Git v2.50.0-rc0, for example, 03f2915541 (xdiff: disable
cleanup_records heuristic with --minimal, 2025-04-29) introduced a
trailing space.
Arguably, non-actionable alerts are worse than no alerts at all, so
let's suppress those alerts that we cannot do anything about, anyway.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The reftable library goes out of its way to use its own set of allocator
functions that can be configured using `reftable_set_alloc()`. However,
Git does not configure this.
That is not typically a problem, except when Git uses a custom allocator
via some definitions in `git-compat-util.h`, as is the case in Git for
Windows (which switched away from the long-unmaintained nedmalloc to
mimalloc).
Then, it is quite possible that Git assigns a `strbuf` (allocated via
the custom allocator) to, say, the `refname` field of a
`reftable_log_record` in `write_transaction_table()`, and later on asks
the reftable library function `reftable_log_record_release()` to release
it, but that function was compiled without using `git-compat-util.h` and
hence calls regular `free()` (i.e. _not_ the custom allocator's own
function).
This has been a problem for a long time and it was a matter of some sort
of "luck" that 1) reftables are not commonly used on Windows, and 2)
mimalloc can often ignore gracefully when it is asked to release memory
that it has not allocated.
However, a recent update to `seen` brought this problem to the
forefront, letting t1460 fail in Git for Windows, with symptoms much in
the same way as the problem I had to address in d02c37c3e6
(t-reftable-basics: allow for `malloc` to be `#define`d, 2025-01-08)
where exit code 127 was also produced in lieu of
`STATUS_HEAP_CORRUPTION` (C0000374) because exit codes are only 7 bits
wide.
It was not possible to figure out what change in particular caused these
new failures within a reasonable time frame, as there are too many
changes in `seen` that conflict with Git for Windows' patches, I had to
stop the investigation after spending four hours on it fruitlessly.
To verify that this patch fixes the issue, I avoided using mimalloc and
temporarily patched in a "custom allocator" that would more reliably
point out problems, like this:
diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c
index 68f38291f84c..9421d630b9f5 100644
--- a/refs/reftable-backend.c
+++ b/refs/reftable-backend.c
@@ -353,6 +353,69 @@ static int reftable_be_fsync(int fd)
return fsync_component(FSYNC_COMPONENT_REFERENCE, fd);
}
+#define DEBUG_REFTABLE_ALLOC
+#ifdef DEBUG_REFTABLE_ALLOC
+#include "khash.h"
+
+static inline khint_t __ac_X31_hash_ptr(void *ptr)
+{
+ union {
+ void *ptr;
+ char s[sizeof(void *)];
+ } u;
+ size_t i;
+ khint_t h;
+
+ u.ptr = ptr;
+ h = (khint_t)*u.s;
+ for (i = 0; i < sizeof(void *); i++)
+ h = (h << 5) - h + (khint_t)u.s[i];
+ return h;
+}
+
+#define kh_ptr_hash_func(key) __ac_X31_hash_ptr(key)
+#define kh_ptr_hash_equal(a, b) ((a) == (b))
+
+KHASH_INIT(ptr, void *, int, 0, kh_ptr_hash_func, kh_ptr_hash_equal)
+
+static kh_ptr_t *my_malloced;
+
+static void *my_malloc(size_t sz)
+{
+ int dummy;
+ void *ptr = malloc(sz);
+ if (ptr)
+ kh_put_ptr(my_malloced, ptr, &dummy);
+ return ptr;
+}
+
+static void *my_realloc(void *ptr, size_t sz)
+{
+ int dummy;
+ if (ptr) {
+ khiter_t pos = kh_get_ptr(my_malloced, ptr);
+ if (pos >= kh_end(my_malloced))
+ die("Was not my_malloc()ed: %p", ptr);
+ kh_del_ptr(my_malloced, pos);
+ }
+ ptr = realloc(ptr, sz);
+ if (ptr)
+ kh_put_ptr(my_malloced, ptr, &dummy);
+ return ptr;
+}
+
+static void my_free(void *ptr)
+{
+ if (ptr) {
+ khiter_t pos = kh_get_ptr(my_malloced, ptr);
+ if (pos >= kh_end(my_malloced))
+ die("Was not my_malloc()ed: %p", ptr);
+ kh_del_ptr(my_malloced, pos);
+ }
+ free(ptr);
+}
+#endif
+
static struct ref_store *reftable_be_init(struct repository *repo,
const char *gitdir,
unsigned int store_flags)
@@ -362,6 +425,11 @@ static struct ref_store *reftable_be_init(struct repository *repo,
int is_worktree;
mode_t mask;
+#ifdef DEBUG_REFTABLE_ALLOC
+ my_malloced = kh_init_ptr();
+ reftable_set_alloc(my_malloc, my_realloc, my_free);
+#endif
+
mask = umask(0);
umask(mask);
I briefly considered contributing this "custom allocator" patch, too,
but it is unwieldy (for example, it would not work at all when compiling
with mimalloc support) and it would only waste space (or even time, if a
compile flag was introduced and exercised as part of the CI builds).
Given that it is highly unlikely that Git will lose the new
`reftable_set_alloc()` call by mistake, I rejected that idea as simply
too wasteful.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The new default of Git is to disable NTLM authentication by default.
To help users find the escape hatch of that config setting, should they
need it, suggest it when the authentication failed and the server had
offered NTLM, i.e. if re-enabling it would fix the problem.
Helped-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Commit 2406bf5 (Win32: detect unix socket support at runtime,
2024-04-03) introduced a runtime detection for whether the operating
system supports unix sockets for Windows, but a mistake snuck into the
tests. When building and testing Git without NO_UNIX_SOCKETS we
currently skip t0301-credential-cache on Windows if unix sockets are
supported and run the tests if they aren't.
Flip that logic to actually work the way it was intended.
Signed-off-by: Matthias Aßhauer <mha1993@live.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
NTLM authentication is relatively weak. This is the case even with the
default setting of modern Windows versions, where NTLMv1 and LanManager
are disabled and only NTLMv2 is enabled: NTLMv2 hashes of even
reasonably complex 8-character passwords can be broken in a matter of
days, given enough compute resources.
Even worse: On Windows, NTLM authentication uses Security Support
Provider Interface ("SSPI"), which provides the credentials without
requiring the user to type them in.
Which means that an attacker could talk an unsuspecting user into
cloning from a server that is under the attacker's control and extracts
the user's NTLMv2 hash without their knowledge.
For that reason, let's disallow NTLM authentication by default.
NTLM authentication is quite simple to set up, though, and therefore
there are still some on-prem Azure DevOps setups out there whose users
and/or automation rely on this type of authentication. To give them an
escape hatch, introduce the `http.<url>.allowNTLMAuth` config setting
that can be set to `true` to opt back into using NTLM for a specific
remote repository.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Although NTLM authentication is considered weak (extending even to
NTLMv2, which purportedly allows brute-forcing reasonably complex
8-character passwords in a matter of days, given ample compute
resources), it _is_ one of the authentication methods supported by
libcurl.
Note: The added test case *cannot* reuse the existing `custom_auth`
facility. The reason is that that facility is backed by an NPH script
("No Parse Headers"), which does not allow handling the 3-phase NTLM
authentication correctly (in my hands, the NPH script would not even be
called upon the Type 3 message, a "200 OK" would be returned, but no
headers, let alone the `git http-backend` output as payload). Having a
separate NTLM authentication script makes the exact workings clearer and
more readable, anyway.
Co-authored-by: Matthew John Cheetham <mjcheetham@outlook.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
As reported in https://lore.kernel.org/git/ZuPKvYP9ZZ2mhb4m@pks.im/,
libcurl v8.10.0 had a regression that was picked up by Git's t5559.30
"large fetch-pack requests can be sent using chunked encoding".
This bug was fixed in libcurl v8.10.1.
Sadly, the macos-13 runner image was updated in the brief window between
these two libcurl versions, breaking each and every CI build, as
reported at https://github.com/git-for-windows/git/issues/5159.
This would usually not matter, we would just ignore the failing CI
builds until the macos-13 runner image is rebuilt in a couple of days,
and then the CI builds would succeed again.
However.
As has become the custom, a surprise Git version was released, and now
that Git for Windows wants to follow suit, since Git for Windows has
this custom of trying to never release a version with a failing CI
build, we _must_ work around it.
This patch implements this work-around, basically for the sake of Git
for Windows v2.46.2's CI build.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
While this command is definitely something we _want_, chances are that
upstreaming this will require substantial changes.
We still want to be able to experiment with this before that, to focus
on what we need out of this command: To assist with diagnosing issues
with large repositories, as well as to help monitoring the growth and
the associated painpoints of such repositories.
To that end, we are about to integrate this command into
`microsoft/git`, to get the tool into the hands of users who need it
most, with the idea to iterate in close collaboration between these
users and the developers familar with Git's internals.
However, we will definitely want to avoid letting anybody have the
impression that this command, its exact inner workings, as well as its
output format, are anywhere close to stable. To make that fact utterly
clear (and thereby protect the freedom to iterate and innovate freely
before upstreaming the command), let's mark its output as experimental
in all-caps, as the first thing we do.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The sparse tree walk algorithm was created in d5d2e93577 (revision:
implement sparse algorithm, 2019-01-16) and involves using the
mark_trees_uninteresting_sparse() method. This method takes a repository
and an oidset of tree IDs, some of which have the UNINTERESTING flag and
some of which do not.
Create a method that has an equivalent set of preconditions but uses a
"dense" walk (recursively visits all reachable trees, as long as they
have not previously been marked UNINTERESTING). This is an important
difference from mark_tree_uninteresting(), which short-circuits if the
given tree has the UNINTERESTING flag.
A use of this method will be added in a later change, with a condition
set whether the sparse or dense approach should be used.
Signed-off-by: Derrick Stolee <stolee@gmail.com>
This comment has been true for the longest time; The combination of the
two preceding commits made it incorrect, so let's drop that comment.
Signed-off-by: Matthias Aßhauer <mha1993@live.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The 'git survey' builtin provides several detail tables, such as "top
files by on-disk size". The size of these tables defaults to 10,
currently.
Allow the user to specify this number via a new --top=<N> option or the
new survey.top config key.
Signed-off-by: Derrick Stolee <stolee@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We map WSAGetLastError() errors to errno errors in winsock_error_to_errno(),
but the MSVC strerror() implementation only produces "Unknown error" for
most of them. Produce some more meaningful error messages in these
cases.
Our builds for ARM64 link against the newer UCRT strerror() that does know
these errors, so we won't change the strerror() used there.
The wording of the messages is copied from glibc strerror() messages.
Reported-by: M Hickford <mirth.hickford@gmail.com>
Signed-off-by: Matthias Aßhauer <mha1993@live.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Since we are already walking our reachable objects using the path-walk API,
let's now collect lists of the paths that contribute most to different
metrics. Specifically, we care about
* Number of versions.
* Total size on disk.
* Total inflated size (no delta or zlib compression).
This information can be critical to discovering which parts of the
repository are causing the most growth, especially on-disk size. Different
packing strategies might help compress data more efficiently, but the toal
inflated size is a representation of the raw size of all snapshots of those
paths. Even when stored efficiently on disk, that size represents how much
information must be processed to complete a command such as 'git blame'.
The exact disk size seems to be not quite robust enough for testing, as
could be seen by the `linux-musl-meson` job consistently failing, possibly
because of zlib-ng deflates differently: t8100.4(git survey
(default)) was failing with a symptom like this:
TOTAL OBJECT SIZES BY TYPE
===============================================
Object Type | Count | Disk Size | Inflated Size
------------+-------+-----------+--------------
- Commits | 10 | 1523 | 2153
+ Commits | 10 | 1528 | 2153
Trees | 10 | 495 | 1706
Blobs | 10 | 191 | 101
- Tags | 4 | 510 | 528
+ Tags | 4 | 547 | 528
This means: the disk size is unlikely something we can verify robustly.
Since zlib-ng seems to increase the disk size of the tags from 528 to
547, we cannot even assume that the disk size is always smaller than the
inflated size. We will most likely want to either skip verifying the
disk size altogether, or go for some kind of fuzzy matching, say, by
replacing `s/ 1[45][0-9][0-9] / ~1.5k /` and `s/ [45][0-9][0-9] / ~½k /`
or something like that.
Signed-off-by: Derrick Stolee <stolee@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The winsock2 library provides functions that work on different data
types than file descriptors, therefore we wrap them.
But that is not the only difference: they also do not set `errno` but
expect the callers to enquire about errors via `WSAGetLastError()`.
Let's translate that into appropriate `errno` values whenever the socket
operations fail so that Git's code base does not have to change its
expectations.
This closes https://github.com/git-for-windows/git/issues/2404
Helped-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
In future changes, we will make use of these methods. The intention is to
keep track of the top contributors according to some metric. We don't want
to store all of the entries and do a sort at the end, so track a
constant-size table and remove rows that get pushed out depending on the
chosen sorting algorithm.
Co-authored-by: Jeff Hostetler <git@jeffhostetler.com>
Signed-off-by; Jeff Hostetler <git@jeffhostetler.com>
Signed-off-by: Derrick Stolee <stolee@gmail.com>
Now that we have explored objects by count, we can expand that a bit more to
summarize the data for the on-disk and inflated size of those objects. This
information is helpful for diagnosing both why disk space (and perhaps
clone or fetch times) is growing but also why certain operations are slow
because the inflated size of the abstract objects that must be processed is
so large.
Note: zlib-ng is slightly more efficient even at those small sizes. Even
between zlib versions, there are slight differences in compression. To
accommodate for that in the tests, not the exact numbers but some rough
approximations are validated (the test should validate `git survey`,
after all, not zlib).
Signed-off-by: Derrick Stolee <stolee@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
At the moment, nothing is obvious about the reason for the use of the
path-walk API, but this will become more prevelant in future iterations. For
now, use the path-walk API to sum up the counts of each kind of object.
For example, this is the reachable object summary output for my local repo:
REACHABLE OBJECT SUMMARY
========================
Object Type | Count
------------+-------
Tags | 1343
Commits | 179344
Trees | 314350
Blobs | 184030
Signed-off-by: Derrick Stolee <stolee@gmail.com>
When 'git survey' provides information to the user, this will be presented
in one of two formats: plaintext and JSON. The JSON implementation will be
delayed until the functionality is complete for the plaintext format.
The most important parts of the plaintext format are headers specifying the
different sections of the report and tables providing concreted data.
Create a custom table data structure that allows specifying a list of
strings for the row values. When printing the table, check each column for
the maximum width so we can create a table of the correct size from the
start.
The table structure is designed to be flexible to the different kinds of
output that will be implemented in future changes.
Signed-off-by: Derrick Stolee <stolee@gmail.com>
By default we will scan all references in "refs/heads/", "refs/tags/"
and "refs/remotes/".
Add command line opts let the use ask for all refs or a subset of them
and to include a detached HEAD.
Signed-off-by: Jeff Hostetler <git@jeffhostetler.com>
Signed-off-by: Derrick Stolee <stolee@gmail.com>
Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Start work on a new 'git survey' command to scan the repository
for monorepo performance and scaling problems. The goal is to
measure the various known "dimensions of scale" and serve as a
foundation for adding additional measurements as we learn more
about Git monorepo scaling problems.
The initial goal is to complement the scanning and analysis performed
by the GO-based 'git-sizer' (https://github.com/github/git-sizer) tool.
It is hoped that by creating a builtin command, we may be able to take
advantage of internal Git data structures and code that is not
accessible from GO to gain further insight into potential scaling
problems.
Co-authored-by: Derrick Stolee <stolee@gmail.com>
Signed-off-by: Jeff Hostetler <git@jeffhostetler.com>
Signed-off-by: Derrick Stolee <stolee@gmail.com>
Git LFS is now built with Go 1.21 which no longer supports Windows 7.
However, Git for Windows still wants to support Windows 7.
Ideally, Git LFS would re-introduce Windows 7 support until Git for
Windows drops support for Windows 7, but that's not going to happen:
https://github.com/git-for-windows/git/issues/4996#issuecomment-2176152565
The next best thing we can do is to let the users know what is
happening, and how to get out of their fix, at least.
This is not quite as easy as it would first seem because programs
compiled with Go 1.21 or newer will simply throw an exception and fail
with an Access Violation on Windows 7.
The only way I found to address this is to replicate the logic from Go's
very own `version` command (which can determine the Go version with
which a given executable was built) to detect the situation, and in that
case offer a helpful error message.
This addresses https://github.com/git-for-windows/git/issues/4996.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
In order to be a better Windows citizenship, Git should
save its configuration files on AppData folder. This can
enables git configuration files be replicated between machines
using the same Microsoft account logon which would reduce the
friction of setting up Git on new systems. Therefore, if
%APPDATA%\Git\config exists, we use it; otherwise
$HOME/.config/git/config is used.
Signed-off-by: Ariel Lourenco <ariellourenco@users.noreply.github.com>
The `__MINGW64__` constant is defined, surprise, surprise, only when
building for a 64-bit CPU architecture.
Therefore using it as a guard to define `_POSIX_C_SOURCE` (so that
`localtime_r()` is declared, among other functions) is not enough, we
also need to check `__MINGW32__`.
Technically, the latter constant is defined even for 64-bit builds. But
let's make things a bit easier to understand by testing for both
constants.
Making it so fixes this compile warning (turned error in GCC v14.1):
archive-zip.c: In function 'dos_time':
archive-zip.c:612:9: error: implicit declaration of function 'localtime_r';
did you mean 'localtime_s'? [-Wimplicit-function-declaration]
612 | localtime_r(&time, &tm);
| ^~~~~~~~~~~
| localtime_s
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
A long time ago, we decided to run tests in Git for Windows' SDK with
the default `winsymlinks` mode: copying instead of linking. This is
still the default mode of MSYS2 to this day.
However, this is not how most users run Git for Windows: As the majority
of Git for Windows' users seem to be on Windows 10 and newer, likely
having enabled Developer Mode (which allows creating symbolic links
without administrator privileges), they will run with symlink support
enabled.
This is the reason why it is crucial to get the fixes for CVE-2024-? to
the users, and also why it is crucial to ensure that the test suite
exercises the related test cases. This commit ensures the latter.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
