Since we are already walking our reachable objects using the path-walk API,
let's now collect lists of the paths that contribute most to different
metrics. Specifically, we care about
* Number of versions.
* Total size on disk.
* Total inflated size (no delta or zlib compression).
This information can be critical to discovering which parts of the
repository are causing the most growth, especially on-disk size. Different
packing strategies might help compress data more efficiently, but the toal
inflated size is a representation of the raw size of all snapshots of those
paths. Even when stored efficiently on disk, that size represents how much
information must be processed to complete a command such as 'git blame'.
The exact disk size seems to be not quite robust enough for testing, as
could be seen by the `linux-musl-meson` job consistently failing, possibly
because of zlib-ng deflates differently: t8100.4(git survey
(default)) was failing with a symptom like this:
TOTAL OBJECT SIZES BY TYPE
===============================================
Object Type | Count | Disk Size | Inflated Size
------------+-------+-----------+--------------
- Commits | 10 | 1523 | 2153
+ Commits | 10 | 1528 | 2153
Trees | 10 | 495 | 1706
Blobs | 10 | 191 | 101
- Tags | 4 | 510 | 528
+ Tags | 4 | 547 | 528
This means: the disk size is unlikely something we can verify robustly.
Since zlib-ng seems to increase the disk size of the tags from 528 to
547, we cannot even assume that the disk size is always smaller than the
inflated size. We will most likely want to either skip verifying the
disk size altogether, or go for some kind of fuzzy matching, say, by
replacing `s/ 1[45][0-9][0-9] / ~1.5k /` and `s/ [45][0-9][0-9] / ~½k /`
or something like that.
Signed-off-by: Derrick Stolee <stolee@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
In future changes, we will make use of these methods. The intention is to
keep track of the top contributors according to some metric. We don't want
to store all of the entries and do a sort at the end, so track a
constant-size table and remove rows that get pushed out depending on the
chosen sorting algorithm.
Co-authored-by: Jeff Hostetler <git@jeffhostetler.com>
Signed-off-by; Jeff Hostetler <git@jeffhostetler.com>
Signed-off-by: Derrick Stolee <stolee@gmail.com>
Now that we have explored objects by count, we can expand that a bit more to
summarize the data for the on-disk and inflated size of those objects. This
information is helpful for diagnosing both why disk space (and perhaps
clone or fetch times) is growing but also why certain operations are slow
because the inflated size of the abstract objects that must be processed is
so large.
Note: zlib-ng is slightly more efficient even at those small sizes. Even
between zlib versions, there are slight differences in compression. To
accommodate for that in the tests, not the exact numbers but some rough
approximations are validated (the test should validate `git survey`,
after all, not zlib).
Signed-off-by: Derrick Stolee <stolee@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
At the moment, nothing is obvious about the reason for the use of the
path-walk API, but this will become more prevelant in future iterations. For
now, use the path-walk API to sum up the counts of each kind of object.
For example, this is the reachable object summary output for my local repo:
REACHABLE OBJECT SUMMARY
========================
Object Type | Count
------------+-------
Tags | 1343
Commits | 179344
Trees | 314350
Blobs | 184030
Signed-off-by: Derrick Stolee <stolee@gmail.com>
When 'git survey' provides information to the user, this will be presented
in one of two formats: plaintext and JSON. The JSON implementation will be
delayed until the functionality is complete for the plaintext format.
The most important parts of the plaintext format are headers specifying the
different sections of the report and tables providing concreted data.
Create a custom table data structure that allows specifying a list of
strings for the row values. When printing the table, check each column for
the maximum width so we can create a table of the correct size from the
start.
The table structure is designed to be flexible to the different kinds of
output that will be implemented in future changes.
Signed-off-by: Derrick Stolee <stolee@gmail.com>
By default we will scan all references in "refs/heads/", "refs/tags/"
and "refs/remotes/".
Add command line opts let the use ask for all refs or a subset of them
and to include a detached HEAD.
Signed-off-by: Jeff Hostetler <git@jeffhostetler.com>
Signed-off-by: Derrick Stolee <stolee@gmail.com>
Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Start work on a new 'git survey' command to scan the repository
for monorepo performance and scaling problems. The goal is to
measure the various known "dimensions of scale" and serve as a
foundation for adding additional measurements as we learn more
about Git monorepo scaling problems.
The initial goal is to complement the scanning and analysis performed
by the GO-based 'git-sizer' (https://github.com/github/git-sizer) tool.
It is hoped that by creating a builtin command, we may be able to take
advantage of internal Git data structures and code that is not
accessible from GO to gain further insight into potential scaling
problems.
Co-authored-by: Derrick Stolee <stolee@gmail.com>
Signed-off-by: Jeff Hostetler <git@jeffhostetler.com>
Signed-off-by: Derrick Stolee <stolee@gmail.com>
The sparse tree walk algorithm was created in d5d2e93577 (revision:
implement sparse algorithm, 2019-01-16) and involves using the
mark_trees_uninteresting_sparse() method. This method takes a repository
and an oidset of tree IDs, some of which have the UNINTERESTING flag and
some of which do not.
Create a method that has an equivalent set of preconditions but uses a
"dense" walk (recursively visits all reachable trees, as long as they
have not previously been marked UNINTERESTING). This is an important
difference from mark_tree_uninteresting(), which short-circuits if the
given tree has the UNINTERESTING flag.
A use of this method will be added in a later change, with a condition
set whether the sparse or dense approach should be used.
Signed-off-by: Derrick Stolee <stolee@gmail.com>
These fixes have been sent to the Git mailing list but have not been
picked up by the Git project yet.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This is hidden in v2.55.0-rc0's own CI because of an omission in
5ba82911bc (ci: enable EXPENSIVE for contributor builds, 2026-05-11)
which fails to enable EXPENSIVE tests for tags.
Due to 7d78d5fc1a (ci: skip GitHub workflow runs for already-tested
commits/trees, 2020-10-08), the CI of `master` is now also mistakenly
green because it reuses the tag's CI run to prove that it's solid.
This is an evil merge by necessity because `survey.c` needs to adapt to
the changed function signatures.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When a Unix socket is initialized, the current directory's path is
stored so that the cleanup code can `chdir()` back to where it was
before exit.
If the path that needs to be stored exceeds the default size of the
`sun_path` attribute of `struct sockaddr_un` (which is defined as a
108-sized byte array on Linux), a larger buffer needs to be allocated so
that it can hold the path, and it is the responsibility of the
`unix_sockaddr_cleanup()` function to release that allocated memory.
In Git's CI, this stack allocation is not necessary because the code is
checked out to `/home/runner/work/git/git`. Concatenate the path
`t/trash directory.t0301-credential-cache/.cache/git/credential/socket`
and a terminating NUL, and you end up with 96 bytes, 12 shy of the
default `sun_path` size.
However, I use worktrees with slightly longer paths:
`/home/me/projects/git/yes/i/nest/worktrees/to/organize/them/` is more
in line with what I have. When I recently tried to locally reproduce a
failure of the `linux-leaks` CI job, this t0301 test failed (where it
had not failed in CI).
The reason: When `credential-cache` tries to reach its daemon initially
by calling `unix_sockaddr_init()`, it is expected that the daemon cannot
be reached (the idea is to spin up the daemon in that case and try
again). However, when this first call to `unix_sockaddr_init()` fails,
the code returns early from the `unix_stream_connect()` function
_without_ giving the cleanup code a chance to run, skipping the
deallocation of above-mentioned path.
The fix is easy: do not return early but instead go directly to the
cleanup code.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
In some implementations, `regexec_buf()` assumes that it is fed lines;
Without `REG_NOTEOL` it thinks the end of the buffer is the end of a
line. Which makes sense, but trips up this case because we are not
feeding lines, but rather a whole buffer. So the final newline is not
the start of an empty line, but the true end of the buffer.
This causes an interesting bug:
$ echo content >file.txt
$ git grep --no-index -n '^$' file.txt
file.txt:2:
This bug is fixed by making the end of the buffer consistently the end
of the final line.
The patch was applied from
https://lore.kernel.org/git/20250113062601.GD767856@coredump.intra.peff.net/
Reported-by: Olly Betts <olly@survex.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
On Windows, symbolic links come in two flavors: file symlinks and
directory symlinks. Since Git was born on Linux where this distinction
does not exist, Git for Windows has to auto-detect the type by looking
at the target. When the target does not yet exist at symlink creation
time, Git for Windows creates a "phantom" file symlink and later, once
checkout is complete, calls `CreateFileW()` on the target to check
whether it is actually a directory.
If the symlink target is a UNC path (e.g. `\\attacker\share`), this
auto-detection triggers an SMB connection to the remote host. Windows
performs NTLM authentication by default for such connections, which
means a crafted repository can exfiltrate the cloning user's NTLMv2
hash to an attacker-controlled server without any user interaction
beyond `git clone -c core.symlinks=true <url>`.
There are ways to specify UNC paths that start with only a single
backslash (e.g. `\??\UNC\host\share`); All of them do start like
that, though, so let's use that as a tell-tale that we should skip
the auto-detection in `process_phantom_symlink()`. The symlink is
then left as a file symlink (the `mklink` default), and a warning is
emitted suggesting the user set the `symlink` gitattribute to `dir`
if a directory symlink is needed. When the attribute is already set,
auto-detection is never invoked in the first place, so that code path
is unaffected.
This is the same class of vulnerability as CVE-2025-66413
(https://github.com/git-for-windows/git/security/advisories/GHSA-hv9c-4jm9-jh3x)
and follows the same general mitigation pattern that MinTTY adopted for
ANSI escape sequences referencing network share paths
(https://github.com/mintty/mintty/security/advisories/GHSA-jf4m-m6rv-p6c5).
Note that there are legitimate paths starting with a single backslash
that are _not_ network paths: drive-less absolute paths are interpreted
as relative to the current working directory's drive. In practice, these
are highly uncommon (and brittle, just one working directory change
away from breaking). In any case, the only consequence is now that the
symlink type of those has to be specified via Git attributes, is all.
Reported-by: Justin Lee <jessdhoctor@gmail.com>
Addresses: CVE-2026-32631
Addresses: https://github.com/git-for-windows/git/security/advisories/GHSA-9j5h-h4m7-85hx
Assisted-by: Claude Opus 4.6
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Every once in a while I need to verify that Microsoft Git's test suite
passes for changes that are not yet meant for public consumption, and
since it was (made) too difficult to keep up a working Azure Pipeline
definition, I have to use GitHub Actions in a private GitHub repository
for that purpose.
In these tests, basically all Dockerized CI jobs fail consistently. The
symptom is something like:
error: cannot create async thread: Resource temporarily unavailable
in the middle of a test, typically in the t5xxx-t6xxx range. The first
such error is immediately followed by plenty more of these errors, and
not a single test succeeds afterwards.
At first, I thought that maybe the massive parallelism I enjoy there is
the problem, and I thought that the cgroups limits might be shared
between the many containers that run on essentially the same physical
machine. But even reducing the matrix to just a single of those
Dockerized jobs runs into the very same problems.
The underlying reason seems to be a substantial difference in the hosted
runners that execute these Dockerized jobs: forcing the PID limit of the
container to a high number lets the jobs pass, even when running the
complete matrix of all 13 Dockerized jobs concurrently. But that's not
the only difference: The jobs seem to take a lot longer in these
containers than, say, in the containers made available to
https://github.com/git/git.
When forcing a PID limit of 64k in that private repository, the jobs
completed successfully, but they also took a lot longer, between 2x to
2.5x longer, i.e. painfully much longer. Reducing the PID limit to 16k,
the CI jobs still passed, but took an equally long amount of time.
Reducing the PID limit to 8k caused the errors to reappear.
Here are the numbers from three example runs, the first one forcing the
PID and nproc limit to 65536, the second one to 16384, the third run is
from the public git/git repository:
Job | 64k | 16k | reference
------------------------------|---------|---------|---------
almalinux-8 | 19m 3s | 16m 0s | 9m 36s
debian-11 | 20m 31s | 20m 3s | 8m 5s
fedora-breaking-changes-meson | 16m 29s | 19m 19s | 9m 40s
linux-asan-ubsan | 1h 10m | 1h 11m | 34m 36s
linux-breaking-changes | 25m 39s | 25m 58s | 13m 15s
linux-leaks | 1h 9m | 1h 10m | 33m 30s
linux-meson | 28m 9s | 27m 4s | 13m 45s
linux-musl-meson | 16m 32s | 13m 39s | 8m 6s
linux-reftable-leaks | 1h 13m | 1h 13m | 34m 34s
linux-reftable | 26m 2s | 25m 48s | 13m 31s
linux-sha256 | 26m 12s | 26m 3s | 12m 36s
linux-TEST-vars | 26m 5s | 25m 21s | 13m 25s
linux32 | 21m 16s | 19m 57s | 10m 44s
It does not look as if the PID limit is the reason for the longer
runtime, seeing as the 64k vs 16k timings deviate no more than as is
usual with GitHub workflows. So let's go for 16k.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The packed object source has been refactored into a proper struct
odb_source.
* ps/odb-source-packed:
odb/source-packed: drop pointer to "files" parent source
midx: refactor interfaces to work on "packed" source
odb/source-packed: stub out remaining functions
odb/source-packed: wire up `freshen_object()` callback
odb/source-packed: wire up `find_abbrev_len()` callback
odb/source-packed: wire up `count_objects()` callback
odb/source-packed: wire up `for_each_object()` callback
odb/source-packed: wire up `read_object_stream()` callback
odb/source-packed: wire up `read_object_info()` callback
packfile: use higher-level interface to implement `has_object_pack()`
odb/source-packed: wire up `reprepare()` callback
odb/source-packed: wire up `close()` callback
odb/source-packed: start converting to a proper `struct odb_source`
odb/source-packed: store pointer to "files" instead of generic source
packfile: move packed source into "odb/" subsystem
packfile: split out packfile list logic
packfile: rename `struct packfile_store` to `odb_source_packed`
Commands that list branches and tags (like git branch and git tag)
have been optimized to pass the namespace prefix when initializing
their ref iterator, avoiding a loose-ref scaling regression in
repositories with many unrelated loose references.
* td/ref-filter-restore-prefix-iteration:
ref-filter: restore prefix-scoped iteration
A hotfix to an earlier attempt to update code paths that assumed
"unsigned long" was long enough for "size_t".
* js/objects-larger-than-4gb-on-windows:
zlib: properly clamp to uLong
* js/objects-larger-than-4gb-on-windows-more:
odb: use size_t for object_info.sizep and the size APIs
packfile,delta: drop the `cast_size_t_to_ulong()` wrappers
pack-objects: use size_t for in-core object sizes
packfile: widen unpack_entry()'s size out-parameter to size_t
pack-objects(check_pack_inflate()): use size_t instead of unsigned long
patch-delta: use size_t for sizes
compat/msvc: use _chsize_s for ftruncate
A hotfix to an earlier attempt to update code paths that assumed
"unsigned long" was long enough for "size_t".
* js/objects-larger-than-4gb-on-windows:
zlib: properly clamp to uLong
"ort" merge backend handles merging corrupt trees better by
aborting when it should.
* en/ort-harden-against-corrupt-trees:
cache-tree: fix verify_cache() to catch non-adjacent D/F conflicts
merge-ort: abort merge when trees have duplicate entries
merge-ort: free diff pairs queue in clear_or_reinit_internal_opts()
merge-ort: drop unnecessary show_all_errors from collect_merge_info()
merge-ort: propagate callback errors from traverse_trees_wrapper()
A regression in the error diagnosis code for invalid .git files has
been fixed, avoiding a potential NULL-pointer crash when reporting
that a .git file does not point to a valid repository.
* jk/setup-gitfile-diag-fix:
read_gitfile(): simplify NOT_A_REPO error message
On platforms where `unsigned long` and `size_t` differ in bit size, we
want to clamp the buffers we pass to zlib to the former's size, as per
d05d666977 (git-zlib: handle data streams larger than 4GB, 2026-05-08).
The logic introduced in that commit performs a clamping to the bits,
though, which fails to do what is needed here: If too many bytes are
available in the buffers, we need to clamp to the maximum value of an
`unsigned long`. Otherwise, we ask zlib to use too small buffers, in the
worst case using 0 as the size (think: a value whose 32 lowest bits are
all zero).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Upstream Git does not test their tags with the expensive set of tests,
so a couple of them seem quite broken for now, even so much as hanging
indefinitely.
It is outside of the responsibility of the Git for Windows project to
fix upstream's own tests for platforms other than Windows, so let's not
exercise them.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Originally introduced as `core.useBuiltinFSMonitor` in Git for Windows
and developed, improved and stabilized there, the built-in FSMonitor
only made it into upstream Git (after unnecessarily long hemming and
hawing and throwing overly perfectionist style review sticks into the
spokes) as `core.fsmonitor = true`.
In Git for Windows, with this topic branch, we re-introduce the
now-obsolete config setting, with warnings suggesting to existing users
how to switch to the new config setting, with the intention to
ultimately drop the patch at some stage.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This topic branch re-adds the deprecated --stdin/-z options to `git
reset`. Those patches were overridden by a different set of options in
the upstream Git project before we could propose `--stdin`.
We offered this in MinGit to applications that wanted a safer way to
pass lots of pathspecs to Git, and these applications will need to be
adjusted.
Instead of `--stdin`, `--pathspec-from-file=-` should be used, and
instead of `-z`, `--pathspec-file-nul`.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
A fix for calling `vim` in Windows Terminal caused a regression and was
reverted. We partially un-revert this, to get the fix again.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
These are Git for Windows' Git GUI and gitk patches. We will have to
decide at some point what to do about them, but that's a little lower
priority (as Git GUI seems to be unmaintained for the time being, and
the gitk maintainer keeps a very low profile on the Git mailing list,
too).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This is the recommended way on GitHub to describe policies revolving around
security issues and about supported versions.
Helped-by: Sven Strickroth <email@cs-ware.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Reintroduce the 'core.useBuiltinFSMonitor' config setting (originally added
in 0a756b2a25 (fsmonitor: config settings are repository-specific,
2021-03-05)) after its removal from the upstream version of FSMonitor.
Upstream, the 'core.useBuiltinFSMonitor' setting was rendered obsolete by
"overloading" the 'core.fsmonitor' setting to take a boolean value. However,
several applications (e.g., 'scalar') utilize the original config setting,
so it should be preserved for a deprecation period before complete removal:
* if 'core.fsmonitor' is a boolean, the user is correctly using the new
config syntax; do not use 'core.useBuiltinFSMonitor'.
* if 'core.fsmonitor' is unspecified, use 'core.useBuiltinFSMonitor'.
* if 'core.fsmonitor' is a path, override and use the builtin FSMonitor if
'core.useBuiltinFSMonitor' is 'true'; otherwise, use the FSMonitor hook
indicated by the path.
Additionally, for this deprecation period, advise users to switch to using
'core.fsmonitor' to specify their use of the builtin FSMonitor.
Signed-off-by: Victoria Dye <vdye@github.com>
The `--stdin` option was a well-established paradigm in other commands,
therefore we implemented it in `git reset` for use by Visual Studio.
Unfortunately, upstream Git decided that it is time to introduce
`--pathspec-from-file` instead.
To keep backwards-compatibility for some grace period, we therefore
reinstate the `--stdin` option on top of the `--pathspec-from-file`
option, but mark it firmly as deprecated.
Helped-by: Victoria Dye <vdye@github.com>
Helped-by: Matthew John Cheetham <mjcheetham@outlook.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Git for Windows accepts pull requests; Core Git does not. Therefore we
need to adjust the template (because it only matches core Git's
project management style, not ours).
Also: direct Git for Windows enhancements to their contributions page,
space out the text for easy reading, and clarify that the mailing list
is plain text, not HTML.
Signed-off-by: Philip Oakley <philipoakley@iee.org>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Rather than using private IFTTT Applets that send mails to this
maintainer whenever a new version of a Git for Windows component was
released, let's use the power of GitHub workflows to make this process
publicly visible.
This workflow monitors the Atom/RSS feeds, and opens a ticket whenever a
new version was released.
Note: Bash sometimes releases multiple patched versions within a few
minutes of each other (i.e. 5.1p1 through 5.1p4, 5.0p15 and 5.0p16). The
MSYS2 runtime also has a similar system. We can address those patches as
a group, so we shouldn't get multiple issues about them.
Note further: We're not acting on newlib releases, OpenSSL alphas, Perl
release candidates or non-stable Perl releases. There's no need to open
issues about them.
Co-authored-by: Matthias Aßhauer <mha1993@live.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Getting started contributing to Git can be difficult on a Windows
machine. CONTRIBUTING.md contains a guide to getting started, including
detailed steps for setting up build tools, running tests, and
submitting patches to upstream.
[includes an example by Pratik Karki how to submit v2, v3, v4, etc.]
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
The Git project followed Git for Windows' lead and added their Code of
Conduct, based on the Contributor Covenant v1.4, later updated to v2.0.
We adapt it slightly to Git for Windows.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
In this time and age, AI is everywhere. However, it's sometimes not very
easy to use. For green-field projects it works quite a bit better than
for existing legacy projects. And Git's source code is _quite_ as legacy
code as they come... 😁
Now, the only way how AI can be used efficiently with legacy code
is by providing enough information by way of prompt context for the
AI to have a chance to make any sense of the code. The structure and
the architecture is, after all, not designed for AI, but rather the
opposite: By virtue of having grown organically over two decades, there
is no design that AI coding models would readily grasp.
So here is a document that describes all kinds of aspects about this
project. The idea is to help AI by providing information that it does
not have ingrained in its weights. The idea is to provide information
that a human prompter might take for granted, but no coding model will
have been trained on specifically.
Assisted-by: Claude Opus 4.5
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The Git for Windows project has grown quite complex over the years,
certainly much more complex than during the first years where the
`msysgit.git` repository was abusing Git for package management purposes
and the `git/git` fork was called `4msysgit.git`.
Let's describe the status quo in a thorough way.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
In e3f7e01b50 (Revert "editor: save and reset terminal after calling
EDITOR", 2021-11-22), we reverted the commit wholesale where the
terminal state would be saved and restored before/after calling an
editor.
The reverted commit was intended to fix a problem with Windows Terminal
where simply calling `vi` would cause problems afterwards.
To fix the problem addressed by the revert, but _still_ keep the problem
with Windows Terminal fixed, let's revert the revert, with a twist: we
restrict the save/restore _specifically_ to the case where `vi` (or
`vim`) is called, and do not do the same for any other editor.
This should still catch the majority of the cases, and will bridge the
time until the original patch is re-done in a way that addresses all
concerns.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
