This topic vendors in mimalloc v2.0.9, a fast allocator that allows Git
for Windows to perform efficiently.
Switch Git for Windows to using mimalloc instead of nedmalloc
In MSYS2, we have two Python interpreters at our disposal, so we can
include the Python stuff in the build.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This topic branch teaches `git clean` to respect NTFS junctions and Unix
bind mounts: it will now stop at those boundaries.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This topic branch allows us to specify absolute paths without the drive
prefix e.g. when cloning.
Example:
C:\Users\me> git clone https://github.com/git/git \upstream-git
This will clone into a new directory C:\upstream-git, in line with how
Windows interprets absolute paths.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
These fixes were necessary for Sverre Rabbelier's remote-hg to work,
but for some magic reason they are not necessary for the current
remote-hg. Makes you wonder how that one gets away with it.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The winsock2 library provides functions that work on different data
types than file descriptors, therefore we wrap them.
But that is not the only difference: they also do not set `errno` but
expect the callers to enquire about errors via `WSAGetLastError()`.
Let's translate that into appropriate `errno` values whenever the socket
operations fail so that Git's code base does not have to change its
expectations.
This closes https://github.com/git-for-windows/git/issues/2404
Helped-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Since commit 0c499ea60f (send-pack: demultiplex a sideband stream with
status data, 2010-02-05) the send-pack builtin uses the side-band-64k
capability if advertised by the server.
Unfortunately this breaks pushing over the dump git protocol if used
over a network connection.
The detailed reasons for this breakage are (by courtesy of Jeff Preshing,
quoted from https://groups.google.com/d/msg/msysgit/at8D7J-h7mw/eaLujILGUWoJ):
MinGW wraps Windows sockets in CRT file descriptors in order to
mimic the functionality of POSIX sockets. This causes msvcrt.dll
to treat sockets as Installable File System (IFS) handles,
calling ReadFile, WriteFile, DuplicateHandle and CloseHandle on
them. This approach works well in simple cases on recent
versions of Windows, but does not support all usage patterns. In
particular, using this approach, any attempt to read & write
concurrently on the same socket (from one or more processes)
will deadlock in a scenario where the read waits for a response
from the server which is only invoked after the write. This is
what send_pack currently attempts to do in the use_sideband
codepath.
The new config option `sendpack.sideband` allows to override the
side-band-64k capability of the server, and thus makes the dumb git
protocol work.
Other transportation methods like ssh and http/https still benefit from
the sideband channel, therefore the default value of `sendpack.sideband`
is still true.
Signed-off-by: Thomas Braun <thomas.braun@byte-physics.de>
Signed-off-by: Oliver Schneider <oliver@assarbad.net>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Thorough benchmarking with repacking a subset of linux.git (the commit
history reachable from 93a6fefe2f ([PATCH] fix the SYSCTL=n compilation,
2007-02-28), to be precise) suggest that this allocator is on par, in
multi-threaded situations maybe even better than nedmalloc:
`git repack -adfq` with mimalloc, 8 threads:
31.166991900 27.576763800 28.712311000 27.373859000 27.163141900
`git repack -adfq` with nedmalloc, 8 threads:
31.915032900 27.149883100 28.244933700 27.240188800 28.580849500
In a different test using GitHub Actions build agents (probably
single-threaded, a core-strength of nedmalloc)):
`git repack -q -d -l -A --unpack-unreachable=2.weeks.ago` with mimalloc:
943.426 978.500 939.709 959.811 954.605
`git repack -q -d -l -A --unpack-unreachable=2.weeks.ago` with nedmalloc:
995.383 952.179 943.253 963.043 980.468
While these measurements were not executed with complete scientific
rigor, as no hardware was set aside specifically for these benchmarks,
it shows that mimalloc and nedmalloc perform almost the same, nedmalloc
with a bit higher variance and also slightly higher average (further
testing suggests that nedmalloc performs worse in multi-threaded
situations than in single-threaded ones).
In short: mimalloc seems to be slightly better suited for our purposes
than nedmalloc.
Seeing that mimalloc is developed actively, while nedmalloc ceased to
see any updates in eight years, let's use mimalloc on Windows instead.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
While Git for Windows does not _ship_ Python (in order to save on
bandwidth), MSYS2 provides very fine Python interpreters that users can
easily take advantage of, by using Git for Windows within its SDK.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Always use the internal "use_weak" random seed when initializing
the "mimalloc" heap when statically linked on Windows.
The imported "mimalloc" routines support several random sources
to seed the heap data structures, including BCrypt.dll and
RtlGenRandom. Crashes have been reported when using BCrypt.dll
if it initialized during an `atexit()` handler function. Granted,
such DLL initialization should not happen in an atexit handler,
but yet the crashes remain.
It should be noted that on Windows when statically linked, the
mimalloc startup code (called by the GCC CRT to initialize static
data prior to calling `main()`) always uses the internal "weak"
random seed. "mimalloc" does not try to load an alternate
random source until after the OS initialization has completed.
Heap data is stored in `__declspec(thread)` TLS data and in theory
each Git thread will have its own heap data. However, testing
shows that the "mimalloc" library doesn't actually call
`os_random_buf()` (to load a new random source) when creating these
new per-thread heap structures.
However, if an atexit handler is forced to run on a non-main
thread, the "mimalloc" library *WILL* try to create a new heap
and seed it with `os_random_buf()`. (The reason for this is still
a mystery to this author.) The `os_random_buf()` call can cause
the (previously uninitialized BCrypt.dll library) to be dynamically
loaded and a call made into it. Crashes have been reported in
v2.40.1.vfs.0.0 while in this call.
As a workaround, the fix here forces the use of the internal
"use_weak" random code for the subsequent `os_random_buf()` calls.
Since we have been using that random generator for the majority
of the program, it seems safe to use it for the final few mallocs
in the atexit handler (of which there really shouldn't be that many.
Signed-off-by: Jeff Hostetler <jeffhostetler@github.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
By defining `USE_MIMALLOC`, Git can now be compiled with that
nicely-fast and small allocator.
Note that we have to disable a couple `DEVELOPER` options to build
mimalloc's source code, as it makes heavy use of declarations after
statements, among other things that disagree with Git's conventions.
We even have to silence some GCC warnings in non-DEVELOPER mode. For
example, the `-Wno-array-bounds` flag is needed because in `-O2` builds,
trying to call `NtCurrentTeb()` (which `_mi_thread_id()` does on
Windows) causes the bogus warning about a system header, likely related
to https://sourceforge.net/p/mingw-w64/mailman/message/37674519/ and to
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99578:
C:/git-sdk-64-minimal/mingw64/include/psdk_inc/intrin-impl.h:838:1:
error: array subscript 0 is outside array bounds of 'long long unsigned int[0]' [-Werror=array-bounds]
838 | __buildreadseg(__readgsqword, unsigned __int64, "gs", "q")
| ^~~~~~~~~~~~~~
Also: The `mimalloc` library uses C11-style atomics, therefore we must
require that standard when compiling with GCC if we want to use
`mimalloc` (instead of requiring "only" C99). This is what we do in the
CMake definition already, therefore this commit does not need to touch
`contrib/buildsystems/`.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We want to compile mimalloc's source code as part of Git, rather than
requiring the code to be built as an external library: mimalloc uses a
CMake-based build, which is not necessarily easy to integrate into the
flavors of Git for Windows (which will be the main benefitting port).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This commit imports mimalloc's source code as per v2.1.2, fetched from
the tag at https://github.com/microsoft/mimalloc.
The .c files are from the src/ subdirectory, and the .h files from the
include/ and include/mimalloc/ subdirectories. We will subsequently
modify the source code to accommodate building within Git's context.
Since we plan on using the `mi_*()` family of functions, we skip the
C++-specific source code, some POSIX compliant functions to interact
with mimalloc, and the code that wants to support auto-magic overriding
of the `malloc()` function (mimalloc-new-delete.h, alloc-posix.c,
mimalloc-override.h, alloc-override.c, alloc-override-osx.c,
alloc-override-win.c and static.c).
To appease the `check-whitespace` job of Git's Continuous Integration,
this commit was washed one time via `git rebase --whitespace=fix`.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We are about to vendor in `mimalloc`'s source code which we will want to
include `git-compat-util.h` after defining that constant.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The mingw-w64 GCC seems to link implicitly to libwinpthread, which does
implement a pthread emulation (that is more complete than Git's). Let's
keep preferring Git's.
To avoid linker errors where it thinks that the `pthread_self` and the
`pthread_create` symbols are defined twice, let's give our version a
`win32_` prefix, just like we already do for `pthread_join()`.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Windows' equivalent to "bind mounts", NTFS junction points, can be
unlinked without affecting the mount target. This is clearly what users
expect to happen when they call `git clean -dfx` in a worktree that
contains NTFS junction points: the junction should be removed, and the
target directory of said junction should be left alone (unless it is
inside the worktree).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When specifying an absolute path without a drive prefix, we convert that
path internally. Let's make sure that we handle that case properly, too
;-)
This fixes the command
git clone https://github.com/git-for-windows/git \G4W
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This topic branch extends the protections introduced for Git GUI's
CVE-2022-41953 to cover `gitk`, too.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
After importing anything with fast-import, we should always let the
garbage collector do its job, since the objects are written to disk
inefficiently.
This brings down an initial import of http://selenic.com/hg from about
230 megabytes to about 14.
In the future, we may want to make this configurable on a per-remote
basis, or maybe teach fast-import about it in the first place.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
It seems to be not exactly rare on Windows to install NTFS junction
points (the equivalent of "bind mounts" on Linux/Unix) in worktrees,
e.g. to map some development tools into a subdirectory.
In such a scenario, it is pretty horrible if `git clean -dfx` traverses
into the mapped directory and starts to "clean up".
Let's just not do that. Let's make sure before we traverse into a
directory that it is not a mount point (or junction).
This addresses https://github.com/git-for-windows/git/issues/607
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
On Windows, there are several categories of absolute paths. One such
category starts with a backslash and is implicitly relative to the
drive associated with the current working directory. Example:
c:
git clone https://github.com/git-for-windows/git \G4W
should clone into C:\G4W.
There is currently a problem with that, in that mingw_mktemp() does not
expect the _wmktemp() function to prefix the absolute path with the
drive prefix, and as a consequence, the resulting path does not fit into
the originally-passed string buffer. The symptom is a "Result too large"
error.
Reported by Juan Carlos Arevalo Baeza.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Just like CVE-2022-41953 for Git GUI, there exists a vulnerability of
`gitk` where it looks for `taskkill.exe` in the current directory before
searching `PATH`.
Note that the many `exec git` calls are unaffected, due to an obscure
quirk in Tcl's `exec` function. Typically, `git.exe` lives next to
`wish.exe` (i.e. the program that is run to execute `gitk` or Git GUI)
in Git for Windows, and that is the saving grace for `git.exe because
`exec` searches the directory where `wish.exe` lives even before the
current directory, according to
https://www.tcl-lang.org/man/tcl/TclCmd/exec.htm#M24:
If a directory name was not specified as part of the application
name, the following directories are automatically searched in
order when attempting to locate the application:
The directory from which the Tcl executable was loaded.
The current directory.
The Windows 32-bit system directory.
The Windows home directory.
The directories listed in the path.
The same is not true, however, for `taskkill.exe`: it lives in the
Windows system directory (never mind the 32-bit, Tcl's documentation is
outdated on that point, it really means `C:\Windows\system32`).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This happens only when the corresponding commits are not exported in
the current fast-export run. This can happen either when the relevant
commit is already marked, or when the commit is explicitly marked
as UNINTERESTING with a negative ref by another argument.
This breaks fast-export basec remote helpers.
Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
Git LFS is now built with Go 1.21 which no longer supports Windows 7.
However, Git for Windows still wants to support Windows 7.
Ideally, Git LFS would re-introduce Windows 7 support until Git for
Windows drops support for Windows 7, but that's [not going to
happen](https://github.com/git-for-windows/git/issues/4996#issuecomment-2176152565).
The next best thing we can do is to let the users know what is
happening, and how to get out of their fix, at least.
This is not quite as easy as it would first seem because programs
compiled with Go 1.21 or newer will simply throw an exception and fail
with an Access Violation on Windows 7.
The only way I found to address this is to replicate the logic from Go's
very own `version` command (which can determine the Go version with
which a given executable was built) to detect the situation, and in that
case offer a helpful error message.
This addresses https://github.com/git-for-windows/git/issues/4996.
/cc @chrisd8088
Git LFS is now built with Go 1.21 which no longer supports Windows 7.
However, Git for Windows still wants to support Windows 7.
Ideally, Git LFS would re-introduce Windows 7 support until Git for
Windows drops support for Windows 7, but that's not going to happen:
https://github.com/git-for-windows/git/issues/4996#issuecomment-2176152565
The next best thing we can do is to let the users know what is
happening, and how to get out of their fix, at least.
This is not quite as easy as it would first seem because programs
compiled with Go 1.21 or newer will simply throw an exception and fail
with an Access Violation on Windows 7.
The only way I found to address this is to replicate the logic from Go's
very own `version` command (which can determine the Go version with
which a given executable was built) to detect the situation, and in that
case offer a helpful error message.
This addresses https://github.com/git-for-windows/git/issues/4996.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The code to deal with modified paths that are out-of-cone in a
sparsely checked out working tree has been optimized.
* ds/sparse-lstat-caching:
sparse-index: improve lstat caching of sparse paths
sparse-index: count lstat() calls
sparse-index: use strbuf in path_found()
sparse-index: refactor path_found()
sparse-checkout: refactor skip worktree retry logic
When bundleURI interface fetches multiple bundles, Git failed to
take full advantage of all bundles and ended up slurping duplicated
objects.
* xx/bundie-uri-fixes:
unbundle: extend object verification for fetches
fetch-pack: expose fsckObjects configuration logic
bundle-uri: verify oid before writing refs
The Bloom filter used for path limited history traversal was broken
on systems whose "char" is unsigned; update the implementation and
bump the format version to 2.
* tb/path-filter-fix:
bloom: introduce `deinit_bloom_filters()`
commit-graph: reuse existing Bloom filters where possible
object.h: fix mis-aligned flag bits table
commit-graph: new Bloom filter version that fixes murmur3
commit-graph: unconditionally load Bloom filters
bloom: prepare to discard incompatible Bloom filters
bloom: annotate filters with hash version
repo-settings: introduce commitgraph.changedPathsVersion
t4216: test changed path filters with high bit paths
t/helper/test-read-graph: implement `bloom-filters` mode
bloom.h: make `load_bloom_filter_from_graph()` public
t/helper/test-read-graph.c: extract `dump_graph_info()`
gitformat-commit-graph: describe version 2 of BDAT
commit-graph: ensure Bloom filters are read with consistent settings
revision.c: consult Bloom filters for root commits
t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`
date parser updates to be more careful about underflowing epoch
based timestamp.
* db/date-underflow-fix:
date: detect underflow/overflow when parsing dates with timezone offset
t0006: simplify prerequisites
When GIT_PAGER failed to spawn, depending on the code path taken,
we failed immediately (correct) or just spew the payload to the
standard output (incorrect). The code now always fail immediately
when GIT_PAGER fails.
* rj/pager-die-upon-exec-failure:
pager: die when paging to non-existing command
"git archive --add-virtual-file=<path>:<contents>" never paid
attention to the --prefix=<prefix> option but the documentation
said it would. The documentation has been corrected.
* jc/archive-prefix-with-add-virtual-file:
archive: document that --add-virtual-file takes full path
* https://github.com/j6t/git-gui:
git-gui: fix inability to quit after closing another instance
git-gui: sv.po: Update Swedish translation (576t0f0u)
git-gui: note the new maintainer
Makefile(s): do not enforce "all indents must be done with tab"
Makefile(s): avoid recipe prefix in conditional statements
doc: switch links to https
doc: update links to current pages
git-gui: po: fix typo in French "aperçu"
The problem can be reproduced on Linux with this sequence:
1. Run git gui from a terminal.
2. Edit the commit message and wait for at least 2 seconds.
3. Terminate the instance from the terminal, for example with Ctrl-C,
to simulate crash. This leaves the file .git/GITGUI_BCK behind.
4. Start two instances of git gui &.
At this point the first instance can be closed (it renames
.git/GITGUI_BCK to .git/GITGUI_MSG), but the seconds brings an error
message about the absent file and cannot be closed thereafter and must
be killed from the command line.
The renaming that happens by the first instance is the correct action
and need not be repeated by the second instance. It is the correct
action to ignore the failed renaming.
On the other hand, the second instance could just edit the commit
message again, wait 2 seconds to write GITGUI_BCK, and then can be
closed without failing. At this point, since the user has edited the
message, it is again correct to preserve the edited version in
GITGUI_MSG.
* os/catch-rename:
git-gui: fix inability to quit after closing another instance
In order to be a better Windows citizenship, Git should
save its configuration files on AppData folder. This can
enables git configuration files be replicated between machines
using the same Microsoft account logon which would reduce the
friction of setting up Git on new systems. Therefore, if
%APPDATA%\Git\config exists, we use it; otherwise
$HOME/.config/git/config is used.
Signed-off-by: Ariel Lourenco <ariellourenco@users.noreply.github.com>
"git diff --no-ext-diff" when diff.external is configured ignored
the "--color-moved" option.
* rs/diff-color-moved-w-no-ext-diff-fix:
diff: allow --color-moved with --no-ext-diff
