Commit Graph

150682 Commits

Author SHA1 Message Date
Johannes Schindelin
1ea725ba35 compat/vcbuild: document preferred way to build in Visual Studio
We used to have that `make vcxproj` hack, but a hack it is. In the
meantime, we have a much cleaner solution: using CMake, either
explicitly, or even more conveniently via Visual Studio's built-in CMake
support (simply open Git's top-level directory via File>Open>Folder...).

Let's let the `README` reflect this.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:33 +02:00
Johannes Schindelin
159c9f27e1 mingw: allow for longer paths in parse_interpreter()
As reported in https://github.com/newren/git-filter-repo/pull/225, it
looks like 99 bytes is not really sufficient to represent e.g. the full
path to Python when installed via Windows Store (and this path is used
in the hasb bang line when installing scripts via `pip`).

Let's increase it to what is probably the maximum sensible path size:
MAX_PATH. This makes `parse_interpreter()` in line with what
`lookup_prog()` handles.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Vilius Šumskas <vilius@sumskas.eu>
2023-08-21 20:10:33 +02:00
Johannes Schindelin
d66ef0984c ci(vs-build): download the vcpkg artifacts using a dedicated Action
We now have a GitHub Action to download and cache Azure Pipelines
artifacts (such as the `vcpkg` artifacts), hiding gnarly internals, and
also providing some robustness against network glitches. Let's use it.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:33 +02:00
Dennis Ameling
9cdb4c0a84 cmake(): allow setting HOST_CPU for cross-compilation
Git's regular Makefile mentions that HOST_CPU should be defined when cross-compiling Git: 37796bca76/Makefile (L438-L439)

This is then used to set the GIT_HOST_CPU variable when compiling Git: 37796bca76/Makefile (L1337-L1341)

Then, when the user runs `git version --build-options`, it returns that value: 37796bca76/help.c (L658)

This commit adds the same functionality to the CMake configuration. Users can now set -DHOST_CPU= to set the target architecture.

Signed-off-by: Dennis Ameling <dennis@dennisameling.com>
2023-08-21 20:10:33 +02:00
Dennis Ameling
610eef87e1 Add schannel to curl installation
Signed-off-by: Dennis Ameling <dennis@dennisameling.com>
2023-08-21 20:10:33 +02:00
Philip Oakley
7879368052 vcpkg_install: add comment regarding slow network connections
The vcpkg downloads may not succeed. Warn careful readers of the time out.

A simple retry will usually resolve the issue.

Signed-off-by: Philip Oakley <philipoakley@iee.email>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:32 +02:00
Yuyi Wang
2feafd8494 cmake: install headless-git.
headless-git is a git executable without opening a console window. It is
useful when other GUI executables want to call git. We should install it
together with git on Windows.

Signed-off-by: Yuyi Wang <Strawberry_Str@hotmail.com>
2023-08-21 20:10:32 +02:00
Dennis Ameling
039f120c07 ci(vs-build) also build Windows/ARM64 artifacts
There are no Windows/ARM64 agents in GitHub Actions yet, therefore we
just skip adjusting the `vs-test` job for now.

Signed-off-by: Dennis Ameling <dennis@dennisameling.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:32 +02:00
Philip Oakley
a1a8f82020 vcpkg_install: detect lack of Git
The vcpkg_install batch file depends on the availability of a
working Git on the CMD path. This may not be present if the user
has selected the 'bash only' option during Git-for-Windows install.

Detect and tell the user about their lack of a working Git in the CMD
window.

Fixes #2348.
A separate PR https://github.com/git-for-windows/build-extra/pull/258
now highlights the recommended path setting during install.

Signed-off-by: Philip Oakley <philipoakley@iee.email>
2023-08-21 20:10:32 +02:00
Dennis Ameling
06ff5087c8 cmake: allow building for Windows/ARM64
Signed-off-by: Dennis Ameling <dennis@dennisameling.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:32 +02:00
Ian Bearman
44d8233c37 vcbuild: add an option to install individual 'features'
In this context, a "feature" is a dependency combined with its own
dependencies.

Signed-off-by: Ian Bearman <ianb@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:32 +02:00
Ian Bearman
28531e7e80 vcbuild: install ARM64 dependencies when building ARM64 binaries
Co-authored-by: Dennis Ameling <dennis@dennisameling.com>
Signed-off-by: Ian Bearman <ianb@microsoft.com>
Signed-off-by: Dennis Ameling <dennis@dennisameling.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:32 +02:00
Ian Bearman
881cddc96a vcxproj: support building Windows/ARM64 binaries
Signed-off-by: Ian Bearman <ianb@microsoft.com>
Signed-off-by: Dennis Ameling <dennis@dennisameling.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:32 +02:00
Johannes Schindelin
884ad997f5 vcxproj: handle GUI programs, too
So far, we only built Console programs, but we are about to introduce a
program that targets the Windows subsystem (i.e. it is a so-called "GUI"
program).

Let's handle this preemptively in the script that generates the Visual
Studio files.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:31 +02:00
Johannes Schindelin
3c175f0dd3 vcxproj: ignore -fno-stack-protector and -fno-common
An upcoming commit will introduce those compile options; MSVC does not
understand them, so let's suppress them when generating the Visual
Studio project files.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:31 +02:00
Johannes Schindelin
f4616bca74 vcxproj: handle resource files, too
On Windows, we also compile a "resource" file, which is similar to
source code, but contains metadata (such as the program version).

So far, we did not compile it in `MSVC` mode, only when compiling Git
for Windows with the GNU C Compiler.

In preparation for including it also when compiling with MS Visual C,
let's teach our `vcxproj` generator to handle those sort of files, too.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:31 +02:00
Johannes Schindelin
eccd8c6ae4 buildsystems: remove duplicate clause
This seems to have been there since 259d87c354 (Add scripts to
generate projects for other buildsystems (MSVC vcproj, QMake),
2009-09-16), i.e. since the beginning of that file.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:31 +02:00
Jeff Hostetler
bcecfecb49 clink.pl: move default linker options for MSVC=1 builds
Move the default `-ENTRY` and `-SUBSYSTEM` arguments for
MSVC=1 builds from `config.mak.uname` into `clink.pl`.
These args are constant for console-mode executables.

Add support to `clink.pl` for generating a Win32 GUI application
using the `-mwindows` argument (to match how GCC does it).  This
changes the `-ENTRY` and `-SUBSYSTEM` arguments accordingly.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
2023-08-21 20:10:31 +02:00
Jeff Hostetler
b5c7c594a0 clink.pl: ignore no-stack-protector arg on MSVC=1 builds
Ignore the `-fno-stack-protector` compiler argument when building
with MSVC.  This will be used in a later commit that needs to build
a Win32 GUI app.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
2023-08-21 20:10:31 +02:00
Jeff Hostetler
6aadf65b7e config.mak.uname: add git.rc to MSVC builds
Teach MSVC=1 builds to depend on the `git.rc` file so that
the resulting executables have Windows-style resources and
version number information within them.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
2023-08-21 20:10:31 +02:00
Jeff Hostetler
d7ba9e8c41 vcbuild: add support for compiling Windows resource files
Create a wrapper for the Windows Resource Compiler (RC.EXE)
for use by the MSVC=1 builds. This is similar to the CL.EXE
and LIB.EXE wrappers used for the MSVC=1 builds.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
2023-08-21 20:10:31 +02:00
Jeff Hostetler
2c68dc2cad Makefile: clean up .ilk files when MSVC=1
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
2023-08-21 20:10:31 +02:00
Jeff Hostetler
1d0ea2a339 clink.pl: fix libexpatd.lib link error when using MSVC
When building with `make MSVC=1 DEBUG=1`, link to `libexpatd.lib`
rather than `libexpat.lib`.

It appears that the `vcpkg` package for "libexpat" has changed and now
creates `libexpatd.lib` for debug mode builds.  Previously, both debug
and release builds created a ".lib" with the same basename.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
2023-08-21 20:10:31 +02:00
Johannes Schindelin
8c742d6d15 Merge branch 'dscho-avoid-d-f-conflict-in-vs-master'
Merge this early to resolve merge conflicts early.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:31 +02:00
Johannes Schindelin
9ad32ef2fc git-gui: accommodate for intent-to-add files
As of Git v2.28.0, the diff for files staged via `git add -N` marks them
as new files. Git GUI was ill-prepared for that, and this patch teaches
Git GUI about them.

Please note that this will not even fix things with v2.28.0, as the
`rp/apply-cached-with-i-t-a` patches are required on Git's side, too.

This fixes https://github.com/git-for-windows/git/issues/2779

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Pratyush Yadav <me@yadavpratyush.com>
2023-08-21 20:10:30 +02:00
Jens Glathe
e9e5b60e38 t0014: fix indentation
For some reason, this test case was indented with 4 spaces instead of 1
horizontal tab. The other test cases in the same test script are fine.

Signed-off-by: Jens Glathe <jens.glathe@oldschoolsolutions.biz>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:30 +02:00
Luke Bonanomi
a16a37163e commit: accept "scissors" with CR/LF line endings
This change enhances `git commit --cleanup=scissors` by detecting
scissors lines ending in either LF (UNIX-style) or CR/LF (DOS-style).

Regression tests are included to specifically test for trailing
comments after a CR/LF-terminated scissors line.

Signed-off-by: Luke Bonanomi <lbonanomi@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:30 +02:00
Johannes Schindelin
98a61b595c git add -i: handle CR/LF line endings in the interactive input
As of Git for Windows v2.27.0, there is an option to use Windows'
newly-introduced Pseudo Console support. When running an interactive add
operation with this support enabled, Git will receive CR/LF line
endings.

Therefore, let's not pretend that we are expecting Unix line endings.

This fixes https://github.com/git-for-windows/git/issues/2729

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:30 +02:00
Johannes Schindelin
511e54c103 t3701: verify that we can add *lots* of files interactively
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:30 +02:00
Johannes Schindelin
08cc551243 t5505/t5516: fix white-space around redirectors
The convention in Git project's shell scripts is to have white-space
_before_, but not _after_ the `>` (or `<`).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:29 +02:00
Johannes Schindelin
b12210ad4e http: use new "best effort" strategy for Secure Channel revoke checking
The native Windows HTTPS backend is based on Secure Channel which lets
the caller decide how to handle revocation checking problems caused by
missing information in the certificate or offline CRL distribution
points.

Unfortunately, cURL chose to handle these problems differently than
OpenSSL by default: while OpenSSL happily ignores those problems
(essentially saying "¯\_(ツ)_/¯"), the Secure Channel backend will error
out instead.

As a remedy, the "no revoke" mode was introduced, which turns off
revocation checking altogether. This is a bit heavy-handed. We support
this via the `http.schannelCheckRevoke` setting.

In https://github.com/curl/curl/pull/4981, we contributed an opt-in
"best effort" strategy that emulates what OpenSSL seems to do.

In Git for Windows, we actually want this to be the default. This patch
makes it so, introducing it as a new value for the
`http.schannelCheckRevoke" setting, which now becmes a tristate: it
accepts the values "false", "true" or "best-effort" (defaulting to the
last one).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:29 +02:00
Johannes Schindelin
c93676b2fc mingw: ignore HOMEDRIVE/HOMEPATH if it points to Windows' system directory
Internally, Git expects the environment variable `HOME` to be set, and
to point to the current user's home directory.

This environment variable is not set by default on Windows, and
therefore Git tries its best to construct one if it finds `HOME` unset.

There are actually two different approaches Git tries: first, it looks
at `HOMEDRIVE`/`HOMEPATH` because this is widely used in corporate
environments with roaming profiles, and a user generally wants their
global Git settings to be in a roaming profile.

Only when `HOMEDRIVE`/`HOMEPATH` is either unset or does not point to a
valid location, Git will fall back to using `USERPROFILE` instead.

However, starting with Windows Vista, for secondary logons and services,
the environment variables `HOMEDRIVE`/`HOMEPATH` point to Windows'
system directory (usually `C:\Windows\system32`).

That is undesirable, and that location is usually write-protected anyway.

So let's verify that the `HOMEDRIVE`/`HOMEPATH` combo does not point to
Windows' system directory before using it, falling back to `USERPROFILE`
if it does.

This fixes git-for-windows#2709

Initial-Path-by: Ivan Pozdeev <vano@mail.mipt.ru>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:29 +02:00
Johannes Schindelin
b1d9548ef1 mingw: implement a platform-specific strbuf_realpath()
There is a Win32 API function to resolve symbolic links, and we can use
that instead of resolving them manually. Even better, this function also
resolves NTFS junction points (which are somewhat similar to bind
mounts).

This fixes https://github.com/git-for-windows/git/issues/2481.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:29 +02:00
Johannes Schindelin
aa0af62246 t5505/t5516: allow running without .git/branches/ in the templates
When we commit the template directory as part of `make vcxproj`, the
`branches/` directory is not actually commited, as it is empty.

Two tests were not prepared for that situation.

This developer tried to get rid of the support for `.git/branches/` a
long time ago, but that effort did not bear fruit, so the best we can do
is work around in these here tests.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:29 +02:00
Johannes Schindelin
00fa77d01f mingw: allow git.exe to be used instead of the "Git wrapper"
Git for Windows wants to add `git.exe` to the users' `PATH`, without
cluttering the latter with unnecessary executables such as `wish.exe`.
To that end, it invented the concept of its "Git wrapper", i.e. a tiny
executable located in `C:\Program Files\Git\cmd\git.exe` (originally a
CMD script) whose sole purpose is to set up a couple of environment
variables and then spawn the _actual_ `git.exe` (which nowadays lives in
`C:\Program Files\Git\mingw64\bin\git.exe` for 64-bit, and the obvious
equivalent for 32-bit installations).

Currently, the following environment variables are set unless already
initialized:

- `MSYSTEM`, to make sure that the MSYS2 Bash and the MSYS2 Perl
  interpreter behave as expected, and

- `PLINK_PROTOCOL`, to force PuTTY's `plink.exe` to use the SSH
  protocol instead of Telnet,

- `PATH`, to make sure that the `bin` folder in the user's home
  directory, as well as the `/mingw64/bin` and the `/usr/bin`
  directories are included. The trick here is that the `/mingw64/bin/`
  and `/usr/bin/` directories are relative to the top-level installation
  directory of Git for Windows (which the included Bash interprets as
  `/`, i.e. as the MSYS pseudo root directory).

Using the absence of `MSYSTEM` as a tell-tale, we can detect in
`git.exe` whether these environment variables have been initialized
properly. Therefore we can call `C:\Program Files\Git\mingw64\bin\git`
in-place after this change, without having to call Git through the Git
wrapper.

Obviously, above-mentioned directories must be _prepended_ to the `PATH`
variable, otherwise we risk picking up executables from unrelated Git
installations. We do that by constructing the new `PATH` value from
scratch, appending `$HOME/bin` (if `HOME` is set), then the MSYS2 system
directories, and then appending the original `PATH`.

Side note: this modification of the `PATH` variable is independent of
the modification necessary to reach the executables and scripts in
`/mingw64/libexec/git-core/`, i.e. the `GIT_EXEC_PATH`. That
modification is still performed by Git, elsewhere, long after making the
changes described above.

While we _still_ cannot simply hard-link `mingw64\bin\git.exe` to `cmd`
(because the former depends on a couple of `.dll` files that are only in
`mingw64\bin`, i.e. calling `...\cmd\git.exe` would fail to load due to
missing dependencies), at least we can now avoid that extra process of
running the Git wrapper (which then has to wait for the spawned
`git.exe` to finish) by calling `...\mingw64\bin\git.exe` directly, via
its absolute path.

Testing this is in Git's test suite tricky: we set up a "new" MSYS
pseudo-root and copy the `git.exe` file into the appropriate location,
then verify that `MSYSTEM` is set properly, and also that the `PATH` is
modified so that scripts can be found in `$HOME/bin`, `/mingw64/bin/`
and `/usr/bin/`.

This addresses https://github.com/git-for-windows/git/issues/2283

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:29 +02:00
Johannes Schindelin
473f31acdb vcxproj: unclash project directories with build outputs
It already caused problems with the test suite that the directory
containing `git.vcxproj` is called the same as the Git executable
without its file extension: `./git` is ambiguous, it could refer both to
the directory `git/` as well as to `git.exe`.

Now there is one more problem: when our GitHub workflow runs on the
`vs/master` branch, it fails in all but the Windows builds, as they want
to write the file `git` but there is already a directory in the way.

Let's just go ahead and append `.proj` to all of those directories, e.g.
`git.proj/` instead of `git/`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:29 +02:00
Johannes Schindelin
f2d8cb2323 mingw: ensure valid CTYPE
A change between versions 2.4.1 and 2.6.0 of the MSYS2 runtime modified
how Cygwin's runtime (and hence Git for Windows' MSYS2 runtime
derivative) handles locales: d16a56306d (Consolidate wctomb/mbtowc calls
for POSIX-1.2008, 2016-07-20).

An unintended side-effect is that "cold-calling" into the POSIX
emulation will start with a locale based on the current code page,
something that Git for Windows is very ill-prepared for, as it expects
to be able to pass a command-line containing non-ASCII characters to the
shell without having those characters munged.

One symptom of this behavior: when `git clone` or `git fetch` shell out
to call `git-upload-pack` with a path that contains non-ASCII
characters, the shell tried to interpret the entire command-line
(including command-line parameters) as executable path, which obviously
must fail.

This fixes https://github.com/git-for-windows/git/issues/1036

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:29 +02:00
Jeff Hostetler
c2d87639ac clink.pl: fix MSVC compile script to handle libcurl-d.lib
Update clink.pl to link with either libcurl.lib or libcurl-d.lib
depending on whether DEBUG=1 is set.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:28 +02:00
Bjoern Mueller
d2d3eb18c6 mingw: fix fatal error working on mapped network drives on Windows
In 1e64d18 (mingw: do resolve symlinks in `getcwd()`) a problem was
introduced that causes git for Windows to stop working with certain
mapped network drives (in particular, drives that are mapped to
locations with long path names). Error message was "fatal: Unable to
read current working directory: No such file or directory". Present
change fixes this issue as discussed in
https://github.com/git-for-windows/git/issues/2480

Signed-off-by: Bjoern Mueller <bjoernm@gmx.de>
2023-08-21 20:10:28 +02:00
Johannes Schindelin
2f8fb13cec mingw: do resolve symlinks in getcwd()
As pointed out in https://github.com/git-for-windows/git/issues/1676,
the `git rev-parse --is-inside-work-tree` command currently fails when
the current directory's path contains symbolic links.

The underlying reason for this bug is that `getcwd()` is supposed to
resolve symbolic links, but our `mingw_getcwd()` implementation did not.

We do have all the building blocks for that, though: the
`GetFinalPathByHandleW()` function will resolve symbolic links. However,
we only called that function if `GetLongPathNameW()` failed, for
historical reasons: the latter function was supported for a long time,
but the former API function was introduced only with Windows Vista, and
we used to support also Windows XP. With that support having been
dropped, we are free to call the symbolic link-resolving function right
away.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:28 +02:00
Johannes Schindelin
5fed95d1a7 mingw: make sure errno is set correctly when socket operations fail
The winsock2 library provides functions that work on different data
types than file descriptors, therefore we wrap them.

But that is not the only difference: they also do not set `errno` but
expect the callers to enquire about errors via `WSAGetLastError()`.

Let's translate that into appropriate `errno` values whenever the socket
operations fail so that Git's code base does not have to change its
expectations.

This closes https://github.com/git-for-windows/git/issues/2404

Helped-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:28 +02:00
Johannes Schindelin
6c39656d23 strbuf_realpath(): use platform-dependent API if available
Some platforms (e.g. Windows) provide API functions to resolve paths
much quicker. Let's offer a way to short-cut `strbuf_realpath()` on
those platforms.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:28 +02:00
Johannes Schindelin
ba9ab72715 mingw: demonstrate a git add issue with NTFS junctions
NTFS junctions are somewhat similar in spirit to Unix bind mounts: they
point to a different directory and are resolved by the filesystem
driver. As such, they appear to `lstat()` as if they are directories,
not as if they are symbolic links.

_Any_ user can create junctions, while symbolic links can only be
created by non-administrators in Developer Mode on Windows 10. Hence
NTFS junctions are much more common "in the wild" than NTFS symbolic
links.

It was reported in https://github.com/git-for-windows/git/issues/2481
that adding files via an absolute path that traverses an NTFS junction:
since 1e64d18 (mingw: do resolve symlinks in `getcwd()`), we resolve not
only symbolic links but also NTFS junctions when determining the
absolute path of the current directory. The same is not true for `git
add <file>`, where symbolic links are resolved in `<file>`, but not NTFS
junctions.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:28 +02:00
Thomas Braun
3820ec911a transport: optionally disable side-band-64k
Since commit 0c499ea60f (send-pack: demultiplex a sideband stream with
status data, 2010-02-05) the send-pack builtin uses the side-band-64k
capability if advertised by the server.

Unfortunately this breaks pushing over the dump git protocol if used
over a network connection.

The detailed reasons for this breakage are (by courtesy of Jeff Preshing,
quoted from https://groups.google.com/d/msg/msysgit/at8D7J-h7mw/eaLujILGUWoJ):

	MinGW wraps Windows sockets in CRT file descriptors in order to
	mimic the functionality of POSIX sockets. This causes msvcrt.dll
	to treat sockets as Installable File System (IFS) handles,
	calling ReadFile, WriteFile, DuplicateHandle and CloseHandle on
	them. This approach works well in simple cases on recent
	versions of Windows, but does not support all usage patterns. In
	particular, using this approach, any attempt to read & write
	concurrently on the same socket (from one or more processes)
	will deadlock in a scenario where the read waits for a response
	from the server which is only invoked after the write. This is
	what send_pack currently attempts to do in the use_sideband
	codepath.

The new config option `sendpack.sideband` allows to override the
side-band-64k capability of the server, and thus makes the dumb git
protocol work.

Other transportation methods like ssh and http/https still benefit from
the sideband channel, therefore the default value of `sendpack.sideband`
is still true.

Signed-off-by: Thomas Braun <thomas.braun@byte-physics.de>
Signed-off-by: Oliver Schneider <oliver@assarbad.net>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:27 +02:00
Johannes Schindelin
c492df81c1 mingw: use mimalloc
Thorough benchmarking with repacking a subset of linux.git (the commit
history reachable from 93a6fefe2f ([PATCH] fix the SYSCTL=n compilation,
2007-02-28), to be precise) suggest that this allocator is on par, in
multi-threaded situations maybe even better than nedmalloc:

`git repack -adfq` with mimalloc, 8 threads:

31.166991900 27.576763800 28.712311000 27.373859000 27.163141900

`git repack -adfq` with nedmalloc, 8 threads:

31.915032900 27.149883100 28.244933700 27.240188800 28.580849500

In a different test using GitHub Actions build agents (probably
single-threaded, a core-strength of nedmalloc)):

`git repack -q -d -l -A --unpack-unreachable=2.weeks.ago` with mimalloc:

943.426 978.500 939.709 959.811 954.605

`git repack -q -d -l -A --unpack-unreachable=2.weeks.ago` with nedmalloc:

995.383 952.179 943.253 963.043 980.468

While these measurements were not executed with complete scientific
rigor, as no hardware was set aside specifically for these benchmarks,
it shows that mimalloc and nedmalloc perform almost the same, nedmalloc
with a bit higher variance and also slightly higher average (further
testing suggests that nedmalloc performs worse in multi-threaded
situations than in single-threaded ones).

In short: mimalloc seems to be slightly better suited for our purposes
than nedmalloc.

Seeing that mimalloc is developed actively, while nedmalloc ceased to
see any updates in eight years, let's use mimalloc on Windows instead.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:27 +02:00
Jeff Hostetler
d5c8653153 mimalloc: use "weak" random seed when statically linked
Always use the internal "use_weak" random seed when initializing
the "mimalloc" heap when statically linked on Windows.

The imported "mimalloc" routines support several random sources
to seed the heap data structures, including BCrypt.dll and
RtlGenRandom.  Crashes have been reported when using BCrypt.dll
if it initialized during an `atexit()` handler function.  Granted,
such DLL initialization should not happen in an atexit handler,
but yet the crashes remain.

It should be noted that on Windows when statically linked, the
mimalloc startup code (called by the GCC CRT to initialize static
data prior to calling `main()`) always uses the internal "weak"
random seed.  "mimalloc" does not try to load an alternate
random source until after the OS initialization has completed.

Heap data is stored in `__declspec(thread)` TLS data and in theory
each Git thread will have its own heap data.  However, testing
shows that the "mimalloc" library doesn't actually call
`os_random_buf()` (to load a new random source) when creating these
new per-thread heap structures.

However, if an atexit handler is forced to run on a non-main
thread, the "mimalloc" library *WILL* try to create a new heap
and seed it with `os_random_buf()`.  (The reason for this is still
a mystery to this author.)  The `os_random_buf()` call can cause
the (previously uninitialized BCrypt.dll library) to be dynamically
loaded and a call made into it.  Crashes have been reported in
v2.40.1.vfs.0.0 while in this call.

As a workaround, the fix here forces the use of the internal
"use_weak" random code for the subsequent `os_random_buf()` calls.
Since we have been using that random generator for the majority
of the program, it seems safe to use it for the final few mallocs
in the atexit handler (of which there really shouldn't be that many.

Signed-off-by: Jeff Hostetler <jeffhostetler@github.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:27 +02:00
Johannes Schindelin
c63f728b09 mimalloc: offer a build-time option to enable it
By defining `USE_MIMALLOC`, Git can now be compiled with that
nicely-fast and small allocator.

Note that we have to disable a couple `DEVELOPER` options to build
mimalloc's source code, as it makes heavy use of declarations after
statements, among other things that disagree with Git's conventions.

We even have to silence some GCC warnings in non-DEVELOPER mode. For
example, the `-Wno-array-bounds` flag is needed because in `-O2` builds,
trying to call `NtCurrentTeb()` (which `_mi_thread_id()` does on
Windows) causes the bogus warning about a system header, likely related
to https://sourceforge.net/p/mingw-w64/mailman/message/37674519/ and to
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99578:

C:/git-sdk-64-minimal/mingw64/include/psdk_inc/intrin-impl.h:838:1:
        error: array subscript 0 is outside array bounds of 'long long unsigned int[0]' [-Werror=array-bounds]
  838 | __buildreadseg(__readgsqword, unsigned __int64, "gs", "q")
      | ^~~~~~~~~~~~~~

Also: The `mimalloc` library uses C11-style atomics, therefore we must
require that standard when compiling with GCC if we want to use
`mimalloc` (instead of requiring "only" C99). This is what we do in the
CMake definition already, therefore this commit does not need to touch
`contrib/buildsystems/`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:27 +02:00
Johannes Schindelin
31131b0cda mimalloc: adjust for building inside Git
We want to compile mimalloc's source code as part of Git, rather than
requiring the code to be built as an external library: mimalloc uses a
CMake-based build, which is not necessarily easy to integrate into the
flavors of Git for Windows (which will be the main benefitting port).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:27 +02:00
Johannes Schindelin
f84ededca8 Import the source code of mimalloc v2.0.9
This commit imports mimalloc's source code as per v2.0.9, fetched from
the tag at https://github.com/microsoft/mimalloc.

The .c files are from the src/ subdirectory, and the .h files from the
include/ subdirectory. We will subsequently modify the source code to
accommodate building within Git's context.

Since we plan on using the `mi_*()` family of functions, we skip the
C++-specific source code, some POSIX compliant functions to interact
with mimalloc, and the code that wants to support auto-magic overriding
of the `malloc()` function (mimalloc-new-delete.h, alloc-posix.c,
mimalloc-override.h, alloc-override.c, alloc-override-osx.c,
alloc-override-win.c and static.c).

To appease the `check-whitespace` job of Git's Continuous Integration,
this commit was washed one time via `git rebase --whitespace=fix`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:27 +02:00
Johannes Schindelin
1bf01acbe8 git-compat-util: avoid redeclaring _DEFAULT_SOURCE
We are about to vendor in `mimalloc`'s source code which we will want to
include `git-compat-util.h` after defining that constant.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2023-08-21 20:10:27 +02:00