Teach FSCACHE to remember "not found" directories.
This is a performance optimization.
FSCACHE is a performance optimization available for Windows. It
intercepts Posix-style lstat() calls into an in-memory directory
using FindFirst/FindNext. It improves performance on Windows by
catching the first lstat() call in a directory, using FindFirst/
FindNext to read the list of files (and attribute data) for the
entire directory into the cache, and short-cut subsequent lstat()
calls in the same directory. This gives a major performance
boost on Windows.
However, it does not remember "not found" directories. When STATUS
runs and there are missing directories, the lstat() interception
fails to find the parent directory and simply return ENOENT for the
file -- it does not remember that the FindFirst on the directory
failed. Thus subsequent lstat() calls in the same directory, each
re-attempt the FindFirst. This completely defeats any performance
gains.
This can be seen by doing a sparse-checkout on a large repo and
then doing a read-tree to reset the skip-worktree bits and then
running status.
This change reduced status times for my very large repo by 60%.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
If multiple threads access a directory that is not yet in the cache, the
directory will be loaded by each thread. Only one of the results is added
to the cache, all others are leaked. This wastes performance and memory.
On cache miss, add a future object to the cache to indicate that the
directory is currently being loaded. Subsequent threads register themselves
with the future object and wait. When the first thread has loaded the
directory, it replaces the future object with the result and notifies
waiting threads.
Signed-off-by: Karsten Blees <blees@dcon.de>
Checking the work tree status is quite slow on Windows, due to slow
`lstat()` emulation (git calls `lstat()` once for each file in the
index). Windows operating system APIs seem to be much better at scanning
the status of entire directories than checking single files.
Add an `lstat()` implementation that uses a cache for lstat data. Cache
misses read the entire parent directory and add it to the cache.
Subsequent `lstat()` calls for the same directory are served directly
from the cache.
Also implement `opendir()`/`readdir()`/`closedir()` so that they create
and use directory listings in the cache.
The cache doesn't track file system changes and doesn't plug into any
modifying file APIs, so it has to be explicitly enabled for git functions
that don't modify the working copy.
Note: in an earlier version of this patch, the cache was always active and
tracked file system changes via ReadDirectoryChangesW. However, this was
much more complex and had negative impact on the performance of modifying
git commands such as 'git checkout'.
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Add a macro to mark code sections that only read from the file system,
along with a config option and documentation.
This facilitates implementation of relatively simple file system level
caches without the need to synchronize with the file system.
Enable read-only sections for 'git status' and preload_index.
Signed-off-by: Karsten Blees <blees@dcon.de>
Emulating the POSIX lstat API on Windows via GetFileAttributes[Ex] is quite
slow. Windows operating system APIs seem to be much better at scanning the
status of entire directories than checking single files. A caching
implementation may improve performance by bulk-reading entire directories
or reusing data obtained via opendir / readdir.
Make the lstat implementation pluggable so that it can be switched at
runtime, e.g. based on a config option.
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Emulating the POSIX `dirent` API on Windows via
`FindFirstFile()`/`FindNextFile()` is pretty staightforward, however,
most of the information provided in the `WIN32_FIND_DATA` structure is
thrown away in the process. A more sophisticated implementation may
cache this data, e.g. for later reuse in calls to `lstat()`.
Make the `dirent` implementation pluggable so that it can be switched at
runtime, e.g. based on a config option.
Define a base DIR structure with pointers to `readdir()`/`closedir()`
that match the `opendir()` implementation (similar to vtable pointers in
Object-Oriented Programming). Define `readdir()`/`closedir()` so that
they call the function pointers in the `DIR` structure. This allows to
choose the `opendir()` implementation on a call-by-call basis.
Make the fixed-size `dirent.d_name` buffer a flex array, as `d_name` may
be implementation specific (e.g. a caching implementation may allocate a
`struct dirent` with _just_ the size needed to hold the `d_name` in
question).
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We will use them in the upcoming "FSCache" patches (to accelerate
sequential lstat() calls).
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Since Git LFS v3.5.x implicitly dropped Windows 7 support, we now want
users to be advised _what_ is going wrong on that Windows version. This
topic branch goes out of its way to provide users with such guidance.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Another (hopefully clean) PR for showing the error warning about atomic
append on windows after failure on APFS, which returns EBADF not EINVAL.
Signed-off-by: David Lomas <dl3@pale-eds.co.uk>
Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
As per
https://github.com/git-for-windows/git/issues/4350#issuecomment-1485041503,
the major block for upgrading Git for Windows' OpenSSL from v1.1 to v3
is the tricky part where such an upgrade would break `git fetch`/`git
clone` and `git push` because the libcurl depends on the OpenSSL DLL,
and the major version bump will _change_ the file name of said DLL.
To overcome that, the plan is to build libcurl flavors for each
supported SSL/TLS backend, aligning with the way MSYS2 builds libcurl,
then switch Git for Windows' SDK to the Secure Channel-flavored libcurl,
and teach Git to look for the specific flavor of libcurl corresponding
to the `http.sslBackend` setting (if that was configured).
Here is the PR to teach Git that trick.
The first three commits are rebased versions of those in gitgitgadget/git#1215. These allow the following:
1. Fix `git config --global foo.bar <path>` from allowing the `<path>`. As a bonus, users with a config value starting with `/` will not get a warning about "old-style" paths needing a "`%(prefix)/`".
2. When in WSL, the path starts with `/` so it needs to be interpolated properly. Update the warning to include `%(prefix)/` to get the right value for WSL users. (This is specifically for using Git for Windows from Git Bash, but in a WSL directory.)
3. When using WSL, the ownership check fails and reports an error message. This is noisy, and happens even if the user has marked the path with `safe.directory`. Remove that error message.
This topic vendors in mimalloc v2.0.9, a fast allocator that allows Git
for Windows to perform efficiently.
Switch Git for Windows to using mimalloc instead of nedmalloc
This topic branch teaches `git clean` to respect NTFS junctions and Unix
bind mounts: it will now stop at those boundaries.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This comment has been true for the longest time; The combination of the
two preceding commits made it incorrect, so let's drop that comment.
Signed-off-by: Matthias Aßhauer <mha1993@live.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We map WSAGetLastError() errors to errno errors in winsock_error_to_errno(),
but the MSVC strerror() implementation only produces "Unknown error" for
most of them. Produce some more meaningful error messages in these
cases.
Our builds for ARM64 link against the newer UCRT strerror() that does know
these errors, so we won't change the strerror() used there.
The wording of the messages is copied from glibc strerror() messages.
Reported-by: M Hickford <mirth.hickford@gmail.com>
Signed-off-by: Matthias Aßhauer <mha1993@live.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The winsock2 library provides functions that work on different data
types than file descriptors, therefore we wrap them.
But that is not the only difference: they also do not set `errno` but
expect the callers to enquire about errors via `WSAGetLastError()`.
Let's translate that into appropriate `errno` values whenever the socket
operations fail so that Git's code base does not have to change its
expectations.
This closes https://github.com/git-for-windows/git/issues/2404
Helped-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Git LFS is now built with Go 1.21 which no longer supports Windows 7.
However, Git for Windows still wants to support Windows 7.
Ideally, Git LFS would re-introduce Windows 7 support until Git for
Windows drops support for Windows 7, but that's not going to happen:
https://github.com/git-for-windows/git/issues/4996#issuecomment-2176152565
The next best thing we can do is to let the users know what is
happening, and how to get out of their fix, at least.
This is not quite as easy as it would first seem because programs
compiled with Go 1.21 or newer will simply throw an exception and fail
with an Access Violation on Windows 7.
The only way I found to address this is to replicate the logic from Go's
very own `version` command (which can determine the Go version with
which a given executable was built) to detect the situation, and in that
case offer a helpful error message.
This addresses https://github.com/git-for-windows/git/issues/4996.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Windows 10 version 1511 (also known as Anniversary Update), according to
https://learn.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences
introduced native support for ANSI sequence processing. This allows
using colors from the entire 24-bit color range.
All we need to do is test whether the console's "virtual processing
support" can be enabled. If it can, we do not even need to start the
`console_thread` to handle ANSI sequences.
Or, almost all we need to do: When `console_thread()` does its work, it
uses the Unicode-aware `write_console()` function to write to the Win32
Console, which supports Git for Windows' implicit convention that all
text that is written is encoded in UTF-8. The same is not necessarily
true if native ANSI sequence processing is used, as the output is then
subject to the current code page. Let's ensure that the code page is set
to `CP_UTF8` as long as Git writes to it.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When running Git for Windows on a remote APFS filesystem, it would
appear that the `mingw_open_append()`/`write()` combination would fail
almost exactly like on some CIFS-mounted shares as had been reported in
https://github.com/git-for-windows/git/issues/2753, albeit with a
different `errno` value.
Let's handle that `errno` value just the same, by suggesting to set
`windows.appendAtomically=false`.
Signed-off-by: David Lomas <dl3@pale-eds.co.uk>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The previous commits introduced a compile-time option to load libcurl
lazily, but it uses the hard-coded name "libcurl-4.dll" (or equivalent
on platforms other than Windows).
To allow for installing multiple libcurl flavors side by side, where
each supports one specific SSL/TLS backend, let's first look whether
`libcurl-<backend>-4.dll` exists, and only use `libcurl-4.dll` as a fall
back.
That will allow us to ship with a libcurl by default that only supports
the Secure Channel backend for the `https://` protocol. This libcurl
won't suffer from any dependency problem when upgrading OpenSSL to a new
major version (which will change the DLL name, and hence break every
program and library that depends on it).
This is crucial because Git for Windows relies on libcurl to keep
working when building and deploying a new OpenSSL package because that
library is used by `git fetch` and `git clone`.
Note that this feature is by no means specific to Windows. On Ubuntu,
for example, a `git` built using `LAZY_LOAD_LIBCURL` will use
`libcurl.so.4` for `http.sslbackend=openssl` and `libcurl-gnutls.so.4`
for `http.sslbackend=gnutls`.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This implements the Windows-specific support code, because everything is
slightly different on Windows, even loading shared libraries.
Note: I specifically do _not_ use the code from
`compat/win32/lazyload.h` here because that code is optimized for
loading individual functions from various system DLLs, while we
specifically want to load _many_ functions from _one_ DLL here, and
distinctly not a system DLL (we expect libcurl to be located outside
`C:\Windows\system32`, something `INIT_PROC_ADDR` refuses to work with).
Also, the `curl_easy_getinfo()`/`curl_easy_setopt()` functions are
declared as vararg functions, which `lazyload.h` cannot handle. Finally,
we are about to optionally override the exact file name that is to be
loaded, which is a goal contrary to `lazyload.h`'s design.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This compile-time option allows to ask Git to load libcurl dynamically
at runtime.
Together with a follow-up patch that optionally overrides the file name
depending on the `http.sslBackend` setting, this kicks open the door for
installing multiple libcurl flavors side by side, and load the one
corresponding to the (runtime-)configured SSL/TLS backend.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Newer compiler versions, like GCC 10 and Clang 12, have built-in
functions for bswap32 and bswap64. This comes in handy, for example,
when targeting CLANGARM64 on Windows, which would not be supported
without this logic.
Signed-off-by: Dennis Ameling <dennis@dennisameling.com>
From the documentation of said setting:
This boolean will enable fsync() when writing object files.
This is a total waste of time and effort on a filesystem that
orders data writes properly, but can be useful for filesystems
that do not use journalling (traditional UNIX filesystems) or
that only journal metadata and not file contents (OS X’s HFS+,
or Linux ext3 with "data=writeback").
The most common file system on Windows (NTFS) does not guarantee that
order, therefore a sudden loss of power (or any other event causing an
unclean shutdown) would cause corrupt files (i.e. files filled with
NULs). Therefore we need to change the default.
Note that the documentation makes it sound as if this causes really bad
performance. In reality, writing loose objects is something that is done
only rarely, and only a handful of files at a time.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Atomic append on windows is only supported on local disk files, and it may
cause errors in other situations, e.g. network file system. If that is the
case, this config option should be used to turn atomic append off.
Co-Authored-By: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: 孙卓识 <sunzhuoshi@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
NtQueryObject under Wine can return a success but fill out no name.
In those situations, Wine will set Buffer to NULL, and set result to
the sizeof(OBJECT_NAME_INFORMATION).
Running a command such as
echo "$(git.exe --version 2>/dev/null)"
will crash due to a NULL pointer dereference when the code attempts to
null terminate the buffer, although, weirdly, removing the subshell or
redirecting stdout to a file will not trigger the crash.
Code has been added to also check Buffer and Length to ensure the check
is as robust as possible due to the current behavior being fragile at
best, and could potentially change in the future
This code is based on the behavior of NtQueryObject under wine and
reactos.
Signed-off-by: Christopher Degawa <ccom@randomderp.com>
For Windows builds >= 15063 set $env:TERM to "xterm-256color" instead of
"cygwin" because they have a more capable console system that supports
this. Also set $env:COLORTERM="truecolor" if unset.
$env:TERM is initialized so that ANSI colors in color.c work, see
29a3963484 (Win32: patch Windows environment on startup, 2012-01-15).
See git-for-windows/git#3629 regarding problems caused by always setting
$env:TERM="cygwin".
This is the same heuristic used by the Cygwin runtime.
Signed-off-by: Rafael Kitover <rkitover@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
In the case of Git for Windows (say, in a Git Bash window) running in a
Windows Subsystem for Linux (WSL) directory, the GetNamedSecurityInfoW()
call in is_path_owned_By_current_side() returns an error code other than
ERROR_SUCCESS. This is consistent behavior across this boundary.
In these cases, the owner would always be different because the WSL
owner is a different entity than the Windows user.
The change here is to suppress the error message that looks like this:
error: failed to get owner for '//wsl.localhost/...' (1)
Before this change, this warning happens for every Git command,
regardless of whether the directory is marked with safe.directory.
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
We used to have that `make vcxproj` hack, but a hack it is. In the
meantime, we have a much cleaner solution: using CMake, either
explicitly, or even more conveniently via Visual Studio's built-in CMake
support (simply open Git's top-level directory via File>Open>Folder...).
Let's let the `README` reflect this.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
