Commit Graph

111267 Commits

Author SHA1 Message Date
Johannes Schindelin
facdc882fe Merge pull request #994 from jeffhostetler/jeffhostetler/fscache_nfd
fscache: add not-found directory cache to fscache
2020-07-10 14:25:11 +02:00
Johannes Schindelin
d88446e5e7 Merge branch 'fscache' 2020-07-10 14:25:10 +02:00
Johannes Schindelin
37f1ce5637 Merge 'add-p-many-files'
This topic branch allows `add -p` and `add -i` with a large number of
files. It is kind of a hack that was never really meant to be
upstreamed. Let's see if we can do better in the built-in `add -p`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2020-07-10 14:25:09 +02:00
Johannes Schindelin
e94166e089 Merge branch 'gitk-and-git-gui-patches'
These are Git for Windows' Git GUI and gitk patches. We will have to
decide at some point what to do about them, but that's a little lower
priority (as Git GUI seems to be unmaintained for the time being, and
the gitk maintainer keeps a very low profile on the Git mailing list,
too).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2020-07-10 14:25:08 +02:00
Johannes Schindelin
11a4795929 Merge branch 'ready-for-upstream'
This is the branch thicket of patches in Git for Windows that are
considered ready for upstream. To keep them in a ready-to-submit shape,
they are kept as close to the beginning of the branch thicket as
possible.
2020-07-10 14:25:08 +02:00
Johannes Schindelin
6ec8028bd4 fscache: add a test for the dir-not-found optimization
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2020-07-10 14:22:26 +02:00
Karsten Blees
20da315e2b fscache: load directories only once
If multiple threads access a directory that is not yet in the cache, the
directory will be loaded by each thread. Only one of the results is added
to the cache, all others are leaked. This wastes performance and memory.

On cache miss, add a future object to the cache to indicate that the
directory is currently being loaded. Subsequent threads register themselves
with the future object and wait. When the first thread has loaded the
directory, it replaces the future object with the result and notifies
waiting threads.

Signed-off-by: Karsten Blees <blees@dcon.de>
2020-07-10 14:22:26 +02:00
Jeff Hostetler
7e56403ff2 fscache: remember not-found directories
Teach FSCACHE to remember "not found" directories.

This is a performance optimization.

FSCACHE is a performance optimization available for Windows.  It
intercepts Posix-style lstat() calls into an in-memory directory
using FindFirst/FindNext.  It improves performance on Windows by
catching the first lstat() call in a directory, using FindFirst/
FindNext to read the list of files (and attribute data) for the
entire directory into the cache, and short-cut subsequent lstat()
calls in the same directory.  This gives a major performance
boost on Windows.

However, it does not remember "not found" directories.  When STATUS
runs and there are missing directories, the lstat() interception
fails to find the parent directory and simply return ENOENT for the
file -- it does not remember that the FindFirst on the directory
failed. Thus subsequent lstat() calls in the same directory, each
re-attempt the FindFirst.  This completely defeats any performance
gains.

This can be seen by doing a sparse-checkout on a large repo and
then doing a read-tree to reset the skip-worktree bits and then
running status.

This change reduced status times for my very large repo by 60%.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2020-07-10 14:22:26 +02:00
Karsten Blees
1f19fe1202 mingw: add a cache below mingw's lstat and dirent implementations
Checking the work tree status is quite slow on Windows, due to slow
`lstat()` emulation (git calls `lstat()` once for each file in the
index). Windows operating system APIs seem to be much better at scanning
the status of entire directories than checking single files.

Add an `lstat()` implementation that uses a cache for lstat data. Cache
misses read the entire parent directory and add it to the cache.
Subsequent `lstat()` calls for the same directory are served directly
from the cache.

Also implement `opendir()`/`readdir()`/`closedir()` so that they create
and use directory listings in the cache.

The cache doesn't track file system changes and doesn't plug into any
modifying file APIs, so it has to be explicitly enabled for git functions
that don't modify the working copy.

Note: in an earlier version of this patch, the cache was always active and
tracked file system changes via ReadDirectoryChangesW. However, this was
much more complex and had negative impact on the performance of modifying
git commands such as 'git checkout'.

Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2020-07-10 14:22:26 +02:00
Jeff Hostetler
cb8185c5f3 fscache: add key for GIT_TRACE_FSCACHE
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2020-07-10 14:22:26 +02:00
Karsten Blees
e2c9916e79 add infrastructure for read-only file system level caches
Add a macro to mark code sections that only read from the file system,
along with a config option and documentation.

This facilitates implementation of relatively simple file system level
caches without the need to synchronize with the file system.

Enable read-only sections for 'git status' and preload_index.

Signed-off-by: Karsten Blees <blees@dcon.de>
2020-07-10 14:22:26 +02:00
Karsten Blees
9c5f3fe8a0 Win32: make the lstat implementation pluggable
Emulating the POSIX lstat API on Windows via GetFileAttributes[Ex] is quite
slow. Windows operating system APIs seem to be much better at scanning the
status of entire directories than checking single files. A caching
implementation may improve performance by bulk-reading entire directories
or reusing data obtained via opendir / readdir.

Make the lstat implementation pluggable so that it can be switched at
runtime, e.g. based on a config option.

Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2020-07-10 14:22:26 +02:00
Karsten Blees
5ac6390960 mingw: make the dirent implementation pluggable
Emulating the POSIX `dirent` API on Windows via
`FindFirstFile()`/`FindNextFile()` is pretty staightforward, however,
most of the information provided in the `WIN32_FIND_DATA` structure is
thrown away in the process. A more sophisticated implementation may
cache this data, e.g. for later reuse in calls to `lstat()`.

Make the `dirent` implementation pluggable so that it can be switched at
runtime, e.g. based on a config option.

Define a base DIR structure with pointers to `readdir()`/`closedir()`
that match the `opendir()` implementation (similar to vtable pointers in
Object-Oriented Programming). Define `readdir()`/`closedir()` so that
they call the function pointers in the `DIR` structure. This allows to
choose the `opendir()` implementation on a call-by-call basis.

Make the fixed-size `dirent.d_name` buffer a flex array, as `d_name` may
be implementation specific (e.g. a caching implementation may allocate a
`struct dirent` with _just_ the size needed to hold the `d_name` in
question).

Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2020-07-10 14:22:26 +02:00
Karsten Blees
34e5a8c37d Win32: dirent.c: Move opendir down
Move opendir down in preparation for the next patch.

Signed-off-by: Karsten Blees <blees@dcon.de>
2020-07-10 14:22:26 +02:00
Karsten Blees
42f2c23ab8 Win32: make FILETIME conversion functions public
We will use them in the upcoming "FSCache" patches (to accelerate
sequential lstat() calls).

Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2020-07-10 14:22:26 +02:00
Johannes Schindelin
1e91fc405b t3701: verify that we can add *lots* of files interactively
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2020-07-10 14:22:25 +02:00
Johannes Schindelin
3de0da06bd Merge 'gitk' into HEAD 2020-07-10 14:22:25 +02:00
Kelly Heller
f758db7d94 Allow add -p and add -i with a large number of files
This fixes https://github.com/msysgit/git/issues/182.

Inspired by Pull Request 218 using code from @PhilipDavis.

[jes: simplified code quite a bit]

Signed-off-by: Kelly Heller <kkheller@cedrus.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2020-07-10 14:22:25 +02:00
Johannes Schindelin
ab1cdfce07 Merge branch 'git-gui-askyesno'
These changes are necessary to support better Git for Windows' new
auto-update feature.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2020-07-10 14:22:24 +02:00
Johannes Schindelin
dcd3857ed9 Merge branch 'git-gui-hooks-path' of https://github.com/dscho/git-gui
Let's try to address #1755 this way.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2020-07-10 14:22:24 +02:00
Johannes Schindelin
ac7b0d1e7b Merge 'git-gui' into HEAD 2020-07-10 14:22:24 +02:00
James J. Raden
e5b9e0df7b gitk: make the "list references" default window width wider
When using remotes (with git-flow especially), the remote reference names
are almost always wordwrapped in the "list references" window because it's
somewhat narrow by default. It's possible to resize it with a mouse,
but it's annoying to have to do this every time, especially on Windows 10,
where the window border seems to be only one (1) pixel wide, thus making
the grabbing of the window border tricky.

Signed-off-by: James J. Raden <james.raden@gmail.com>
2020-07-10 14:22:23 +02:00
Johannes Schindelin
a2c16f0138 git-gui--askyesno (mingw): use Git for Windows' icon, if available
For additional GUI goodness.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2020-07-10 14:22:23 +02:00
Johannes Schindelin
3dd1c7c836 gitk: fix arrow keys in input fields with Tcl/Tk >= 8.6
Tcl/Tk 8.6 introduced new events for the cursor left/right keys and
apparently changed the behavior of the previous event.

Let's work around that by using the new events when we are running with
Tcl/Tk 8.6 or later.

This fixes https://github.com/git-for-windows/git/issues/495

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2020-07-10 14:22:23 +02:00
Johannes Schindelin
0945c77a8a git-gui--askyesno: allow overriding the window title
"Question?" is maybe not the most informative thing to ask. In the
absence of better information, it is the best we can do, of course.

However, Git for Windows' auto updater just learned the trick to use
git-gui--askyesno to ask the user whether to update now or not. And in
this scripted scenario, we can easily pass a command-line option to
change the window title.

So let's support that with the new `--title <title>` option.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2020-07-10 14:22:23 +02:00
Sebastian Schuberth
68c9c0522b gitk: Use an external icon file on Windows
Git for Windows now ships with the new Git icon from git-scm.com. Use that
icon file if it exists instead of the old procedurally drawn one.

This patch was sent upstream but so far no decision on its inclusion was
made, so commit it to our fork.

Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
2020-07-10 14:22:23 +02:00
Johannes Schindelin
28b4b7027b git-gui--askyesno: fix funny text wrapping
The text wrapping seems to be aligned to the right side of the Yes
button, leaving an awful lot of empty space.

Let's try to counter this by using pixel units.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2020-07-10 14:22:23 +02:00
Karsten Blees
fcb44872bf gitk: Unicode file name support
Assumes file names in git tree objects are UTF-8 encoded.

On most unix systems, the system encoding (and thus the TCL system
encoding) will be UTF-8, so file names will be displayed correctly.

On Windows, it is impossible to set the system encoding to UTF-8.
Changing the TCL system encoding (via 'encoding system ...', e.g. in the
startup code) is explicitly discouraged by the TCL docs.

Change gitk functions dealing with file names to always convert
from and to UTF-8.

Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2020-07-10 14:22:23 +02:00
Johannes Schindelin
f0ef5894b1 Merge pull request #2618 from dscho/avoid-d/f-conflict-in-vs/master
ci: avoid d/f conflict in vs/master
2020-07-10 14:22:22 +02:00
Johannes Schindelin
f58fd40ec2 respect core.hooksPath, falling back to .git/hooks
Since v2.9.0, Git knows about the config variable core.hookspath
that allows overriding the path to the directory containing the
Git hooks.

Since v2.10.0, the `--git-path` option respects that config
variable, too, so we may just as well use that command.

For Git versions older than v2.5.0 (which was the first version to
support the `--git-path` option for the `rev-parse` command), we
simply fall back to the previous code.

This fixes https://github.com/git-for-windows/git/issues/1755

Initial-patch-by: Philipp Gortan <philipp@gortan.org>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2020-07-10 14:22:22 +02:00
Johannes Schindelin
6ecf4d0c23 git gui: set GIT_ASKPASS=git-gui--askpass if not set yet
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2020-07-10 14:22:22 +02:00
Heiko Voigt
b65aededef git-gui: provide question helper for retry fallback on Windows
Make use of the new environment variable GIT_ASK_YESNO to support the
recently implemented fallback in case unlink, rename or rmdir fail for
files in use on Windows. The added dialog will present a yes/no question
to the the user which will currently be used by the windows compat layer
to let the user retry a failed file operation.

Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
2020-07-10 14:22:22 +02:00
Jeff Hostetler
39599a820b Merge pull request #2589 from jeffhostetler/unlink-perf-gfw
mingw: improve performance of mingw_unlink()
2020-07-10 14:22:21 +02:00
Johannes Schindelin
2a48df2396 Merge pull request #2549 from remybohmer/master
[fix] Gitk fails to open on a bare Git repository on Windows
2020-07-10 14:22:21 +02:00
Johannes Schindelin
0676888bb6 Merge pull request #2535 from dscho/schannel-revoke-best-effort
Introduce and use the new "best effort" strategy for Secure Channel revoke checking
2020-07-10 14:22:20 +02:00
Johannes Schindelin
4f0acb35fc Merge pull request #2516 from nyckyta/crlf-aware-git-credential
credential.c: fix credential reading with regards to CR/LF
2020-07-10 14:22:20 +02:00
Johannes Schindelin
46132bf38b Merge pull request #2506 from dscho/issue-2283
Allow running Git directly from `C:\Program Files\Git\mingw64\bin\git.exe`
2020-07-10 14:22:19 +02:00
Johannes Schindelin
2facb08be7 Merge pull request #2504 from dscho/access-repo-via-junction
Handle `git add <file>` where <file> traverses an NTFS junction
2020-07-10 14:22:19 +02:00
Johannes Schindelin
2bb6cd763a Merge pull request #2501 from jeffhostetler/clink-debug-curl
clink.pl: fix MSVC compile script to handle libcurl-d.lib
2020-07-10 14:22:18 +02:00
Johannes Schindelin
a14864ca60 Merge pull request #2488 from bmueller84/master
mingw: fix fatal error working on mapped network drives on Windows
2020-07-10 14:22:18 +02:00
Johannes Schindelin
2a13d40c59 Merge pull request #2449 from dscho/mingw-getcwd-and-symlinks
Do resolve symlinks in `getcwd()`
2020-07-10 14:22:17 +02:00
Johannes Schindelin
81b03da160 Merge pull request #2405 from dscho/mingw-setsockopt
Make sure `errno` is set when socket operations fail
2020-07-10 14:22:17 +02:00
Johannes Schindelin
8e0c19f5d6 Merge pull request #2375 from assarbad/reintroduce-sideband-config
Config option to disable side-band-64k for transport
2020-07-10 14:22:16 +02:00
Johannes Schindelin
7eff9756b4 Merge branch 'msys2-python'
In MSYS2, we have two Python interpreters at our disposal, so we can
include the Python stuff in the build.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2020-07-10 14:22:16 +02:00
Johannes Schindelin
d1669f04db Merge branch 'move-gfw-system-config-to-top-level'
Make `git config --system` work like you think it should on Windows: it
should edit the file that is located in `<Git>\etc\gitconfig`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2020-07-10 14:22:15 +02:00
Johannes Schindelin
af4aad05b7 Merge pull request #2351 from PhilipOakley/vcpkg-tip
Vcpkg Install: detect lack of working Git, and note possible vcpkg time outs
2020-07-10 14:22:15 +02:00
Johannes Schindelin
2411eb6615 Merge pull request #2316 from carenas/win-pcre1-cleanup
config.mak.uname: PCRE1 cleanup
2020-07-10 14:22:14 +02:00
Johannes Schindelin
8663d8be84 Merge branch 'dont-clean-junctions'
This topic branch teaches `git clean` to respect NTFS junctions and Unix
bind mounts: it will now stop at those boundaries.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2020-07-10 14:22:14 +02:00
Johannes Schindelin
888deab073 Merge pull request #2170 from dscho/gitk-long-cmdline
Fix gitk (long cmdline)
2020-07-10 14:22:13 +02:00
Johannes Schindelin
716404a96c Merge branch 'fsync-object-files-always'
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2020-07-10 14:22:13 +02:00