These are Git for Windows' Git GUI and gitk patches. We will have to
decide at some point what to do about them, but that's a little lower
priority (as Git GUI seems to be unmaintained for the time being, and
the gitk maintainer keeps a very low profile on the Git mailing list,
too).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When trying to ensure that long paths are handled correctly, we
first normalize absolute paths as we encounter them.
However, if the path is a so-called "drive-less" absolute path, i.e. if
it is relative to the current drive but _does_ start with a directory
separator, we would want the normalized path to be such a drive-less
absolute path, too.
Let's do that, being careful to still include the drive prefix when we
need to go through the `\\?\` dance (because there, the drive prefix is
absolutely required).
This fixes https://github.com/git-for-windows/git/issues/4586.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We already avoid traversing NTFS junction points in `git clean -dfx`.
With this topic branch, we do that when the FSCache is enabled, too.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Windows paths are typically limited to MAX_PATH = 260 characters, even
though the underlying NTFS file system supports paths up to 32,767 chars.
This limitation is also evident in Windows Explorer, cmd.exe and many
other applications (including IDEs).
Particularly annoying is that most Windows APIs return bogus error codes
if a relative path only barely exceeds MAX_PATH in conjunction with the
current directory, e.g. ERROR_PATH_NOT_FOUND / ENOENT instead of the
infinitely more helpful ERROR_FILENAME_EXCED_RANGE / ENAMETOOLONG.
Many Windows wide char APIs support longer than MAX_PATH paths through the
file namespace prefix ('\\?\' or '\\?\UNC\') followed by an absolute path.
Notable exceptions include functions dealing with executables and the
current directory (CreateProcess, LoadLibrary, Get/SetCurrentDirectory) as
well as the entire shell API (ShellExecute, SHGetSpecialFolderPath...).
Introduce a handle_long_path function to check the length of a specified
path properly (and fail with ENAMETOOLONG), and to optionally expand long
paths using the '\\?\' file namespace prefix. Short paths will not be
modified, so we don't need to worry about device names (NUL, CON, AUX).
Contrary to MSDN docs, the GetFullPathNameW function doesn't seem to be
limited to MAX_PATH (at least not on Win7), so we can use it to do the
heavy lifting of the conversion (translate '/' to '\', eliminate '.' and
'..', and make an absolute path).
Add long path error checking to xutftowcs_path for APIs with hard MAX_PATH
limit.
Add a new MAX_LONG_PATH constant and xutftowcs_long_path function for APIs
that support long paths.
While improved error checking is always active, long paths support must be
explicitly enabled via 'core.longpaths' option. This is to prevent end
users to shoot themselves in the foot by checking out files that Windows
Explorer, cmd/bash or their favorite IDE cannot handle.
Test suite:
Test the case is when the full pathname length of a dir is close
to 260 (MAX_PATH).
Bug report and an original reproducer by Andrey Rogozhnikov:
https://github.com/msysgit/git/pull/122#issuecomment-43604199
[jes: adjusted test number to avoid conflicts, added support for
chdir(), etc]
Thanks-to: Martin W. Kirst <maki@bitkings.de>
Thanks-to: Doug Kelly <dougk.ff7@gmail.com>
Original-test-by: Andrey Rogozhnikov <rogozhnikov.andrey@gmail.com>
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Stepan Kasal <kasal@ucw.cz>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
There is a problem in the way 9ac3f0e5b3 (pack-objects: fix
performance issues on packing large deltas, 2018-07-22) initializes that
mutex in the `packing_data` struct. The problem manifests in a
segmentation fault on Windows, when a mutex (AKA critical section) is
accessed without being initialized. (With pthreads, you apparently do
not really have to initialize them?)
This was reported in https://github.com/git-for-windows/git/issues/1839.
Signed-off-by: Doug Kelly <dougk.ff7@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When updating the skip-worktree bits in the index to align with new
values in a sparse-checkout file, Git scans the entire working
directory with lstat() calls. In a sparse-checkout, many of these
lstat() calls are for paths that do not exist.
Enable the fscache feature during this scan.
In a local test of a repo with ~2.2 million paths, updating the index
with `git read-tree -m -u HEAD` with a sparse-checkout file containing
only `/.gitattributes` improved from 2-3 minutes to 15-20 seconds.
More work could be done to stop running lstat() calls when recursing
into directories that are known to not exist.
This brings substantial wins in performance because the FSCache is now
per-thread, being merged to the primary thread only at the end, so we do
not have to lock (except while merging).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
These changes are necessary to support better Git for Windows' new
auto-update feature.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When using remotes (with git-flow especially), the remote reference names
are almost always wordwrapped in the "list references" window because it's
somewhat narrow by default. It's possible to resize it with a mouse,
but it's annoying to have to do this every time, especially on Windows 10,
where the window border seems to be only one (1) pixel wide, thus making
the grabbing of the window border tricky.
Signed-off-by: James J. Raden <james.raden@gmail.com>
Tcl/Tk 8.6 introduced new events for the cursor left/right keys and
apparently changed the behavior of the previous event.
Let's work around that by using the new events when we are running with
Tcl/Tk 8.6 or later.
This fixes https://github.com/git-for-windows/git/issues/495
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
"Question?" is maybe not the most informative thing to ask. In the
absence of better information, it is the best we can do, of course.
However, Git for Windows' auto updater just learned the trick to use
git-gui--askyesno to ask the user whether to update now or not. And in
this scripted scenario, we can easily pass a command-line option to
change the window title.
So let's support that with the new `--title <title>` option.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Git for Windows now ships with the new Git icon from git-scm.com. Use that
icon file if it exists instead of the old procedurally drawn one.
This patch was sent upstream but so far no decision on its inclusion was
made, so commit it to our fork.
Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
Assumes file names in git tree objects are UTF-8 encoded.
On most unix systems, the system encoding (and thus the TCL system
encoding) will be UTF-8, so file names will be displayed correctly.
On Windows, it is impossible to set the system encoding to UTF-8.
Changing the TCL system encoding (via 'encoding system ...', e.g. in the
startup code) is explicitly discouraged by the TCL docs.
Change gitk functions dealing with file names to always convert
from and to UTF-8.
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The `git clean` command needs to enumerate plenty of files and
directories, and can therefore benefit from the FSCache.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When updating the skip-worktree bits in the index to align with new
values in a sparse-checkout file, Git scans the entire working
directory with lstat() calls. In a sparse-checkout, many of these
lstat() calls are for paths that do not exist.
Enable the fscache feature during this scan. Since enable_fscache()
calls nest, the disable_fscache() method decrements a counter and
would only clear the cache if that counter reaches zero.
In a local test of a repo with ~2.2 million paths, updating the index
with git read-tree -m -u HEAD with a sparse-checkout file containing
only /.gitattributes improved from 2-3 minutes to ~6 seconds.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Using FindFirstFileExW() requires the OS to allocate a 64K buffer for each
directory and then free it when we call FindClose(). Update fscache to call
the underlying kernel API NtQueryDirectoryFile so that we can do the buffer
management ourselves. That allows us to allocate a single buffer for the
lifetime of the cache and reuse it for each directory.
This change improves performance of 'git status' by 18% in a repo with ~200K
files and 30k folders.
Documentation for NtQueryDirectoryFile can be found at:
https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/content/ntifs/nf-ntifs-ntquerydirectoryfilehttps://docs.microsoft.com/en-us/windows/desktop/FileIO/file-attribute-constantshttps://docs.microsoft.com/en-us/windows/desktop/fileio/reparse-point-tags
To determine if the specified directory is a symbolic link, inspect the
FileAttributes member to see if the FILE_ATTRIBUTE_REPARSE_POINT flag is
set. If so, EaSize will contain the reparse tag (this is a so far
undocumented feature, but confirmed by the NTFS developers). To
determine if the reparse point is a symbolic link (and not some other
form of reparse point), test whether the tag value equals the value
IO_REPARSE_TAG_SYMLINK.
The NtQueryDirectoryFile() call works best (and on Windows 8.1 and
earlier, it works *only*) with buffer sizes up to 64kB. Which is 32k
wide characters, so let's use that as our buffer size.
Signed-off-by: Ben Peart <benpeart@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The recent change to make fscache thread specific relied on fscache_enable()
being called first from the primary thread before being called in parallel
from worker threads. Make that more robust and protect it with a critical
section to avoid any issues.
Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Ben Peart <benpeart@microsoft.com>
Now that the fscache is single threaded, take advantage of the mem_pool as
the allocator to significantly reduce the cost of allocations and frees.
With the reduced cost of free, in future patches, we can start freeing the
fscache at the end of commands instead of just leaking it.
Signed-off-by: Ben Peart <benpeart@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The text wrapping seems to be aligned to the right side of the Yes
button, leaving an awful lot of empty space.
Let's try to counter this by using pixel units.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When FSCache is active, we can cache the reparse tag and use it directly
to determine whether a path refers to an NTFS junction, without any
additional, costly I/O.
Note: this change only makes a difference with the next commit, which
will make use of the FSCache in `git clean` (contingent on
`core.fscache` set, of course).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Make use of the new environment variable GIT_ASK_YESNO to support the
recently implemented fallback in case unlink, rename or rmdir fail for
files in use on Windows. The added dialog will present a yes/no question
to the the user which will currently be used by the windows compat layer
to let the user retry a failed file operation.
Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
We will use this in the next commit to implement an FSCache-aware
version of is_mount_point().
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Track fscache hits and misses for lstat and opendir requests. Reporting of
statistics is done when the cache is disabled for the last time and freed
and is only reported if GIT_TRACE_FSCACHE is set.
Sample output is:
11:33:11.836428 compat/win32/fscache.c:433 fscache: lstat 3775, opendir 263, total requests/misses 4052/269
Signed-off-by: Ben Peart <benpeart@microsoft.com>
At the end of the status command, disable and free the fscache so that we
don't leak the memory and so that we can dump the fscache statistics.
Signed-off-by: Ben Peart <benpeart@microsoft.com>
Use FindFirstFileExW with FindExInfoBasic to avoid forcing NTFS to look up
the short name. Also switch to a larger (64K vs 4K) buffer using
FIND_FIRST_EX_LARGE_FETCH to minimize round trips to the kernel.
In a repo with ~200K files, this drops warm cache status times from 3.19
seconds to 2.67 seconds for a 16% savings.
Signed-off-by: Ben Peart <benpeart@microsoft.com>
On file systems that support it, this can dramatically speed up operations
like add, commit, describe, rebase, reset, rm that would otherwise have to
lstat() every file to "re-match" the stat information in the index to that
of the file system.
On a synthetic repo with 1M files, "git reset" dropped from 52.02 seconds to
14.42 seconds for a savings of 72%.
Signed-off-by: Ben Peart <benpeart@microsoft.com>
This is retry of #1419.
I added flush_fscache macro to flush cached stats after disk writing
with tests for regression reported in #1438 and #1442.
git checkout checks each file path in sorted order, so cache flushing does not
make performance worse unless we have large number of modified files in
a directory containing many files.
Using chromium repository, I tested `git checkout .` performance when I
delete 10 files in different directories.
With this patch:
TotalSeconds: 4.307272
TotalSeconds: 4.4863595
TotalSeconds: 4.2975562
Avg: 4.36372923333333
Without this patch:
TotalSeconds: 20.9705431
TotalSeconds: 22.4867685
TotalSeconds: 18.8968292
Avg: 20.7847136
I confirmed this patch passed all tests in t/ with core_fscache=1.
Signed-off-by: Takuto Ikuta <tikuta@chromium.org>
When I do git fetch, git call file stats under .git/objects for each
refs. This takes time when there are many refs.
By enabling fscache, git takes file stats by directory traversing and that
improved the speed of fetch-pack for repository having large number of
refs.
In my windows workstation, this improves the time of `git fetch` for
chromium repository like below. I took stats 3 times.
* With this patch
TotalSeconds: 9.9825165
TotalSeconds: 9.1862075
TotalSeconds: 10.1956256
Avg: 9.78811653333333
* Without this patch
TotalSeconds: 15.8406702
TotalSeconds: 15.6248053
TotalSeconds: 15.2085938
Avg: 15.5580231
Signed-off-by: Takuto Ikuta <tikuta@chromium.org>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The threading model for fscache has been to have a single, global cache.
This puts requirements on it to be thread safe so that callers like
preload-index can call it from multiple threads. This was implemented
with a single mutex and completion events which introduces contention
between the calling threads.
Simplify the threading model by making fscache thread specific. This allows
us to remove the global mutex and synchronization events entirely and instead
associate a fscache with every thread that requests one. This works well with
the current multi-threading which divides the cache entries into blocks with
a separate thread processing each block.
At the end of each worker thread, if there is a fscache on the primary
thread, merge the cached results from the worker into the primary thread
cache. This enables us to reuse the cache later especially when scanning for
untracked files.
In testing, this reduced the time spent in preload_index() by about 25% and
also reduced the CPU utilization significantly. On a repo with ~200K files,
it reduced overall status times by ~12%.
Signed-off-by: Ben Peart <benpeart@microsoft.com>
Update enable_fscache() to take an optional initial size parameter which is
used to initialize the hashmap so that it can avoid having to rehash as
additional entries are added.
Add a separate disable_fscache() macro to make the code clearer and easier
to read.
Signed-off-by: Ben Peart <benpeart@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
