This merges the current version of the patch that tries to address Git
GUI's problems with intent-to-add files.
This patch will likely be improved substantially before it is merged
into Git GUI's main branch, but we want to have _something_ resembling a
fix already in Git for Windows v2.29.0.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This topic branch allows `add -p` and `add -i` with a large number of
files. It is kind of a hack that was never really meant to be
upstreamed. Let's see if we can do better in the built-in `add -p`.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
In MSYS2, we have two Python interpreters at our disposal, so we can
include the Python stuff in the build.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Make `git config --system` work like you think it should on Windows: it
should edit the file that is located in `<Git>\etc\gitconfig`.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This topic branch teaches `git clean` to respect NTFS junctions and Unix
bind mounts: it will now stop at those boundaries.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This topic branch avoids spawning `gzip` when asking `git archive` to
create `.tar.gz` files.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When compiling Git with a runtime prefix (so that it can be installed
into any location, finding its libexec/ directory relative to the
location of the `git` executable), it is convenient to provide
"absolute" Unix-y paths e.g. for http.sslCAInfo, and have those absolute
paths be resolved relative to the runtime prefix.
This patch makes it so for Windows. It is up for discussion whether we
want this for other platforms, too, as long as building with
RUNTIME_PREFIX.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This topic branch allows us to specify absolute paths without the drive
prefix e.g. when cloning.
Example:
C:\Users\me> git clone https://github.com/git/git \upstream-git
This will clone into a new directory C:\upstream-git, in line with how
Windows interprets absolute paths.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
These fixes were necessary for Sverre Rabbelier's remote-hg to work,
but for some magic reason they are not necessary for the current
remote-hg. Makes you wonder how that one gets away with it.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The entire point of the FSMonitor is to monitor the worktree changes in
a more efficient manner than `lstat()`ing all worktree files every time
we refresh the index.
But if there is no worktree, FSMonitor has nothing to monitor.
So let's ignore if an FSMonitor is configured (e.g. in `~/.gitconfig`)
and we're running in a repository without worktree.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
If `feature.experimental` and `feature.manyFiles` are set, we now start
the built-in FSMonitor by default.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Builtin FSMonitor Feature
Here is V2 of my patch series to add a builtin FSMonitor daemon to Git.
V2 includes addresses most of the review comments from the RFC and many of
the issues seen during out beta-testing with selected GVFS users. However,
there are still a few items that I need to address:
[ ] Revisit the how the client handles the IPC_STATE__NOT_LISTENING state
(where a daemon appears to be running, but is non-responsive) [ ] Revisit
use of global core_fsmonitor as both a pathname and a boolean. The existing
fsmonitor code uses it as the pathname to the fsmonitor hook and as a flag
to indicate that a hook is configured. [ ] Consider having daemon chdir()
out of the working directory to avoid directory handle issues on Windows. [
] Some documentation recommendations. [ ] Split up the commit containing the
tests and move some earlier in patch series. [ ] Move my FSMonitor PREREQ to
test-lib.sh instead of having it in my test scripts. [ ] Document
performance gains. [ ] On Windows, If the daemon is started as an elevated
process, then client commands might not have access to communicate with it.
[ ] Review if/how we decide to shutdown the FSMonitor daemon after and a
significant idle period. [ ] Investigate ways to temporarily shutdown
FSMonitor daemons processes so that the Git for Windows installer can
install an upgrade.
In this version, the first commit updates the Simple IPC API to make it
easier to pass binary data using {char *, size_t} rather than assuming that
the message is a null-terminated string. FSMonitor does not use binary
messages and doesn't really need this API change, but I thought it best to
fix the API now before we have other callers of IPC.
This V2 version will hopefully be previewed as an experimental feature in
Git for Windows v2.32.0.windows.*.
Jeff Hostetler (26):
simple-ipc: preparations for supporting binary messages.
fsmonitor--daemon: man page
fsmonitor--daemon: update fsmonitor documentation
fsmonitor-ipc: create client routines for git-fsmonitor--daemon
help: include fsmonitor--daemon feature flag in version info
fsmonitor--daemon: add a built-in fsmonitor daemon
fsmonitor--daemon: implement client command options
t/helper/fsmonitor-client: create IPC client to talk to FSMonitor
Daemon
fsmonitor-fs-listen-win32: stub in backend for Windows
fsmonitor-fs-listen-macos: stub in backend for MacOS
fsmonitor--daemon: implement daemon command options
fsmonitor--daemon: add pathname classification
fsmonitor--daemon: define token-ids
fsmonitor--daemon: create token-based changed path cache
fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows
fsmonitor-fs-listen-macos: add macos header files for FSEvent
fsmonitor-fs-listen-macos: implement FSEvent listener on MacOS
fsmonitor--daemon: implement handle_client callback
fsmonitor--daemon: periodically truncate list of modified files
fsmonitor--daemon: use a cookie file to sync with file system
fsmonitor: enhance existing comments
fsmonitor: force update index after large responses
t7527: create test for fsmonitor--daemon
p7519: add fsmonitor--daemon
t7527: test status with untracked-cache and fsmonitor--daemon
t/perf: avoid copying builtin fsmonitor files into test repo
Johannes Schindelin (2):
config: FSMonitor is repository-specific
fsmonitor: introduce `core.useBuiltinFSMonitor` to call the daemon via
IPC
.gitignore | 1 +
Documentation/config/core.txt | 56 +-
Documentation/git-fsmonitor--daemon.txt | 75 +
Documentation/git-update-index.txt | 27 +-
Documentation/githooks.txt | 3 +-
Makefile | 16 +
builtin.h | 1 +
builtin/fsmonitor--daemon.c | 1511 ++++++++++++++++++
builtin/update-index.c | 4 +-
compat/fsmonitor/fsmonitor-fs-listen-macos.c | 497 ++++++
compat/fsmonitor/fsmonitor-fs-listen-win32.c | 553 +++++++
compat/fsmonitor/fsmonitor-fs-listen.h | 49 +
compat/simple-ipc/ipc-unix-socket.c | 14 +-
compat/simple-ipc/ipc-win32.c | 14 +-
config.c | 9 +-
config.h | 2 +-
config.mak.uname | 4 +
contrib/buildsystems/CMakeLists.txt | 8 +
fsmonitor--daemon.h | 140 ++
fsmonitor-ipc.c | 179 +++
fsmonitor-ipc.h | 48 +
fsmonitor.c | 132 +-
git.c | 1 +
help.c | 4 +
repo-settings.c | 3 +
repository.h | 2 +
simple-ipc.h | 7 +-
t/helper/test-fsmonitor-client.c | 125 ++
t/helper/test-simple-ipc.c | 34 +-
t/helper/test-tool.c | 1 +
t/helper/test-tool.h | 1 +
t/perf/p7519-fsmonitor.sh | 42 +-
t/perf/perf-lib.sh | 2 +-
t/t7527-builtin-fsmonitor.sh | 572 +++++++
34 files changed, 4069 insertions(+), 68 deletions(-)
create mode 100644 Documentation/git-fsmonitor--daemon.txt
create mode 100644 builtin/fsmonitor--daemon.c
create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-macos.c
create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-win32.c
create mode 100644 compat/fsmonitor/fsmonitor-fs-listen.h
create mode 100644 fsmonitor--daemon.h
create mode 100644 fsmonitor-ipc.c
create mode 100644 fsmonitor-ipc.h
create mode 100644 t/helper/test-fsmonitor-client.c
create mode 100755 t/t7527-builtin-fsmonitor.sh
base-commit: b0c09ab879
Submitted-As: https://lore.kernel.org/git/pull.923.v2.git.1621691828.gitgitgadget@gmail.com
In-Reply-To: https://lore.kernel.org/git/pull.923.git.1617291666.gitgitgadget@gmail.com
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Do not try to copy a fsmonitor--daemon socket from the current
development directory into the test trash directory.
When we run the perf suite without an explicit source repo set,
we copy of the current $GIT_DIR into the test trash directory.
Unix domain sockets cannot be copied in that manner, so the test
setup fails.
Additionally, omit any other fsmonitor--daemon temp files inside
the $GIT_DIR directory.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Create 2x2 test matrix with the untracked-cache and fsmonitor--daemon
features and a series of edits and verify that status output is
identical.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Repeat all of the fsmonitor perf tests using `git fsmonitor--daemon` and
the "Simple IPC" interface.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Set the `FSMONITOR_CHANGED` bit on `istate->cache_changed` when
FSMonitor returns a very large repsonse to ensure that the index is
written to disk.
Normally, when the FSMonitor response includes a tracked file, the
index is always updated. Similarly, the index might be updated when
the response alters the untracked-cache (when enabled). However, in
cases where neither of those cause the index to be considered changed,
the FSMonitor response is wasted. Subsequent Git commands will make
requests with the same token and receive the same response.
If that response is very large, performance may suffer. It would be
more efficient to force update the index now (and the token in the
index extension) in order to reduce the size of the response received
by future commands.
This was observed on Windows after a large checkout. On Windows, the
kernel emits events for the files that are changed as they are
changed. However, it might delay events for the containing
directories until the system is more idle (or someone scans the
directory (so it seems)). The first status following a checkout would
get the list of files. The subsequent status commands would get the
list of directories as the events trickled out. But they would never
catch up because the token was not advanced because the index wasn't
updated.
This list of directories caused `wt_status_collect_untracked()` to
unnecessarily spend time actually scanning them during each command.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Teach fsmonitor--daemon client threads to create a cookie file
inside the .git directory and then wait until FS events for the
cookie are observed by the FS listener thread.
This helps address the racy nature of file system events by
blocking the client response until the kernel has drained any
event backlog.
This is especially important on MacOS where kernel events are
only issued with a limited frequency. See the `latency` argument
of `FSeventStreamCreate()`. The kernel only signals every `latency`
seconds, but does not guarantee that the kernel queue is completely
drained, so we may have to wait more than one interval. If we
increase the frequency, the system is more likely to drop events.
We avoid these issues by having each client thread create a unique
cookie file and then wait until it is seen in the event stream.
Co-authored-by: Kevin Willford <Kevin.Willford@microsoft.com>
Co-authored-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Teach fsmonitor--daemon to periodically truncate the list of
modified files to save some memory.
Clients will ask for the set of changes relative to a token that they
found in the FSMN index extension in the index. (This token is like a
point in time, but different). Clients will then update the index to
contain the response token (so that subsequent commands will be
relative to this new token).
Therefore, the daemon can gradually truncate the in-memory list of
changed paths as they become obsolete (older than the previous token).
Since we may have multiple clients making concurrent requests with a
skew of tokens and clients may be racing to the talk to the daemon,
we lazily truncate the list.
We introduce a 5 minute delay and truncate batches 5 minutes after
they are considered obsolete.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Teach fsmonitor--daemon to respond to IPC requests from client
Git processes and respond with a list of modified pathnames
relative to the provided token.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Include MacOS system declarations to allow us to use FSEvent and
CoreFoundation APIs. We need GCC and clang versions because of
compiler and header file conflicts.
While it is quite possible to #include Apple's CoreServices.h when
compiling C source code with clang, trying to build it with GCC
currently fails with this error:
In file included
from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/AuthSession.h:32,
from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/Security.h:42,
from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/CSIdentity.h:43,
from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/OSServices.h:29,
from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Headers/IconsCore.h:23,
from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Headers/LaunchServices.h:23,
from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:45,
/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/Authorization.h:193:7: error: variably modified 'bytes' at file scope
193 | char bytes[kAuthorizationExternalFormLength];
| ^~~~~
The underlying reason is that GCC (rightfully) objects that an `enum`
value such as `kAuthorizationExternalFormLength` is not a constant
(because it is not, the preprocessor has no knowledge of it, only the
actual C compiler does) and can therefore not be used to define the size
of a C array.
This is a known problem and tracked in GCC's bug tracker:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082
In the meantime, let's not block things and go the slightly ugly route
of declaring/defining the FSEvents constants, data structures and
functions that we need, so that we can avoid above-mentioned issue.
Let's do this _only_ for GCC, though, so that the CI/PR builds (which
build both with clang and with GCC) can guarantee that we _are_ using
the correct data types.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
