Implement `stop` and `status` client commands to control and query the
status of a `fsmonitor--daemon` server process (and implicitly start a
server process if necessary).
Later commits will implement the actual server and monitor the file
system.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Create a built-in file system monitoring daemon that can be used by
the existing `fsmonitor` feature (protocol API and index extension)
to improve the performance of various Git commands, such as `status`.
The `fsmonitor--daemon` feature builds upon the `Simple IPC` API and
provides an alternative to hook access to existing fsmonitors such
as `watchman`.
This commit merely adds the new command without any functionality.
Co-authored-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Use simple IPC to directly communicate with the new builtin file
system monitor daemon.
Define a new config setting `core.useBuiltinFSMonitor` to enable the
builtin file system monitor.
The `core.fsmonitor` setting has already been defined as a HOOK
pathname. Historically, this has been set to a HOOK script that will
talk with Watchman. For compatibility reasons, we do not want to
overload that definition (and cause problems if users have multiple
versions of Git installed).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
This commit refactors `git_config_get_fsmonitor()` into the `repo_*()`
form that takes a parameter `struct repository *r`.
That change prepares for the upcoming `core.useBuiltinFSMonitor` flag which
will be stored in the `repo_settings` struct.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Add the "feature: fsmonitor--daemon" message to the output of
`git version --build-options`.
This allows users to know if the built-in fsmonitor feature is
supported on their platform.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Create fsmonitor_ipc__*() client routines to spawn the built-in file
system monitor daemon and send it an IPC request using the `Simple
IPC` API.
Stub in empty fsmonitor_ipc__*() functions for unsupported platforms.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Update references to `core.fsmonitor` and `core.fsmonitorHookVersion` and
pointers to `Watchman` to mention the new built-in `fsmonitor--daemon`.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Add `command_len` argument to the Simple IPC API.
In my original Simple IPC API, I assumed that the request
would always be a null-terminated string of text characters.
The command arg was just a `const char *`.
I found a caller that would like to pass a binary command
to the daemon, so I want to ammend the Simple IPC API to
take `const char *command, size_t command_len` and pass
that to the daemon. (Really, the first arg should just be
a `void *` or `const unsigned byte *` to make that clearer.)
Note, the response side has always been a `struct strbuf`
which includes the buffer and length, so we already support
returning a binary answer. (Yes, it feels a little weird
returning a binary buffer in a `strbuf`, but it works.)
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Fix-up to a topic that is already in 'master'.
* en/dir-traversal:
dir: introduce readdir_skip_dot_and_dotdot() helper
dir: update stale description of treat_directory()
Revert "dir: update stale description of treat_directory()"
Revert "dir: introduce readdir_skip_dot_and_dotdot() helper"
Many places in the code were doing
while ((d = readdir(dir)) != NULL) {
if (is_dot_or_dotdot(d->d_name))
continue;
...process d...
}
Introduce a readdir_skip_dot_and_dotdot() helper to make that a one-liner:
while ((d = readdir_skip_dot_and_dotdot(dir)) != NULL) {
...process d...
}
This helper particularly simplifies checks for empty directories.
Also use this helper in read_cached_dir() so that our statistics are
consistent across platforms. (In other words, read_cached_dir() should
have been using is_dot_or_dotdot() and skipping such entries, but did
not and left it to treat_path() to detect and mark such entries as
path_none.)
Signed-off-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The documentation comment for treat_directory() was originally written
in 095952 (Teach directory traversal about subprojects, 2007-04-11)
which was before the 'struct dir_struct' split its bitfield of named
options into a 'flags' enum in 7c4c97c0 (Turn the flags in struct
dir_struct into a single variable, 2009-02-16). When those flags
changed, the comment became stale, since members like
'show_other_directories' transitioned into flags like
DIR_SHOW_OTHER_DIRECTORIES.
Update the comments for treat_directory() to use these flag names rather
than the old member names.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
"ld" on Solaris fails to link some test helpers, which has been
worked around by reshuffling the inline function definitions from a
header file to a source file that is the only user of them.
* ab/pack-linkage-fix:
pack-objects: move static inline from a header to the sole consumer
Workaround flaky tests introduced recently.
* ds/t1092-fix-flake-from-progress:
t1092: revert the "-1" hack for emulating "no progress meter"
t1092: use GIT_PROGRESS_DELAY for consistent results
Move the code that is only used in builtin/pack-objects.c out of
pack-objects.h.
This fixes an issue where Solaris's SunCC hasn't been able to compile
git since 483fa7f42d (t/helper/test-bitmap.c: initial commit,
2021-03-31).
The real origin of that issue is that in 898eba5e63 (pack-objects:
refer to delta objects by index instead of pointer, 2018-04-14)
utility functions only needed by builtin/pack-objects.c were added to
pack-objects.h. Since then the header has been used in a few other
places, but 483fa7f42d was the first time it was used by test helper.
Since Solaris is stricter about linking and the oe_get_size_slow()
function lives in builtin/pack-objects.c the build started failing
with:
Undefined first referenced
symbol in file
oe_get_size_slow t/helper/test-bitmap.o
ld: fatal: symbol referencing errors. No output written to t/helper/test-tool
On other platforms this is presumably OK because the compiler and/or
linker detects that the "static inline" functions that reference
oe_get_size_slow() aren't used.
Let's solve this by moving the relevant code from pack-objects.h to
builtin/pack-objects.c. This is almost entirely a code-only move, but
because of the early macro definitions in that file referencing some
of these inline functions we need to move the definition of "static
struct packing_data to_pack" earlier, and declare these inline
functions above the macros.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
t2080 makes a few copies of a test repository and later performs a
branch switch on each one of the copies to verify that parallel checkout
and sequential checkout produce the same results. However, the
repository is copied with `cp -R` which, on some systems, defaults to
following symlinks on the directory hierarchy and copying their target
files instead of copying the symlinks themselves. AIX is one example of
system where this happens. Because the symlinks are not preserved, the
copied repositories have paths that do not match what is in the index,
causing git to abort the checkout operation that we want to test. This
makes the test fail on these systems.
Fix this by copying the repository with the POSIX flag '-P', which
forces cp to copy the symlinks instead of following them. Note that we
already use this flag for other cp invocations in our test suite (see
t7001). With this change, t2080 now passes on AIX.
Reported-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Move the newly added "hooks_path" API in Git.pm to its only user in
git-send-email.perl. This was added in c8243933c7 (git-send-email:
Respect core.hooksPath setting, 2021-03-23), meaning that it hasn't
yet made it into a non-rc release of git.
The consensus with Git.pm is that we need to be considerate of
out-of-tree users who treat it as a public documented interface. We
should therefore be less willing to add new functionality to it, least
we be stuck supporting it after our own uses for it disappear.
In this case the git-send-email.perl hook invocation will probably be
replaced by a future "git hook run" command, and in the commit
preceding this one the "hooks_path" become nothing but a trivial
wrapper for "rev-parse --git-path hooks" anyway (with no
Cwd::abs_path() call), so let's just inline this command in
git-send-email.perl itself.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In c8243933c7 (git-send-email: Respect core.hooksPath setting,
2021-03-23) we started supporting core.hooksPath in "send-email". It's
been reported that on Windows[1] doing this by calling abs_path()
results in different canonicalizations of the absolute path.
This wasn't an issue in c8243933c7 itself, but was revealed by my
ea7811b37e (git-send-email: improve --validate error output,
2021-04-06) when we started emitting the path to the hook, which was
previously only internal to git-send-email.perl.
The just-landed 53753a37d0 (t9001-send-email.sh: fix expected
absolute paths on Windows, 2021-05-24) narrowly fixed this issue, but
I believe we can do better here. We should not be relying on whatever
changes Perl's abs_path() makes to the path "rev-parse --git-path
hooks" hands to us. Let's instead trust it, and hand it to Perl's
system() in git-send-email.perl. It will handle either a relative or
absolute path.
So let's revert most of 53753a37d0 and just have "hooks_path" return
what we get from "rev-parse" directly without modification. This has
the added benefit of making the error message friendlier in the common
case, we'll no longer print an absolute path for repository-local hook
errors.
1. http://lore.kernel.org/git/bb30fe2b-cd75-4782-24a6-08bb002a0367@kdbg.org
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This looked like a good idea, but it seems to break tests on 32-bit
builds rather badly. Revert to just use "100 thousands must be big
enough" for now.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The entire point of the FSMonitor is to monitor the worktree changes in
a more efficient manner than `lstat()`ing all worktree files every time
we refresh the index.
But if there is no worktree, FSMonitor has nothing to monitor.
So let's ignore if an FSMonitor is configured (e.g. in `~/.gitconfig`)
and we're running in a repository without worktree.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The t1092-sparse-checkout-compatibility.sh tests compare the stdout and
stderr for several Git commands across both full checkouts, sparse
checkouts with a full index, and sparse checkouts with a sparse index.
Since these are direct comparisons, sometimes a progress indicator can
flush at unpredictable points, especially on slower machines. This
causes the tests to be flaky.
One standard way to avoid this is to add GIT_PROGRESS_DELAY=0 to the Git
commands that are run, as this will force every progress indicator
created with start_progress_delay() to be created immediately. However,
there are some progress indicators that are created in the case of a
full index that are not created with a sparse index. Moreover, their
values may be different as those indexes have a different number of
entries.
Instead, use GIT_PROGRESS_DELAY=-1 (which will turn into UINT_MAX)
to ensure that any reasonable machine running these tests would
never display delayed progress indicators.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
We used to read the init.templateDir setting at builtin/init-db.c using
a git_config() callback that, in turn, called git_config_pathname(). To
simplify the config reading logic at this file and plug a memory leak,
this was replaced by a direct call to git_config_get_value() at
e4de4502e6 ("init: remove git_init_db_config() while fixing leaks",
2021-03-14). However, this function doesn't provide path expanding
semantics, like git_config_pathname() does, so paths with '~/' and
'~user/' are treated literally. This makes 'git init' fail to handle
init.templateDir paths using these constructs:
$ git config init.templateDir '~/templates_dir'
$ git init
'warning: templates not found in ~/templates_dir'
Replace the git_config_get_value() call by git_config_get_pathname(),
which does the '~/' and '~user/' expansions. Also add a regression test.
Note that unlike git_config_get_value(), the config cache does not own
the memory for the path returned by git_config_get_pathname(), so we
must free() it.
Reported on IRC by rkta.
Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Fix a regression with the "the editor exited uncleanly, aborting
everything" error message going missing after my
d21616c039 (git-send-email: refactor duplicate $? checks into a
function, 2021-04-06).
I introduced a $msg variable, but did not actually use it. This caused
us to miss the optional error message supplied by the "do_edit"
codepath. Fix that, and add tests to check that this works.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Git for Windows is a native Windows program that works with native
absolute paths in the drive letter style C:\dir. The auxiliary
infrastructure is based on MSYS2, which uses POSIX style /C/dir.
When we test for output of absolute paths produced by git.exe, we
usally have to expect C:\dir style paths. To produce such expected
paths, we have to use $(pwd) in the test scripts; the alternative,
$PWD, produces a POSIX style path. ($PWD is a shell variable, and the
shell is bash, an MSYS2 program, and operates in the POSIX realm.)
There are two recently added tests that were written to expect C:\dir
paths. The output that is tested is produced by `git send-email`, but
behind the scenes, this is a Perl script, which also works in the
POSIX realm and produces /C/dir style output.
In the first test case that is changed here, replace $(pwd) by $PWD
so that the expected path is constructed using /C/dir style.
The second test case sets core.hooksPath to an absolute path. Since
the test script talks to native git.exe, it is supposed to place a
C:/dir style path into the configuration; therefore, keep $(pwd).
When this configuration value is consumed by the Perl script, it is
transformed to /C/dir style by the MSYS2 layer and echoed back in
this form in the error message. Hence, do use $PWD for the expected
value.
Signed-off-by: Johannes Sixt <j6t@kdbg.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
If `feature.experimental` and `feature.manyFiles` are set, we now start
the built-in FSMonitor by default.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Builtin FSMonitor Feature
Here is V2 of my patch series to add a builtin FSMonitor daemon to Git.
V2 includes addresses most of the review comments from the RFC and many of
the issues seen during out beta-testing with selected GVFS users. However,
there are still a few items that I need to address:
[ ] Revisit the how the client handles the IPC_STATE__NOT_LISTENING state
(where a daemon appears to be running, but is non-responsive) [ ] Revisit
use of global core_fsmonitor as both a pathname and a boolean. The existing
fsmonitor code uses it as the pathname to the fsmonitor hook and as a flag
to indicate that a hook is configured. [ ] Consider having daemon chdir()
out of the working directory to avoid directory handle issues on Windows. [
] Some documentation recommendations. [ ] Split up the commit containing the
tests and move some earlier in patch series. [ ] Move my FSMonitor PREREQ to
test-lib.sh instead of having it in my test scripts. [ ] Document
performance gains. [ ] On Windows, If the daemon is started as an elevated
process, then client commands might not have access to communicate with it.
[ ] Review if/how we decide to shutdown the FSMonitor daemon after and a
significant idle period. [ ] Investigate ways to temporarily shutdown
FSMonitor daemons processes so that the Git for Windows installer can
install an upgrade.
In this version, the first commit updates the Simple IPC API to make it
easier to pass binary data using {char *, size_t} rather than assuming that
the message is a null-terminated string. FSMonitor does not use binary
messages and doesn't really need this API change, but I thought it best to
fix the API now before we have other callers of IPC.
This V2 version will hopefully be previewed as an experimental feature in
Git for Windows v2.32.0.windows.*.
Jeff Hostetler (26):
simple-ipc: preparations for supporting binary messages.
fsmonitor--daemon: man page
fsmonitor--daemon: update fsmonitor documentation
fsmonitor-ipc: create client routines for git-fsmonitor--daemon
help: include fsmonitor--daemon feature flag in version info
fsmonitor--daemon: add a built-in fsmonitor daemon
fsmonitor--daemon: implement client command options
t/helper/fsmonitor-client: create IPC client to talk to FSMonitor
Daemon
fsmonitor-fs-listen-win32: stub in backend for Windows
fsmonitor-fs-listen-macos: stub in backend for MacOS
fsmonitor--daemon: implement daemon command options
fsmonitor--daemon: add pathname classification
fsmonitor--daemon: define token-ids
fsmonitor--daemon: create token-based changed path cache
fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows
fsmonitor-fs-listen-macos: add macos header files for FSEvent
fsmonitor-fs-listen-macos: implement FSEvent listener on MacOS
fsmonitor--daemon: implement handle_client callback
fsmonitor--daemon: periodically truncate list of modified files
fsmonitor--daemon: use a cookie file to sync with file system
fsmonitor: enhance existing comments
fsmonitor: force update index after large responses
t7527: create test for fsmonitor--daemon
p7519: add fsmonitor--daemon
t7527: test status with untracked-cache and fsmonitor--daemon
t/perf: avoid copying builtin fsmonitor files into test repo
Johannes Schindelin (2):
config: FSMonitor is repository-specific
fsmonitor: introduce `core.useBuiltinFSMonitor` to call the daemon via
IPC
.gitignore | 1 +
Documentation/config/core.txt | 56 +-
Documentation/git-fsmonitor--daemon.txt | 75 +
Documentation/git-update-index.txt | 27 +-
Documentation/githooks.txt | 3 +-
Makefile | 16 +
builtin.h | 1 +
builtin/fsmonitor--daemon.c | 1511 ++++++++++++++++++
builtin/update-index.c | 4 +-
compat/fsmonitor/fsmonitor-fs-listen-macos.c | 497 ++++++
compat/fsmonitor/fsmonitor-fs-listen-win32.c | 553 +++++++
compat/fsmonitor/fsmonitor-fs-listen.h | 49 +
compat/simple-ipc/ipc-unix-socket.c | 14 +-
compat/simple-ipc/ipc-win32.c | 14 +-
config.c | 9 +-
config.h | 2 +-
config.mak.uname | 4 +
contrib/buildsystems/CMakeLists.txt | 8 +
fsmonitor--daemon.h | 140 ++
fsmonitor-ipc.c | 179 +++
fsmonitor-ipc.h | 48 +
fsmonitor.c | 132 +-
git.c | 1 +
help.c | 4 +
repo-settings.c | 3 +
repository.h | 2 +
simple-ipc.h | 7 +-
t/helper/test-fsmonitor-client.c | 125 ++
t/helper/test-simple-ipc.c | 34 +-
t/helper/test-tool.c | 1 +
t/helper/test-tool.h | 1 +
t/perf/p7519-fsmonitor.sh | 42 +-
t/perf/perf-lib.sh | 2 +-
t/t7527-builtin-fsmonitor.sh | 572 +++++++
34 files changed, 4069 insertions(+), 68 deletions(-)
create mode 100644 Documentation/git-fsmonitor--daemon.txt
create mode 100644 builtin/fsmonitor--daemon.c
create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-macos.c
create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-win32.c
create mode 100644 compat/fsmonitor/fsmonitor-fs-listen.h
create mode 100644 fsmonitor--daemon.h
create mode 100644 fsmonitor-ipc.c
create mode 100644 fsmonitor-ipc.h
create mode 100644 t/helper/test-fsmonitor-client.c
create mode 100755 t/t7527-builtin-fsmonitor.sh
base-commit: b0c09ab879
Submitted-As: https://lore.kernel.org/git/pull.923.v2.git.1621691828.gitgitgadget@gmail.com
In-Reply-To: https://lore.kernel.org/git/pull.923.git.1617291666.gitgitgadget@gmail.com
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This topic branch re-adds the deprecated --stdin/-z options to `git
reset`. Those patches were overridden by a different set of options in
the upstream Git project before we could propose `--stdin`.
We offered this in MinGit to applications that wanted a safer way to
pass lots of pathspecs to Git, and these applications will need to be
adjusted.
Instead of `--stdin`, `--pathspec-from-file=-` should be used, and
instead of `-z`, `--pathspec-file-nul`.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This branch allows third-party tools to call `git status
--no-lock-index` to avoid lock contention with the interactive Git usage
of the actual human user.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
