Commit Graph

9727 Commits

Author SHA1 Message Date
Jeff Hostetler
c19eef7f78 fsmonitor--daemon: prepare for adding health thread
Refactor daemon thread startup to make it easier to start
a third thread class to monitor the health of the daemon.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
2021-10-29 20:38:35 -04:00
Jeff Hostetler
32501c9a98 fsmonitor--daemon: cd out of worktree root
Teach the fsmonitor--daemon to CD outside of the worktree
before starting up.

The common Git startup mechanism causes the CWD of the daemon process
to be in the root of the worktree.  On Windows, this causes the daemon
process to hold a locked handle on the CWD and prevents other
processes from moving or deleting the worktree while the daemon is
running.

CD to HOME before entering main event loops.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
2021-10-29 20:38:35 -04:00
Jeff Hostetler
257fd1f4c2 fsmonitor--daemon: print start message only if fsmonitor.announceStartup
Teach fsmonitor--daemon to print a startup message only when
`fsmonitor.announceStartup` is true.  This setting is false by default
so that the output of client commands, like `git status`, is not
changed if the daemon is implicitly started.

The message is conditionally printed by "run" and "start" subcommands
and is sent to stderr.  It contains the path to the work tree root.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
2021-10-29 20:38:35 -04:00
Jeff Hostetler
8fb34c85f0 fsmonitor-settings: bare repos are incompatible with FSMonitor
Bare repos do not have a worktree, so there is nothing for the
daemon watch.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
2021-10-29 20:38:35 -04:00
Jeff Hostetler
3a87e9b13a fsmonitor--daemon: use a cookie file to sync with file system
Teach fsmonitor--daemon client threads to create a cookie file
inside the .git directory and then wait until FS events for the
cookie are observed by the FS listener thread.

This helps address the racy nature of file system events by
blocking the client response until the kernel has drained any
event backlog.

This is especially important on MacOS where kernel events are
only issued with a limited frequency.  See the `latency` argument
of `FSeventStreamCreate()`.  The kernel only signals every `latency`
seconds, but does not guarantee that the kernel queue is completely
drained, so we may have to wait more than one interval.  If we
increase the frequency, the system is more likely to drop events.
We avoid these issues by having each client thread create a unique
cookie file and then wait until it is seen in the event stream.

Co-authored-by: Kevin Willford <Kevin.Willford@microsoft.com>
Co-authored-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
2021-10-29 20:38:35 -04:00
Jeff Hostetler
9c42eff3fd fsmonitor--daemon: periodically truncate list of modified files
Teach fsmonitor--daemon to periodically truncate the list of
modified files to save some memory.

Clients will ask for the set of changes relative to a token that they
found in the FSMN index extension in the index.  (This token is like a
point in time, but different).  Clients will then update the index to
contain the response token (so that subsequent commands will be
relative to this new token).

Therefore, the daemon can gradually truncate the in-memory list of
changed paths as they become obsolete (older than the previous token).
Since we may have multiple clients making concurrent requests with a
skew of tokens and clients may be racing to the talk to the daemon,
we lazily truncate the list.

We introduce a 5 minute delay and truncate batches 5 minutes after
they are considered obsolete.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
2021-10-29 20:38:35 -04:00
Jeff Hostetler
aad79810b6 fsmonitor--daemon: implement handle_client callback
Teach fsmonitor--daemon to respond to IPC requests from client
Git processes and respond with a list of modified pathnames
relative to the provided token.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
2021-10-29 20:38:34 -04:00
Jeff Hostetler
1b481ca550 fsmonitor--daemon: create token-based changed path cache
Teach fsmonitor--daemon to build a list of changed paths and associate
them with a token-id.  This will be used by the platform-specific
backends to accumulate changed paths in response to filesystem events.

The platform-specific file system listener thread receives file system
events containing one or more changed pathnames (with whatever bucketing
or grouping that is convenient for the file system).  These paths are
accumulated (without locking) by the file system layer into a `fsmonitor_batch`.

When the file system layer has drained the kernel event queue, it will
"publish" them to our token queue and make them visible to concurrent
client worker threads.  The token layer is free to combine and/or de-dup
paths within these batches for efficient presentation to clients.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
2021-10-29 20:38:34 -04:00
Jeff Hostetler
2646363a60 fsmonitor--daemon: define token-ids
Teach fsmonitor--daemon to create token-ids and define the
overall token naming scheme.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
2021-10-29 20:38:34 -04:00
Jeff Hostetler
780661d035 fsmonitor--daemon: add pathname classification
Teach fsmonitor--daemon to classify relative and absolute
pathnames and decide how they should be handled.  This will
be used by the platform-specific backend to respond to each
filesystem event.

When we register for filesystem notifications on a directory,
we get events for everything (recursively) in the directory.
We want to report to clients changes to tracked and untracked
paths within the working directory.  We do not want to report
changes within the .git directory, for example.

This classification will be used in a later commit by the
different backends to classify paths as events are received.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
2021-10-29 20:38:34 -04:00
Jeff Hostetler
cff0633921 fsmonitor--daemon: implement 'start' command
Implement 'git fsmonitor--daemon start' command.  This command starts
an instance of 'git fsmonitor--daemon run' in the background using
the new 'start_bg_command()' function.

We avoid the fork-and-call technique on Unix systems in favor of a
fork-and-exec technique.  This gives us more uniform Trace2 child-*
events.  It also makes our usage more consistent with Windows usage.

On Windows, teach 'git fsmonitor--daemon run' to optionally call
'FreeConsole()' to release handles to the inherited Win32 console
(despite being passed invalid handles for stdin/out/err).  Without
this, command prompts and powershell terminal windows could hang
in "exit" until the last background child process exited or released
their Win32 console handle.  (This was not seen with git-bash shells
because they don't have a Win32 console attached to them.)

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
2021-10-29 20:38:34 -04:00
Jeff Hostetler
13d73a0ef0 fsmonitor--daemon: implement 'run' command
Implement `run` command to try to begin listening for file system events.

This version defines the thread structure with a single fsmonitor_fs_listen
thread to watch for file system events and a simple IPC thread pool to
watch for connection from Git clients over a well-known named pipe or
Unix domain socket.

This commit does not actually do anything yet because the platform
backends are still just stubs.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
2021-10-29 20:38:34 -04:00
Jeff Hostetler
63ed66753b fsmonitor--daemon: implement 'stop' and 'status' commands
Implement `stop` and `status` client commands to control and query the
status of a `fsmonitor--daemon` server process (and implicitly start a
server process if necessary).

Later commits will implement the actual server and monitor the file
system.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
2021-10-29 20:38:34 -04:00
Carlo Marcelo Arenas Belón
49189d69a1 builtin/fsmonitor--daemon: use parse-options API fully
--help and -h are already handled internally so just parse_options()
do the parsing and extract the command from the remaining options.

as a side effect, avoid setting a variable argc to a value that was
never used.

Signed-off-by: Carlo Marcelo Arenas Belón <carenas@gmail.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
2021-10-29 20:38:34 -04:00
Jeff Hostetler
681c53a074 fsmonitor--daemon: add a built-in fsmonitor daemon
Create a built-in file system monitoring daemon that can be used by
the existing `fsmonitor` feature (protocol API and index extension)
to improve the performance of various Git commands, such as `status`.

The `fsmonitor--daemon` feature builds upon the `Simple IPC` API and
provides an alternative to hook access to existing fsmonitors such
as `watchman`.

This commit merely adds the new command without any functionality.

Co-authored-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
2021-10-29 20:38:34 -04:00
Jeff Hostetler
521d3fd3ea fsmonitor: config settings are repository-specific
Move FSMonitor config settings to a new `struct fsmonitor_settings`
structure.  Add a lazily-loaded pointer to `struct repo_settings`.
Create `fsm_settings__get_*()` getters to lazily look up fsmonitor-
related config settings.

Get rid of the `core_fsmonitor` global variable, and add support for
the new `core.useBuiltinFSMonitor` config setting.  Move config code
to lookup the existing `core.fsmonitor` value to `fsmonitor-settings.[ch]`.

The `core_fsmonitor` global variable was used to store the pathname to
the FSMonitor hook and it was used as a boolean to see if FSMonitor
was enabled.  This dual usage will lead to confusion when we add
support for a builtin FSMonitor based on IPC, since the builtin
FSMonitor doesn't need the hook pathname.

Replace the boolean usage with an `enum fsmonitor_mode` to represent
the state of FSMonitor.  And only set the pathname when in HOOK mode.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
2021-10-29 20:38:34 -04:00
Junio C Hamano
f54c172bb3 Merge branch 'ks/submodule-add-message-fix'
Message regression fix.

* ks/submodule-add-message-fix:
  submodule: drop unused sm_name parameter from append_fetch_remotes()
  submodule--helper: fix incorrect newlines in an error message
2021-10-29 15:43:14 -07:00
Junio C Hamano
192a3fa31d Merge branch 'ab/plug-random-leaks'
Leakfix.

* ab/plug-random-leaks:
  reflog: free() ref given to us by dwim_log()
  submodule--helper: fix small memory leaks
  clone: fix a memory leak of the "git_dir" variable
  grep: fix a "path_list" memory leak
  grep: use object_array_clear() in cmd_grep()
  grep: prefer "struct grep_opt" over its "void *" equivalent
2021-10-29 15:43:13 -07:00
Junio C Hamano
c3673a8eb2 Merge branch 'ab/ref-filter-leakfix'
"git for-each-ref" family of commands were leaking the ref_sorting
instances that hold sorting keys specified by the user; this has
been corrected.

* ab/ref-filter-leakfix:
  branch: use ref_sorting_release()
  ref-filter API user: add and use a ref_sorting_release()
  tag: use a "goto cleanup" pattern, leak less memory
2021-10-29 15:43:12 -07:00
Junio C Hamano
735907bde1 Merge branch 'jk/http-push-status-fix'
"git push" client talking to an HTTP server did not diagnose the
lack of the final status report from the other side correctly,
which has been corrected.

* jk/http-push-status-fix:
  transport-helper: recognize "expecting report" error from send-pack
  send-pack: complain about "expecting report" with --helper-status
2021-10-29 15:43:12 -07:00
Jeff King
6b615dbece submodule: drop unused sm_name parameter from append_fetch_remotes()
Commit c21fb4676f (submodule--helper: fix incorrect newlines in an error
message, 2021-10-23) accidentally added a new, unused parameter while
changing the name and signature of show_fetch_remotes() to
append_fetch_remotes(). We can drop this to keep things simpler (and
satisfy -Wunused-parameter).

The error is likely because c21fb4676f is fixing a problem from
8c8195e9c3 (submodule--helper: introduce add-clone subcommand,
2021-07-10). An earlier iteration of that second commit introduced the
same unused parameter (though it was dropped before it finally made it
to 'next'), and the fix on top accidentally carried forward the extra
parameter.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-27 10:42:11 -07:00
Junio C Hamano
06355d72dc Merge branch 'ab/pkt-line-cleanup'
Code clean-up.

* ab/pkt-line-cleanup:
  pkt-line.[ch]: remove unused packet_read_line_buf()
  pkt-line.[ch]: remove unused packet_buf_write_len()
2021-10-25 16:07:00 -07:00
Junio C Hamano
54c4f8ce52 Merge branch 'ab/mark-leak-free-tests-more'
Bunch of tests are marked as "passing leak check".

* ab/mark-leak-free-tests-more:
  merge: add missing strbuf_release()
  ls-files: add missing string_list_clear()
  ls-files: fix a trivial dir_clear() leak
  tests: fix test-oid-array leak, test in SANITIZE=leak
  tests: fix a memory leak in test-oidtree.c
  tests: fix a memory leak in test-parse-options.c
  tests: fix a memory leak in test-prio-queue.c
2021-10-25 16:06:59 -07:00
Junio C Hamano
65ca3245f9 Merge branch 'ab/parse-options-cleanup'
Random changes to parse-options implementation.

* ab/parse-options-cleanup:
  parse-options: change OPT_{SHORT,UNSET} to an enum
  parse-options tests: test optname() output
  parse-options.[ch]: make opt{bug,name}() "static"
  commit-graph: stop using optname()
  parse-options.c: move optname() earlier in the file
  parse-options.h: make the "flags" in "struct option" an enum
  parse-options.c: use exhaustive "case" arms for "enum parse_opt_result"
  parse-options.[ch]: consistently use "enum parse_opt_result"
  parse-options.[ch]: consistently use "enum parse_opt_flags"
  parse-options.h: move PARSE_OPT_SHELL_EVAL between enums
2021-10-25 16:06:59 -07:00
Junio C Hamano
18c6653da0 Merge branch 'fs/ssh-signing'
Use ssh public crypto for object and push-cert signing.

* fs/ssh-signing:
  ssh signing: test that gpg fails for unknown keys
  ssh signing: tests for logs, tags & push certs
  ssh signing: duplicate t7510 tests for commits
  ssh signing: verify signatures using ssh-keygen
  ssh signing: provide a textual signing_key_id
  ssh signing: retrieve a default key from ssh-agent
  ssh signing: add ssh key format and signing code
  ssh signing: add test prereqs
  ssh signing: preliminary refactoring and clean-up
2021-10-25 16:06:58 -07:00
Junio C Hamano
061a21d36d Merge branch 'ab/fsck-unexpected-type'
"git fsck" has been taught to report mismatch between expected and
actual types of an object better.

* ab/fsck-unexpected-type:
  fsck: report invalid object type-path combinations
  fsck: don't hard die on invalid object types
  object-file.c: stop dying in parse_loose_header()
  object-file.c: return ULHR_TOO_LONG on "header too long"
  object-file.c: use "enum" return type for unpack_loose_header()
  object-file.c: simplify unpack_loose_short_header()
  object-file.c: make parse_loose_header_extended() public
  object-file.c: return -1, not "status" from unpack_loose_header()
  object-file.c: don't set "typep" when returning non-zero
  cat-file tests: test for current --allow-unknown-type behavior
  cat-file tests: add corrupt loose object test
  cat-file tests: test for missing/bogus object with -t, -s and -p
  cat-file tests: move bogus_* variable declarations earlier
  fsck tests: test for garbage appended to a loose object
  fsck tests: test current hash/type mismatch behavior
  fsck tests: refactor one test to use a sub-repo
  fsck tests: add test for fsck-ing an unknown type
2021-10-25 16:06:56 -07:00
Kaartic Sivaraam
c21fb4676f submodule--helper: fix incorrect newlines in an error message
A refactoring[1] done as part of the recent conversion of
'git submodule add' to builtin, changed the error message
shown when a Git directory already exists locally for a submodule
name. Before the refactoring, the error used to appear like so:

  --- START OF OUTPUT ---
  $ git submodule add ../sub/ subm
  A git directory for 'subm' is found locally with remote(s):
    origin        /me/git-repos-for-test/sub
  If you want to reuse this local git directory instead of cloning again from
    /me/git-repos-for-test/sub
  use the '--force' option. If the local git directory is not the correct repo
  or you are unsure what this means choose another name with the '--name' option.
  ---  END OF OUTPUT  ---

After the refactoring the error started appearing like so:

  --- START OF OUTPUT ---
  $ git submodule add ../sub/ subm
  A git directory for 'subm' is found locally with remote(s):  origin     /me/git-repos-for-test/sub
  fatal: If you want to reuse this local git directory instead of cloning again from
  /me/git-repos-for-test/sub
  use the '--force' option. If the local git directory is not the correct repo
  or if you are unsure what this means, choose another name with the '--name' option.

  ---  END OF OUTPUT  ---

As one could observe the remote information is printed along with the
first line rather than on its own line. Also, there's an additional
newline following output.

Make the error message consistent with the error message that used to be
printed before the refactoring.

This also moves the 'fatal:' prefix that appears in the middle of the
error message to the first line as it would more appropriate to have
it in the first line. The output after the change would look like:

  --- START OF OUTPUT ---
  $ git submodule add ../sub/ subm
  fatal: A git directory for 'subm' is found locally with remote(s):
    origin        /me/git-repos-for-test/sub
  If you want to reuse this local git directory instead of cloning again from
    /me/git-repos-for-test/sub
  use the '--force' option. If the local git directory is not the correct repo
  or you are unsure what this means choose another name with the '--name' option.
  ---  END OF OUTPUT  ---

[1]: https://lore.kernel.org/git/20210710074801.19917-5-raykar.ath@gmail.com/#t

Signed-off-by: Kaartic Sivaraam <kaartic.sivaraam@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-23 23:01:56 -07:00
Ævar Arnfjörð Bjarmason
3c8150497f reflog: free() ref given to us by dwim_log()
When dwim_log() returns the "ref" is always ether NULL or an
xstrdup()'d string.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-23 10:45:25 -07:00
Ævar Arnfjörð Bjarmason
c270b055d9 submodule--helper: fix small memory leaks
Add a missing strbuf_release() and a clear_pathspec() to the
submodule--helper.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-23 10:45:25 -07:00
Ævar Arnfjörð Bjarmason
27ff1fbc5d clone: fix a memory leak of the "git_dir" variable
At this point in cmd_clone the "git_dir" is always either an
xstrdup()'d string, or something we got from mkpathdup(). Let's free()
it before we clobber it.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-23 10:45:25 -07:00
Ævar Arnfjörð Bjarmason
b202e51b15 grep: fix a "path_list" memory leak
Free the "path_list" used in builtin/grep.c, it was declared as
STRING_LIST_INIT_NODUP, let's change it to a STRING_LIST_INIT_DUP
since an early user in cmd_grep() appends a string passed via
parse-options.c to it, which needs to be duplicated.

Let's then convert the remaining callers to use
string_list_append_nodup() instead, allowing us to free the list.

This makes all the tests in t7811-grep-open.sh pass, 6/10 would fail
before this change. The only remaining failure would have been due to
a stray "git checkout" (which still leaks memory). In this case we can
use a "git reset --hard" instead, so let's do that, and move the
test_when_finished() above the code that would modify the relevant
file.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-23 10:45:25 -07:00
Ævar Arnfjörð Bjarmason
96c101257b grep: use object_array_clear() in cmd_grep()
Free the "struct object_array" before exiting. This makes grep tests
(e.g.  "t7815-grep-binary.sh") a bit happer under SANITIZE=leak.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-23 10:45:25 -07:00
Ævar Arnfjörð Bjarmason
a2fb7672c0 grep: prefer "struct grep_opt" over its "void *" equivalent
Stylistically fix up code added in bfac23d953 (grep: Fix two memory
leaks, 2010-01-30). We usually don't use the "arg" at all once we've
casted it to the struct we want, let's not do that here when we're
freeing it. Perhaps it was thought that a cast to "void *" would
otherwise be needed?

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-23 10:45:25 -07:00
Ævar Arnfjörð Bjarmason
d72d4f92e2 branch: use ref_sorting_release()
Use a ref_sorting_release() in branch.c to free the memory from the
ref_sorting_options(). This plugs the final in-tree memory leak of
that API.

In the preceding commit the "sorting" variable was left in the
cmd_branch() scope, even though that wasn't needed anymore. Move it to
the "else if (list)" scope instead. We can also move the "struct
string_list" only used for that branch to be declared in that block

That "struct ref_sorting" does not need to be "static" (and isn't
re-used). The "ref_sorting_options()" will return a valid one, we
don't need to make it "static" to have it zero'd out. That it was
static was another artifact of the pre-image of the preceding commit.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-20 11:36:13 -07:00
Ævar Arnfjörð Bjarmason
e5fb028688 ref-filter API user: add and use a ref_sorting_release()
Add a ref_sorting_release() and use it for some of the current API
users, the ref_sorting_default() function and its siblings will do a
malloc() which wasn't being free'd previously.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-20 11:36:13 -07:00
Ævar Arnfjörð Bjarmason
37766b61cd tag: use a "goto cleanup" pattern, leak less memory
Change cmd_tag() to free its "struct strbuf"'s instead of using an
UNLEAK() pattern. This changes code added in 886e1084d7 (builtin/:
add UNLEAKs, 2017-10-01).

As shown in the context of the declaration of the "struct
msg_arg" (which I'm changing to use a designated initializer while at
it, and to show the context in this change), that struct is just a
thin wrapper around an int and "struct strbuf".

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-20 11:36:13 -07:00
Junio C Hamano
092228ee5c Merge branch 'jk/cat-file-batch-all-wo-replace'
"git cat-file --batch" with the "--batch-all-objects" option is
supposed to iterate over all the objects found in a repository, but
it used to translate these object names using the replace mechanism,
which defeats the point of enumerating all objects in the repository.
This has been corrected.

* jk/cat-file-batch-all-wo-replace:
  cat-file: use packed_object_info() for --batch-all-objects
  cat-file: split ordered/unordered batch-all-objects callbacks
  cat-file: disable refs/replace with --batch-all-objects
  cat-file: mention --unordered along with --batch-all-objects
  t1006: clean up broken objects
2021-10-18 15:47:57 -07:00
Junio C Hamano
a4b9fb6a5c Merge branch 'ab/designated-initializers-more'
Code clean-up.

* ab/designated-initializers-more:
  builtin/remote.c: add and use SHOW_INFO_INIT
  builtin/remote.c: add and use a REF_STATES_INIT
  urlmatch.[ch]: add and use URLMATCH_CONFIG_INIT
  builtin/blame.c: refactor commit_info_init() to COMMIT_INFO_INIT macro
  daemon.c: refactor hostinfo_init() to HOSTINFO_INIT macro
2021-10-18 15:47:57 -07:00
Junio C Hamano
0b69bb0fb1 Merge branch 'tb/repack-write-midx'
"git repack" has been taught to generate multi-pack reachability
bitmaps.

* tb/repack-write-midx:
  test-read-midx: fix leak of bitmap_index struct
  builtin/repack.c: pass `--refs-snapshot` when writing bitmaps
  builtin/repack.c: make largest pack preferred
  builtin/repack.c: support writing a MIDX while repacking
  builtin/repack.c: extract showing progress to a variable
  builtin/repack.c: rename variables that deal with non-kept packs
  builtin/repack.c: keep track of existing packs unconditionally
  midx: preliminary support for `--refs-snapshot`
  builtin/multi-pack-index.c: support `--stdin-packs` mode
  midx: expose `write_midx_file_only()` publicly
2021-10-18 15:47:57 -07:00
Junio C Hamano
223a1bfb58 Merge branch 'js/retire-preserve-merges'
The "--preserve-merges" option of "git rebase" has been removed.

* js/retire-preserve-merges:
  sequencer: restrict scope of a formerly public function
  rebase: remove a no-longer-used function
  rebase: stop mentioning the -p option in comments
  rebase: remove obsolete code comment
  rebase: drop the internal `rebase--interactive` command
  git-svn: drop support for `--preserve-merges`
  rebase: drop support for `--preserve-merges`
  pull: remove support for `--rebase=preserve`
  tests: stop testing `git rebase --preserve-merges`
  remote: warn about unhandled branch.<name>.rebase values
  t5520: do not use `pull.rebase=preserve`
2021-10-18 15:47:56 -07:00
Jeff King
e4c9538a9c send-pack: complain about "expecting report" with --helper-status
When pushing to a server which erroneously omits the final ref-status
report, the client side should complain about the refs for which we
didn't receive the status (because we can't just assume they were
updated). This works over most transports like ssh, but for http we'll
print a very misleading "Everything up-to-date".

It works for ssh because send-pack internally sets the status of each
ref to REF_STATUS_EXPECTING_REPORT, and then if the server doesn't tell
us about a particular ref, it will stay at that value. When we print the
final status table, we'll see that we're still on EXPECTING_REPORT and
complain then.

But for http, we go through remote-curl, which invokes send-pack with
"--stateless-rpc --helper-status". The latter option causes send-pack to
return a machine-readable list of ref statuses to the remote helper. But
ever since its inception in de1a2fdd38 (Smart push over HTTP: client
side, 2009-10-30), the send-pack code has simply omitted mention of any
ref which ended up in EXPECTING_REPORT.

In the remote helper, we then take the absence of any status report
from send-pack to mean that the ref was not even something we tried to
send, and thus it prints "Everything up-to-date". Fortunately it does
detect the eventual non-zero exit from send-pack, and propagates that in
its own non-zero exit code. So at least a careful script invoking "git
push" would notice the failure.  But sending the misleading message on
stderr is certainly confusing for humans (not to mention the
machine-readable "push --porcelain" output, though again, any careful
script should be checking the exit code from push, too).

Nobody seems to have noticed because the server in this instance has to
be misbehaving: it has promised to support the ref-status capability
(otherwise the client will not set EXPECTING_REPORT at all), but didn't
send us any. If the connection were simply cut, then send-pack would
complain about getting EOF while trying to read the status. But if the
server actually sends a flush packet (i.e., saying "now you have all of
the ref statuses" without actually sending any), then the client ends up
in this confused situation.

The fix is simple: we should return an error message from "send-pack
--helper-status", just like we would for any other error per-ref error
condition (in the test I included, the server simply omits all ref
status responses, but a more insidious version of this would skip only
some of them).

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-18 13:26:52 -07:00
Ævar Arnfjörð Bjarmason
ec9a37d69b pkt-line.[ch]: remove unused packet_read_line_buf()
This function was added in 4981fe750b (pkt-line: share
buffer/descriptor reading implementation, 2013-02-23), but in
01f9ec64c8 (Use packet_reader instead of packet_read_line,
2018-12-29) the code that was using it was removed.

Since it's being removed we can in turn remove the "src" and "src_len"
arguments to packet_read(), all the remaining users just passed a
NULL/NULL pair to it.

That function is only a thin wrapper for packet_read_with_status()
which still needs those arguments, but for the thin packet_read()
convenience wrapper we can do away with it for now.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-15 13:09:40 -07:00
Junio C Hamano
d7bc852151 Merge branch 'ab/align-parse-options-help'
When "git cmd -h" shows more than one line of usage text (e.g.
the cmd subcommand may take sub-sub-command), parse-options API
learned to align these lines, even across i18n/l10n.

* ab/align-parse-options-help:
  parse-options: properly align continued usage output
  git rev-parse --parseopt tests: add more usagestr tests
  send-pack: properly use parse_options() API for usage string
  parse-options API users: align usage output in C-strings
2021-10-13 15:15:58 -07:00
Junio C Hamano
62f035aee3 Merge branch 'ab/help-config-vars'
Teach "git help -c" into helping the command line completion of
configuration variables.

* ab/help-config-vars:
  help: move column config discovery to help.c library
  help / completion: make "git help" do the hard work
  help tests: test --config-for-completion option & output
  help: simplify by moving to OPT_CMDMODE()
  help: correct logic error in combining --all and --guides
  help: correct logic error in combining --all and --config
  help tests: add test for --config output
  help: correct usage & behavior of "git help --guides"
  help: correct the usage string in -h and documentation
2021-10-13 15:15:58 -07:00
Junio C Hamano
a5e61a4225 Merge branch 'ab/config-based-hooks-1'
Mostly preliminary clean-up in the hook API.

* ab/config-based-hooks-1:
  hook-list.h: add a generated list of hooks, like config-list.h
  hook.c users: use "hook_exists()" instead of "find_hook()"
  hook.c: add a hook_exists() wrapper and use it in bugreport.c
  hook.[ch]: move find_hook() from run-command.c to hook.c
  Makefile: remove an out-of-date comment
  Makefile: don't perform "mv $@+ $@" dance for $(GENERATED_H)
  Makefile: stop hardcoding {command,config}-list.h
  Makefile: mark "check" target as .PHONY
2021-10-13 15:15:57 -07:00
Junio C Hamano
a7c2daa06d Merge branch 'en/removing-untracked-fixes'
Various fixes in code paths that move untracked files away to make room.

* en/removing-untracked-fixes:
  Documentation: call out commands that nuke untracked files/directories
  Comment important codepaths regarding nuking untracked files/dirs
  unpack-trees: avoid nuking untracked dir in way of locally deleted file
  unpack-trees: avoid nuking untracked dir in way of unmerged file
  Change unpack_trees' 'reset' flag into an enum
  Remove ignored files by default when they are in the way
  unpack-trees: make dir an internal-only struct
  unpack-trees: introduce preserve_ignored to unpack_trees_options
  read-tree, merge-recursive: overwrite ignored files by default
  checkout, read-tree: fix leak of unpack_trees_options.dir
  t2500: add various tests for nuking untracked files
2021-10-13 15:15:57 -07:00
Junio C Hamano
2d498a7c89 Merge branch 'ds/add-rm-with-sparse-index'
"git add", "git mv", and "git rm" have been adjusted to avoid
updating paths outside of the sparse-checkout definition unless
the user specifies a "--sparse" option.

* ds/add-rm-with-sparse-index:
  advice: update message to suggest '--sparse'
  mv: refuse to move sparse paths
  rm: skip sparse paths with missing SKIP_WORKTREE
  rm: add --sparse option
  add: update --renormalize to skip sparse paths
  add: update --chmod to skip sparse paths
  add: implement the --sparse option
  add: skip tracked paths outside sparse-checkout cone
  add: fail when adding an untracked sparse file
  dir: fix pattern matching on dirs
  dir: select directories correctly
  t1092: behavior for adding sparse files
  t3705: test that 'sparse_entry' is unstaged
2021-10-13 15:15:56 -07:00
Junio C Hamano
ef09a7fbbe Merge branch 'da/difftool-dir-diff-symlink-fix' into maint
"git difftool --dir-diff" mishandled symbolic links.

* da/difftool-dir-diff-symlink-fix:
  difftool: fix symlink-file writing in dir-diff mode
2021-10-12 13:51:48 -07:00
Junio C Hamano
c365967f21 Merge branch 'jk/clone-unborn-head-in-bare' into maint
"git clone" from a repository whose HEAD is unborn into a bare
repository didn't follow the branch name the other side used, which
is corrected.

* jk/clone-unborn-head-in-bare:
  clone: handle unborn branch in bare repos
2021-10-12 13:51:47 -07:00
Junio C Hamano
e61304f21d Merge branch 'en/stash-df-fix' into maint
"git stash", where the tentative change involves changing a
directory to a file (or vice versa), was confused, which has been
corrected.

* en/stash-df-fix:
  stash: restore untracked files AFTER restoring tracked files
  stash: avoid feeding directories to update-index
  t3903: document a pair of directory/file bugs
2021-10-12 13:51:47 -07:00