Every once in a while I need to verify that Microsoft Git's test suite
passes for changes that are not yet meant for public consumption, and
since it was (made) too difficult to keep up a working Azure Pipeline
definition, I have to use GitHub Actions in a private GitHub repository
for that purpose.
In these tests, basically all Dockerized CI jobs fail consistently. The
symptom is something like:
error: cannot create async thread: Resource temporarily unavailable
in the middle of a test, typically in the t5xxx-t6xxx range. The first
such error is immediately followed by plenty more of these errors, and
not a single test succeeds afterwards.
At first, I thought that maybe the massive parallelism I enjoy there is
the problem, and I thought that the cgroups limits might be shared
between the many containers that run on essentially the same physical
machine. But even reducing the matrix to just a single of those
Dockerized jobs runs into the very same problems.
The underlying reason seems to be a substantial difference in the hosted
runners that execute these Dockerized jobs: forcing the PID limit of the
container to a high number lets the jobs pass, even when running the
complete matrix of all 13 Dockerized jobs concurrently. But that's not
the only difference: The jobs seem to take a lot longer in these
containers than, say, in the containers made available to
https://github.com/git/git.
When forcing a PID limit of 64k in that private repository, the jobs
completed successfully, but they also took a lot longer, between 2x to
2.5x longer, i.e. painfully much longer. Reducing the PID limit to 16k,
the CI jobs still passed, but took an equally long amount of time.
Reducing the PID limit to 8k caused the errors to reappear.
Here are the numbers from three example runs, the first one forcing the
PID and nproc limit to 65536, the second one to 16384, the third run is
from the public git/git repository:
Job | 64k | 16k | reference
------------------------------|---------|---------|---------
almalinux-8 | 19m 3s | 16m 0s | 9m 36s
debian-11 | 20m 31s | 20m 3s | 8m 5s
fedora-breaking-changes-meson | 16m 29s | 19m 19s | 9m 40s
linux-asan-ubsan | 1h 10m | 1h 11m | 34m 36s
linux-breaking-changes | 25m 39s | 25m 58s | 13m 15s
linux-leaks | 1h 9m | 1h 10m | 33m 30s
linux-meson | 28m 9s | 27m 4s | 13m 45s
linux-musl-meson | 16m 32s | 13m 39s | 8m 6s
linux-reftable-leaks | 1h 13m | 1h 13m | 34m 34s
linux-reftable | 26m 2s | 25m 48s | 13m 31s
linux-sha256 | 26m 12s | 26m 3s | 12m 36s
linux-TEST-vars | 26m 5s | 25m 21s | 13m 25s
linux32 | 21m 16s | 19m 57s | 10m 44s
It does not look as if the PID limit is the reason for the longer
runtime, seeing as the 64k vs 16k timings deviate no more than as is
usual with GitHub workflows. So let's go for 16k.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The final step, split from earlier attempt by Dscho, to loosen the
sideband restriction for now and tighten later at Git v3.0 boundary.
Will retract.
(this branch uses jc/neuter-sideband-fixup.)
* jc/neuter-sideband-post-3.0:
sideband: delay sanitizing by default to Git v3.0
The `git multi-pack-index write --incremental` command has been
corrected to properly honor the `--base` option. Previously, the
custom base was ignored by the normal write path, and the pack
exclusion logic incorrectly skipped packs from layers above the
selected base, breaking reachability closure for bitmaps.
* tb/midx-incremental-custom-base:
midx-write: include packs above custom incremental base
midx: pass custom '--base' through incremental writes
t5334: expose shared `nth_line()` helper
The contributor guide has been updated to advise new contributors to
trim irrelevant quoted text when replying to review comments, matching
the existing advice given to reviewers.
* wy/doc-myfirstcontribution-trim-quotes:
MyFirstContribution: mention trimming quoted text in replies
The trailer sections in SubmittingPatches have been updated to
encourage use of standard trailers.
* kh/submittingpatches-trailers:
SubmittingPatches: note that trailer order matters
SubmittingPatches: be consistent with trailer markup
SubmittingPatches: document Based-on-patch-by trailer
SubmittingPatches: discourage common Linux trailers
SubmittingPatches: discuss non-ident trailers
SubmittingPatches: encourage trailer use for substantial help
The reference backends have been converted to always use absolute
paths internally. This allows dropping the calls to
`chdir_notify_reparent()` and fixes a memory leak in how the
reference database is constructed with an "onbranch" condition.
* ps/refs-avoid-chdir-notify-reparent:
refs: always use absolute paths for reference stores
refs: drop local buffer in `refs_compute_filesystem_location()`
refs: fix recursing `get_main_ref_store()` with "onbranch" config
repository: free main reference database
chdir-notify: drop unused `chdir_notify_reparent()`
refs: unregister reference stores from "chdir_notify"
setup: don't apply "GIT_REFERENCE_BACKEND" without a repository
setup: stop applying repository format twice
setup: inline `check_and_apply_repository_format()`
git replay learns --linearize option to drop merge commits and
linearize the replayed history, mimicking git rebase
--no-rebase-merges.
* tc/replay-linearize:
replay: offer an option to linearize the commit topology
replay: add helper to put entry into mapped_commits
replay: refactor enum replay_mode into a bool
'git branch --contains' and 'git for-each-ref --contains' have
been optimized to use the memoized commit traversal previously
used only by 'git tag --contains', significantly speeding up
connectivity checks across many candidate refs with shared
history.
* td/ref-filter-memoize-contains:
commit-reach: die on contains walk errors
ref-filter: memoize --contains with generations
commit-reach: reject cycles in contains walk
"git branch" command learned "--prune-merged" option to remove
local branches that have already been merged to the remote-tracking
branches they track.
* hn/branch-prune-merged:
branch: add --dry-run for --prune-merged
branch: add branch.<name>.pruneMerged opt-out
branch: add --prune-merged <branch>
branch: prepare delete_branches for a bulk caller
branch: let delete_branches warn instead of error on bulk refusal
branch: add --forked filter for --list mode
The lazy priority queue optimization pattern (deferring actual removal
in prio_queue_get() to allow get+put fusion) has been folded directly
into prio_queue itself, speeding up commit traversal workflows and
simplifying callers.
* kk/prio-queue-get-put-fusion:
prio-queue: fold lazy_queue into prio_queue for automatic get+put fusion
prio-queue: rename .nr to .nr_ and add accessor helpers
The `remote-object-info` command has been added to `git cat-file
--batch-command`, allowing clients to request object metadata
(currently size) from a remote server via protocol v2 without
downloading the entire object.
The client dynamically filters format placeholders based on
server-advertised capabilities and safely returns empty strings for
inapplicable or unsupported fields.
* ps/cat-file-remote-object-info:
cat-file: make remote-object-info allow-list dynamic
cat-file: validate remote atoms with allow_list
cat-file: add remote-object-info to batch-command
transport: add client support for object-info
serve: advertise object-info feature
fetch-pack: move fetch initialization
connect: refactor packet writing
fetch-pack: move function to connect.c
t1006: split test utility functions into new "lib-cat-file.sh"
cat-file: add declaration of variable i inside its for loop
git-compat-util: add strtoul_ul() with error handling
transport-helper: fix memory leak of helper on disconnect
Two new mechanisms, the GIT_CONFIG_INCLUDES environment variable and
the top-level --no-includes command-line option, have been introduced
to ignore configuration include directives.
* ds/config-no-includes:
git: add --no-includes top-level option
config: add GIT_CONFIG_INCLUDES
git-config.adoc: fix paragraph break
Commands that list branches and tags (like git branch and git tag)
have been optimized to pass the namespace prefix when initializing
their ref iterator, avoiding a loose-ref scaling regression in
repositories with many unrelated loose references.
* td/ref-filter-restore-prefix-iteration:
ref-filter: restore prefix-scoped iteration
The packed object source has been refactored into a proper struct
odb_source.
* ps/odb-source-packed:
odb/source-packed: drop pointer to "files" parent source
midx: refactor interfaces to work on "packed" source
odb/source-packed: stub out remaining functions
odb/source-packed: wire up `freshen_object()` callback
odb/source-packed: wire up `find_abbrev_len()` callback
odb/source-packed: wire up `count_objects()` callback
odb/source-packed: wire up `for_each_object()` callback
odb/source-packed: wire up `read_object_stream()` callback
odb/source-packed: wire up `read_object_info()` callback
packfile: use higher-level interface to implement `has_object_pack()`
odb/source-packed: wire up `reprepare()` callback
odb/source-packed: wire up `close()` callback
odb/source-packed: start converting to a proper `struct odb_source`
odb/source-packed: store pointer to "files" instead of generic source
packfile: move packed source into "odb/" subsystem
packfile: split out packfile list logic
packfile: rename `struct packfile_store` to `odb_source_packed`
When cURL follows a redirect, the WWW-Authenticate headers from the
redirect target were lost because credential_from_url() cleared the
credential state. This has been fixed by preserving the collected
headers across the redirect update.
* ap/http-redirect-wwwauth-fix:
http: preserve wwwauth_headers across redirects
A regression in the error diagnosis code for invalid .git files has
been fixed, avoiding a potential NULL-pointer crash when reporting
that a .git file does not point to a valid repository.
* jk/setup-gitfile-diag-fix:
read_gitfile_gently(): return non-repo path on error
The experimental "git history" command has been taught a new "drop"
subcommand to remove a commit and replay its descendants onto its
parent.
* ps/history-drop:
builtin/history: implement "drop" subcommand
builtin/history: split handling of ref updates into two phases
reset: stop assuming that the caller passes in a clean index
reset: allow the caller to specify the current HEAD object
reset: introduce ability to skip updating HEAD
reset: introduce dry-run mode
reset: modernize flags passed to `reset_working_tree()`
reset: rename `reset_head()`
reset: drop `USE_THE_REPOSITORY_VARIABLE`
read-cache: split out function to drop unmerged entries to stage 0
The "git repo info" command has been taught new keys to output both
absolute and relative paths for "gitdir" and "commondir", supported by
a new path-formatting helper extracted from "git rev-parse".
* jk/repo-info-path-keys:
repo: add path.gitdir with absolute and relative suffix formatting
repo: add path.commondir with absolute and relative suffix formatting
rev-parse: use append_formatted_path() for path formatting
path: introduce append_formatted_path() for shared path formatting
prio_queue_get() has been optimized by using a cascade-down approach
(promoting the smaller child at each level and sifting up the last
element from the leaf vacancy), which halves the number of comparisons
per extract-min operation in the common case.
* kk/prio-queue-cascade-sift:
prio-queue: use cascade-down for faster extract-min
The 'trust_executable_bit' (coming from 'core.filemode'
configuration) has been migrated into 'repo_config_values' to tie it
to a specific repository instance.
* ty/migrate-trust-executable-bit:
environment: move trust_executable_bit into repo_config_values
read-cache: move 'ce_mode_from_stat()' to 'read-cache.c'
read-cache: remove redundant extern declarations
The pack-objects command now supports using reachability bitmaps and
delta-islands concurrently with the `--path-walk` option, allowing
faster packaging by falling back to path-walk when bitmaps cannot
fully satisfy the request.
* tb/pack-path-walk-bitmap-delta-islands:
pack-objects: support `--delta-islands` with `--path-walk`
pack-objects: extract `record_tree_depth()` helper
pack-objects: support reachability bitmaps with `--path-walk`
t/perf: drop p5311's lookup-table permutation
"git rebase --update-refs" has been taught to resolve local branch
symrefs to their referents before queuing updates. This correctly
skips aliases of the current branch and avoids duplicate updates for
underlying real branches, fixing failures when branch aliases (like a
default branch rename) are present.
* sn/rebase-update-refs-symrefs:
rebase: skip branch symref aliases
The -m/-F/-c/-C options to supply commit log message from outside the
editor are now supported for all "git commit --fixup" variations.
* ec/commit-fixup-options:
commit: allow -c/-C for all kinds of --fixup
commit: allow -m/-F for all kinds of --fixup
The [includeIf "condition"] conditional inclusion facility for
configuration files has learned to use the location of worktree
in its condition.
* cl/conditional-config-on-worktree-path:
config: add "worktree" and "worktree/i" includeIf conditions
config: refactor include_by_gitdir() into include_by_path()
When fetching from a transport that provides a self-contained pack,
pass the transport pointer to the post-fetch `check_connected()` call
to optimize connectivity check.
Retracted.
cf. <CAL71e4MrVqC1=AR6x0_8S=8kVqPdDkhgCZRb4etFsxTzd6s_8Q@mail.gmail.com>
* kk/fetch-store-ref-optimization:
fetch: pass transport to post-fetch connectivity check
"git checkout --track=..." learned to optionally fetch the branch
from the remote the new branch will work with.
* hn/checkout-track-fetch:
checkout: extend --track with a "fetch" mode to refresh start-point
branch: expose helpers for finding the remote owning a tracking ref
Configuration file locking now retries for a short period, avoiding
failures when multiple processes attempt to update the configuration
simultaneously.
* jt/config-lock-timeout:
config: retry acquiring config.lock, configurable via core.configLockTimeout
The parse-options library learned to auto-correct misspelled
subcommand names.
* js/parseopt-subcommand-autocorrection:
SQUASH???
doc: document autocorrect API
parseopt: add tests for subcommand autocorrection
parseopt: enable subcommand autocorrection for git-remote and git-notes
parseopt: autocorrect mistyped subcommands
autocorrect: provide config resolution API
autocorrect: rename AUTOCORRECT_SHOW to AUTOCORRECT_HINT
autocorrect: use mode and delay instead of magic numbers
help: move tty check for autocorrection to autocorrect.c
help: make autocorrect handling reusable
parseopt: extract subcommand handling from parse_options_step()
The display of the rebase todo list in "git status" has been
improved to correctly abbreviate object IDs for more commands and
avoid misinterpreting refs as object IDs.
* pw/status-rebase-todo:
status: improve rebase todo list parsing
sequencer: factor out parsing of todo commands
When fetching objects into a lazily cloned repository, .promisor
files are created with information meant to help debugging. "git
repack" has been taught to carry this information forward to
packfiles that are newly created.
Retracted.
cf. <agx_GPfBKpkSc3Gx@lorenzo-VM>
* lp/repack-propagate-promisor-debugging-info:
repack-promisor: add missing headers
t7703: test for promisor file content after geometric repack
t7700: test for promisor file content after repack
repack-promisor: preserve content of promisor files after repack
repack-promisor add helper to fill promisor file after repack
pack-write: add explanation to promisor file content
"ort" merge backend handles merging corrupt trees better by
aborting when it should.
* en/ort-harden-against-corrupt-trees:
cache-tree: fix verify_cache() to catch non-adjacent D/F conflicts
merge-ort: abort merge when trees have duplicate entries
merge-ort: free diff pairs queue in clear_or_reinit_internal_opts()
merge-ort: drop unnecessary show_all_errors from collect_merge_info()
merge-ort: propagate callback errors from traverse_trees_wrapper()
Documentation updates.
* kh/doc-trailers:
doc: interpret-trailers: document comment line treatment
doc: interpret-trailers: commit to “trailer block” term
doc: interpret-trailers: join new-trailers again
doc: interpret-trailers: add key format example
doc: interpret-trailers: explain key format
doc: interpret-trailers: explain the format after the intro
doc: interpret-trailers: not just for commit messages
doc: interpret-trailers: use “metadata” in Name as well
doc: interpret-trailers: replace “lines” with “metadata”
doc: interpret-trailers: stop fixating on RFC 822
The path completion for commands like `git rm` and `git mv`, is being
updated to hide dotfiles by default, unless the user explicitly starts
the path with a dot, matching standard shell-completion behavior.
* za/completion-hide-dotfiles:
completion: hide dotfiles for selected path completion
Doc update for "git replay" to actually refer to its configuration
variables.
* kh/doc-replay-config:
doc: replay: move “default” to the right-hand side
doc: replay: use a nested description list
doc: replay: improve config description
doc: link to config for git-replay(1)
`git ls-files --modified` and `git ls-files --deleted` have been
optimized to filter with pathspec before calling lstat() when there is
only a single pathspec item, avoiding unnecessary filesystem access
for entries that will not be shown.
* td/ls-files-pathspec-prefilter:
ls-files: filter pathspec before lstat
Various AsciiDoc markup fixes in 'git config' documentation and
related files to ensure lists and formatting are rendered correctly.
* ta/doc-config-adoc-fixes:
doc: git-config: escape erroneous highlight markup
doc: config/sideband: fix description list delimiter
doc: config: terminate runaway lists
Continuation of "setup.c" refactoring to drop remaining global state
(`git_work_tree_cfg`, `is_bare_repository_cfg`). The most notable
outcome is that `is_bare_repository()` has been updated to no longer
implicitly rely on `the_repository`.
* ps/setup-drop-global-state:
treewide: drop USE_THE_REPOSITORY_VARIABLE
environment: stop using `the_repository` in `is_bare_repository()`
environment: split up concerns of `is_bare_repository_cfg`
builtin/init: stop modifying `is_bare_repository_cfg`
setup: remove global `git_work_tree_cfg` variable
builtin/init: simplify logic to configure worktree
builtin/init: stop modifying global `git_work_tree_cfg` variable
'git describe' has been taught to pass the 'refs/tags/' prefix down to
the ref iterator when '--all' is not requested, avoiding unnecessary
iteration over non-tag refs.
* td/describe-tag-iteration:
describe: limit default ref iteration to tags
The global configuration variables protect_hfs and protect_ntfs have
been migrated into struct repo_config_values to tie them to
per-repository configuration state.
* ty/move-protect-hfs-ntfs:
environment: move 'protect_hfs' and 'protect_ntfs' into 'repo_config_values'
The TSAN race in transfer_debug() within transport-helper.c has been
resolved by initializing the debug flag early in
bidirectional_transfer_loop() before spawning worker threads, allowing
the removal of a TSAN suppression.
* ps/transport-helper-tsan-fix:
transport-helper: fix TSAN race in transfer_debug()
Project-specific configuration for b4 has been introduced, and the
documentation has been updated to recommend using it as a
streamlined method for submitting patches.
* ps/doc-recommend-b4:
b4: introduce configuration for the Git project
MyFirstContribution: recommend the use of b4
MyFirstContribution: recommend shallow threading of cover letters
The handling of promisor-remote protocol capability has been
loosened to allow the other side to add to the list of promisor
remotes via the promisor.acceptFromServerURL configuration
variable.
* cc/promisor-auto-config-url-more:
doc: promisor: improve acceptFromServer entry
promisor-remote: auto-configure unknown remotes
promisor-remote: trust known remotes matching acceptFromServerUrl
promisor-remote: introduce promisor.acceptFromServerUrl
promisor-remote: add 'local_name' to 'struct promisor_info'
urlmatch: add url_normalize_pattern() helper
urlmatch: change 'allow_globs' arg to bool
t5710: simplify 'mkdir X' followed by 'git -C X init'
Advice shown by "git status" when the local branch is behind or has
diverged from its push branch has been updated to suggest "git pull
<remote> <branch>".
* hn/status-pull-advice-qualified:
remote: qualify "git pull" advice for non-upstream compareBranches
"git index-pack" has been optimized by retaining child bases in the
delta cache instead of immediately freeing them, letting the existing
cache limit policy decide eviction.
* ab/index-pack-retain-child-bases:
index-pack: retain child bases in delta cache
