The `index_blob_packfile_transaction()` function streams blob data
directly from an fd. This makes it difficult to reuse as part of a
generic transactional object writing interface.
Refactor the packfile write path to operate on a `struct
odb_write_stream`, allowing callers to supply data from arbitrary
sources.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In certain scenarios, Git handles writing blobs that exceed
"core.bigFileThreshold" differently by streaming the object directly
into a packfile. When there is an active ODB transaction, these blobs
are streamed to the same packfile instead of using a separate packfile
for each. If "pack.packSizeLimit" is configured and streaming another
object causes the packfile to exceed the configured limit, the packfile
is truncated back to the previous object and the object write is
restarted in a new packfile.
This works fine, but requires the fd being read from to save a
checkpoint so it becomes possible to rewind the input source via seeking
back to a known offset at the beginning. In a subsequent commit, blob
streaming is converted to use `struct odb_write_stream` as a more
generic input source instead of an fd which doesn't provide a mechanism
for rewinding.
For this use case though, rewinding the fd is not strictly necessary
because the inflated size of the object is known and can be used to
approximate whether writing the object would cause the packfile to
exceed the configured limit prior to writing anything. These blobs
written to the packfile are never deltified thus the size difference
between what is written versus the inflated size is due to zlib
compression. While this does prevent packfiles from being filled to the
potential maximum is some cases, it should be good enough and still
prevents the packfile from exceeding any configured limit.
Use the inflated blob size to determine whether writing an object to a
packfile will exceed the configured "pack.packSizeLimit".
Signed-off-by: Justin Tobler <jltobler@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The `index_blob_packfile_transaction()` function handles streaming a
blob from an fd to compute its object ID and conditionally writes the
object directly to a packfile if the INDEX_WRITE_OBJECT flag is set. A
subsequent commit will make these packfile object writes part of the
transaction interface. Consequently, having the object write be
conditional on this flag is a bit awkward.
In preparation for this change, introduce a dedicated
`hash_blob_stream()` helper that only computes the OID from a `struct
odb_write_stream`. This is invoked by `index_fd()` instead when the
INDEX_WRITE_OBJECT is not set. The object write performed via
`index_blob_packfile_transaction()` is made unconditional accordingly.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The `read()` callback used by `struct odb_write_stream` currently
returns a pointer to an internal buffer along with the number of bytes
read. This makes buffer ownership unclear and provides no way to report
errors.
Update the interface to instead require the caller to provide a buffer,
and have the callback return the number of bytes written to it or a
negative value on error. While at it, also move the `struct
odb_write_stream` definition to "odb/streaming.h". Call sites are
updated accordingly.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Each ODB source is expected to provide an ODB transaction implementation
that should be used when starting a transaction. With d6fc6fe6f8
(odb/source: make `begin_transaction()` function pluggable, 2026-03-05),
the `struct odb_source` now provides a pluggable callback for beginning
transactions. Use the callback provided by the ODB source accordingly.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The current ODB transaction interface is colocated with other ODB
interfaces in "odb.{c,h}". Subsequent commits will expand `struct
odb_transaction` to support write operations on the transaction
directly. To keep things organized and prevent "odb.{c,h}" from becoming
more unwieldy, split out `struct odb_transaction` into a separate
header.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The unsigned integer that is used as an bitset to specify the kind
of branches interpret_branch_name() function has been changed to
use a dedicated enum type.
* jw/object-name-bitset-to-enum:
object-name: turn INTERPRET_BRANCH_* constants into enum values
"git apply" now reports the name of the input file along with the
line number when it encounters a corrupt patch, and correctly
resets the line counter when processing multiple patch files.
* jw/apply-corrupt-location:
apply: report input location in binary and garbage patch errors
apply: report input location in header parsing errors
apply: report the location of corrupt patches
split-index.c has been updated to not use the global the_repository
and the_hash_algo variables.
* rs/split-index-the-repo-fix:
split-index: stop using the_repository and the_hash_algo
The cleanup of remaining bitmaps in "ahead_behind()" has been
simplified.
* rs/ahead-behind-cleanup-optimization:
commit-reach: simplify cleanup of remaining bitmaps in ahead_behind ()
Code clean-up overdue by 19 years.
* jc/rerere-modern-strbuf-handling:
cocci: strbuf.buf is never NULL
rerere: update to modern representation of empty strbufs
Doc updates.
* kh/doc-interpret-trailers-1:
interpret-trailers: use placeholder instead of *
doc: config: convert trailers section to synopsis style
doc: interpret-trailers: normalize and fill out options
doc: interpret-trailers: convert to synopsis style
The reference-transaction hook was taught to be triggered before
taking locks on references in the "preparing" phase.
* ej/ref-transaction-hook-preparing:
refs: add 'preparing' phase to the reference-transaction hook
merge-file --object-id used to trigger a BUG when run in a linked
worktree, which has been fixed.
* mr/merge-file-object-id-worktree-fix:
merge-file: fix BUG when --object-id is used in a worktree
Uses of prio_queue as a LIFO stack of commits have been written
with commit_stack.
* rs/prio-queue-to-commit-stack:
use commit_stack instead of prio_queue in LIFO mode
The handling of the incomplete lines at the end by "git
diff-highlight" has been fixed.
* jk/diff-highlight-identical-pairs:
contrib/diff-highlight: do not highlight identical pairs
The logic to count objects has been cleaned up.
* ps/object-counting:
odb: introduce generic object counting
odb/source: introduce generic object counting
object-file: generalize counting objects
object-file: extract logic to approximate object count
packfile: extract logic to count number of objects
odb: stop including "odb/source.h"
Further work on incremental repacking using MIDX/bitmap
* tb/incremental-midx-part-3.2:
midx: enable reachability bitmaps during MIDX compaction
midx: implement MIDX compaction
t/helper/test-read-midx.c: plug memory leak when selecting layer
midx-write.c: factor fanout layering from `compute_sorted_entries()`
midx-write.c: enumerate `pack_int_id` values directly
midx-write.c: extract `fill_pack_from_midx()`
midx-write.c: introduce `midx_pack_perm()` helper
midx: do not require packs to be sorted in lexicographic order
midx-write.c: introduce `struct write_midx_opts`
midx-write.c: don't use `pack_perm` when assigning `bitmap_pos`
t/t5319-multi-pack-index.sh: fix copy-and-paste error in t5319.39
git-multi-pack-index(1): align SYNOPSIS with 'git multi-pack-index -h'
git-multi-pack-index(1): remove non-existent incompatibility
builtin/multi-pack-index.c: make '--progress' a common option
midx: introduce `midx_get_checksum_hex()`
midx: rename `get_midx_checksum()` to `midx_get_checksum_hash()`
midx: mark `get_midx_checksum()` arguments as const
Add a coccinelle rule to break the build when "struct strbuf" gets
passed by value.
* dd/cocci-do-not-pass-strbuf-by-value:
stash: do not pass strbuf by value
coccinelle: detect struct strbuf passed by value
"git diff -U<num>" was too lenient in its command line parsing and
took an empty string as a valid <num>.
* ty/doc-diff-u-wo-number:
diff: document -U without <n> as using default context
Reduce system overhead "git upload-pack" spends on relaying "git
pack-objects" output to the "git fetch" running on the other end of
the connection.
* ps/upload-pack-buffer-more-writes:
builtin/pack-objects: reduce lock contention when writing packfile data
csum-file: drop `hashfd_throughput()`
csum-file: introduce `hashfd_ext()`
sideband: use writev(3p) to send pktlines
wrapper: introduce writev(3p) wrappers
compat/posix: introduce writev(3p) wrapper
upload-pack: reduce lock contention when writing packfile data
upload-pack: prefer flushing data over sending keepalive
upload-pack: adapt keepalives based on buffering
upload-pack: fix debug statement when flushing packfile data
The final clean-up phase of the diff output could turn the result of
histogram diff algorithm suboptimal, which has been corrected.
* yc/histogram-hunk-shift-fix:
xdiff: re-diff shifted change groups when using histogram algorithm
"git history" learned the "split" subcommand.
* ps/history-split:
builtin/history: implement "split" subcommand
builtin/history: split out extended function to create commits
cache-tree: allow writing in-memory index as tree
add-patch: allow disabling editing of hunks
add-patch: add support for in-memory index patching
add-patch: remove dependency on "add-interactive" subsystem
add-patch: split out `struct interactive_options`
add-patch: split out header from "add-interactive.h"
"git fast-import" learned to optionally replace signature on
commits whose signatures get invalidated due to replaying by
signing afresh.
* jt/fast-import-sign-again:
fast-import: add mode to sign commits with invalid signatures
gpg-interface: allow sign_buffer() to use default signing key
commit: remove unused forward declaration
"imap-send" used to use functions whose use is going to be removed
with OpenSSL 4.0; rewrite them using public API that has been
available since OpenSSL 1.1 since 2016 or so.
* bb/imap-send-openssl-4.0-prep:
imap-send: move common code into function host_matches()
imap-send: use the OpenSSL API to access the subject common name
imap-send: use the OpenSSL API to access the subject alternative names
The code in "git help" that shows configuration items in sorted
order was awkwardly organized and prone to bugs.
* ac/help-sort-correctly:
help: cleanup the contruction of keys_uniq
Instead of hardcoded 'origin', use the configured default remote
when fetching from submodules.
* ng/submodule-default-remote:
submodule: fetch missing objects from default remote
Small code clean-up around the constness area.
* cf/constness-fixes:
dir: avoid -Wdiscarded-qualifiers in remove_path()
bloom: remove a misleading const qualifier
* 'master' of https://github.com/j6t/git-gui:
git-gui: grey out comment lines in commit message
git-gui: wire up "git-gui--askyesno" with Meson
git-gui: massage "git-gui--askyesno" with "generate-script.sh"
git-gui: prefer shell at "/bin/sh" with Meson
git-gui: fix use of GIT_CEILING_DIRECTORIES
git-gui: shift tabstops to account for the first column of patch text
* 'master' of https://github.com/j6t/gitk:
gitk: l10n: make PO headers identify the Gitk project
gitk: ignore generated POT file
gitk: i18n: use "Gitk" as package name in POT file
gitk: commit translation files without file information
gitk: support link color in the Preferences dialog
gitk: use config settings for head/tag colors
