Commit Graph

116 Commits

Author SHA1 Message Date
Karsten Blees
04f0eea085 mingw: add a cache below mingw's lstat and dirent implementations
Checking the work tree status is quite slow on Windows, due to slow
`lstat()` emulation (git calls `lstat()` once for each file in the
index). Windows operating system APIs seem to be much better at scanning
the status of entire directories than checking single files.

Add an `lstat()` implementation that uses a cache for lstat data. Cache
misses read the entire parent directory and add it to the cache.
Subsequent `lstat()` calls for the same directory are served directly
from the cache.

Also implement `opendir()`/`readdir()`/`closedir()` so that they create
and use directory listings in the cache.

The cache doesn't track file system changes and doesn't plug into any
modifying file APIs, so it has to be explicitly enabled for git functions
that don't modify the working copy.

Note: in an earlier version of this patch, the cache was always active and
tracked file system changes via ReadDirectoryChangesW. However, this was
much more complex and had negative impact on the performance of modifying
git commands such as 'git checkout'.

Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2022-08-30 22:28:05 +02:00
Johannes Schindelin
e481084bd2 Merge branch 'ci-fixes'
Backport a couple fixes to make the CI build run again (so much for
reproducible builds...).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2022-08-30 22:28:04 +02:00
Johannes Schindelin
54fd462a31 vcxproj: avoid escaping double quotes in the defines
Visual Studio 2022 does not like that at all.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2022-08-30 22:28:01 +02:00
Johannes Schindelin
9ee72897ac vcxproj: handle libreftable_test, too
Since ef8a6c6268 (reftable: utility functions, 2021-10-07) we not only
have a libreftable, but also a libreftable_test.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2022-08-30 22:28:01 +02:00
Johannes Schindelin
c628fe3770 vcxproj: ignore the -pedantic option
This is now passed by default, ever since 6a8cbc41ba (developer: enable
pedantic by default, 2021-09-03).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2022-08-30 22:28:01 +02:00
Johannes Schindelin
02da463e98 vcxproj: require C11
This fixes the build after 7bc341e21b (git-compat-util: add a test
balloon for C99 support, 2021-12-01).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2022-08-30 22:28:01 +02:00
Johannes Schindelin
a9d8865d06 Merge pull request #3306 from PhilipOakley/vs-sln
Make Git for Windows start builds in modern Visual Studio
2022-08-30 22:23:47 +02:00
Johannes Schindelin
588d4b4373 Merge pull request #3327 from dennisameling/fix-host-cpu
cmake(): allow setting HOST_CPU for cross-compilation
2022-08-30 22:23:46 +02:00
Johannes Schindelin
369a76d892 Merge pull request #2915 from dennisameling/windows-arm64-support
Windows arm64 support
2022-08-30 22:23:45 +02:00
Philip Oakley
f43728f317 CMake: show Win32 and Generator_platform build-option values
Ensure key CMake option values are part of the CMake output to
facilitate user support when tool updates impact the wider CMake
actions, particularly ongoing 'improvements' in Visual Studio.

These CMake displays perform the same function as the build-options.txt
provided in the main Git for Windows. CMake is already chatty.
The setting of CMAKE_EXPORT_COMPILE_COMMANDS is also reported.

Include the environment's CMAKE_EXPORT_COMPILE_COMMANDS value which
may have been propogated to CMake's internal value.

Testing the CMAKE_EXPORT_COMPILE_COMMANDS processing can be difficult
in the Visual Studio environment, as it may be cached in many places.
The 'environment' may include the OS, the user shell, CMake's
own environment, along with the Visual Studio presets and caches.

See previous commit for arefacts that need removing for a clean test.

Signed-off-by: Philip Oakley <philipoakley@iee.email>
2022-08-30 22:21:36 +02:00
Philip Oakley
8b603fbc83 CMakeLists: add default "x64-windows" arch for Visual Studio
In Git-for-Windows, work on using ARM64 has progressed. The
commit 2d94b77b27 (cmake: allow building for Windows/ARM64, 2020-12-04)
failed to notice that /compat/vcbuild/vcpkg_install.bat will default to
using the "x64-windows" architecture for the vcpkg installation if not set,
but CMake is not told of this default. Commit 635b6d99b3 (vcbuild: install
ARM64 dependencies when building ARM64 binaries, 2020-01-31) later updated
vcpkg_install.bat to accept an arch (%1) parameter, but retained the default.

This default is neccessary for the use case where the project directory is
opened directly in Visual Studio, which will find and build a CMakeLists.txt
file without any parameters, thus expecting use of the default setting.

Also Visual studio will generate internal .sln solution and .vcxproj project
files needed for some extension tools. Inform users of the additional
.sln/.vcxproj generation.

** How to test:
 rm -rf '.vs' # remove old visual studio settings
 rm -rf 'compat/vcbuild/vcpkg' # remove any vcpkg downloads
 rm -rf 'contrib/buildsystems/out' # remove builds & CMake artifacts
 with a fresh Visual Studio Community Edition, File>>Open>>(git *folder*)
   to load the project (which will take some time!).
 check for successful compilation.
The implicit .sln (etc.) are in the hidden .vs directory created by
Visual Studio.

Signed-off-by: Philip Oakley <philipoakley@iee.email>
2022-08-30 22:21:36 +02:00
Philip Oakley
b771d3ad5c CMake: default Visual Studio generator has changed
Correct some wording and inform users regarding the Visual Studio
changes (from V16.6) to the default generator.

Subsequent commits ensure that Git for Windows can be directly
opened in modern Visual Studio without needing special configuration
of the CMakeLists settings.

It appeares that internally Visual Studio creates it's own version of the
.sln file (etc.) for extension tools that expect them.

The large number of references below document the shifting of Visual Studio
default and CMake setting options.

refs: https://docs.microsoft.com/en-us/search/?scope=C%2B%2B&view=msvc-150&terms=Ninja

1. https://docs.microsoft.com/en-us/cpp/linux/cmake-linux-configure?view=msvc-160
(note the linux bit)
 "In Visual Studio 2019 version 16.6 or later ***, Ninja is the default
generator for configurations targeting a remote system or WSL. For more
information, see this post on the C++ Team Blog
[https://devblogs.microsoft.com/cppblog/linux-development-with-visual-studio-first-class-support-for-gdbserver-improved-build-times-with-ninja-and-updates-to-the-connection-manager/].

For more information about these settings, see CMakeSettings.json reference
[https://docs.microsoft.com/en-us/cpp/build/cmakesettings-reference?view=msvc-160]."

2. https://docs.microsoft.com/en-us/cpp/build/cmake-presets-vs?view=msvc-160
"CMake supports two files that allow users to specify common configure,
build, and test options and share them with others: CMakePresets.json
and CMakeUserPresets.json."

" Both files are supported in Visual Studio 2019 version 16.10 or later.
***"
3. https://devblogs.microsoft.com/cppblog/linux-development-with-visual-studio-first-class-support-for-gdbserver-improved-build-times-with-ninja-and-updates-to-the-connection-manager/
" Ninja has been the default generator (underlying build system) for
CMake configurations targeting Windows for some time***, but in Visual
Studio 2019 version 16.6 Preview 3*** we added support for Ninja on Linux."

4. https://docs.microsoft.com/en-us/cpp/build/cmakesettings-reference?view=msvc-160
" `generator`: specifies CMake generator to use for this configuration.
May be one of:

    Visual Studio 2019 only:
        Visual Studio 16 2019
        Visual Studio 16 2019 Win64
        Visual Studio 16 2019 ARM

    Visual Studio 2017 and later:
        Visual Studio 15 2017
        Visual Studio 15 2017 Win64
        Visual Studio 15 2017 ARM
        Visual Studio 14 2015
        Visual Studio 14 2015 Win64
        Visual Studio 14 2015 ARM
        Unix Makefiles
        Ninja

Because Ninja is designed for fast build speeds instead of flexibility
and function, it is set as the default. However, some CMake projects may
be unable to correctly build using Ninja. If this occurs, you can
instruct CMake to generate Visual Studio projects instead.

To specify a Visual Studio generator in Visual Studio 2017, open the
settings editor from the main menu by choosing CMake | Change CMake
Settings. Delete "Ninja" and type "V". This activates IntelliSense,
which enables you to choose the generator you want."

"To specify a Visual Studio generator in Visual Studio 2019, right-click
on the CMakeLists.txt file in Solution Explorer and choose CMake
Settings for project > Show Advanced Settings > CMake Generator.

When the active configuration specifies a Visual Studio generator, by
default MSBuild.exe is invoked with` -m -v:minimal` arguments."

5. https://docs.microsoft.com/en-us/cpp/build/cmake-presets-vs?view=msvc-160#enable-cmakepresetsjson-integration-in-visual-studio-2019
"Enable CMakePresets.json integration in Visual Studio 2019

CMakePresets.json integration isn't enabled by default in Visual Studio
2019. You can enable it for all CMake projects in Tools > Options >
CMake > General: (tick a box)" ... see more.

6. https://docs.microsoft.com/en-us/cpp/build/cmakesettings-reference?view=msvc-140
(whichever v140 is..)
"CMake projects are supported in Visual Studio 2017 and later."

7. https://docs.microsoft.com/en-us/cpp/overview/what-s-new-for-cpp-2017?view=msvc-150
"Support added for the CMake Ninja generator."

8. https://docs.microsoft.com/en-us/cpp/overview/what-s-new-for-cpp-2017?view=msvc-150#cmake-support-via-open-folder
"CMake support via Open Folder
Visual Studio 2017 introduces support for using CMake projects without
converting to MSBuild project files (.vcxproj). For more information,
see CMake projects in Visual
Studio[https://docs.microsoft.com/en-us/cpp/build/cmake-projects-in-visual-studio?view=msvc-150].
Opening CMake projects with Open Folder automatically configures the
environment for C++ editing, building, and debugging." ... +more!

9. https://docs.microsoft.com/en-us/cpp/build/cmake-presets-vs?view=msvc-160#supported-cmake-and-cmakepresetsjson-versions
"Visual Studio reads and evaluates CMakePresets.json and
CMakeUserPresets.json itself and doesn't invoke CMake directly with the
--preset option. So, CMake version 3.20 or later isn't strictly required
when you're building with CMakePresets.json inside Visual Studio. We
recommend using CMake version 3.14 or later."

10. https://docs.microsoft.com/en-us/cpp/build/cmake-presets-vs?view=msvc-160#enable-cmakepresetsjson-integration-in-visual-studio-2019
"If you don't want to enable CMakePresets.json integration for all CMake
projects, you can enable CMakePresets.json integration for a single
CMake project by adding a CMakePresets.json file to the root of the open
folder. You must close and reopen the folder in Visual Studio to
activate the integration.

11. https://docs.microsoft.com/en-us/cpp/build/cmake-presets-vs?view=msvc-160#default-configure-presets
***(doesn't actually say which version..)
"Default Configure Presets
If no CMakePresets.json or CMakeUserPresets.json file exists, or if
CMakePresets.json or CMakeUserPresets.json is invalid, Visual Studio
will fall back*** on the following default Configure Presets:

Windows example
JSON
{
  "name": "windows-default",
  "displayName": "Windows x64 Debug",
  "description": "Sets Ninja generator, compilers, x64 architecture,
build and install directory, debug build type",
  "generator": "Ninja",
  "binaryDir": "${sourceDir}/out/build/${presetName}",
  "architecture": {
    "value": "x64",
    "strategy": "external"
  },
  "cacheVariables": {
    "CMAKE_BUILD_TYPE": "Debug",
    "CMAKE_INSTALL_PREFIX": "${sourceDir}/out/install/${presetName}"
  },
  "vendor": {
    "microsoft.com/VisualStudioSettings/CMake/1.0": {
      "hostOS": [ "Windows" ]
    }
  }
},
"

Signed-off-by: Philip Oakley <philipoakley@iee.email>
2022-08-30 22:21:36 +02:00
Dennis Ameling
c8bd5242fc cmake(): allow setting HOST_CPU for cross-compilation
Git's regular Makefile mentions that HOST_CPU should be defined when cross-compiling Git: 37796bca76/Makefile (L438-L439)

This is then used to set the GIT_HOST_CPU variable when compiling Git: 37796bca76/Makefile (L1337-L1341)

Then, when the user runs `git version --build-options`, it returns that value: 37796bca76/help.c (L658)

This commit adds the same functionality to the CMake configuration. Users can now set -DHOST_CPU= to set the target architecture.

Signed-off-by: Dennis Ameling <dennis@dennisameling.com>
2022-08-30 22:21:34 +02:00
Dennis Ameling
98cd770653 cmake: allow building for Windows/ARM64
Signed-off-by: Dennis Ameling <dennis@dennisameling.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2022-08-30 22:21:33 +02:00
Ian Bearman
38b7c5da5b vcbuild: install ARM64 dependencies when building ARM64 binaries
Co-authored-by: Dennis Ameling <dennis@dennisameling.com>
Signed-off-by: Ian Bearman <ianb@microsoft.com>
Signed-off-by: Dennis Ameling <dennis@dennisameling.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2022-08-30 22:21:33 +02:00
Ian Bearman
10370480f8 vcxproj: support building Windows/ARM64 binaries
Signed-off-by: Ian Bearman <ianb@microsoft.com>
Signed-off-by: Dennis Ameling <dennis@dennisameling.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2022-08-30 22:21:33 +02:00
Johannes Schindelin
788496e7d8 win32: add a helper to run git.exe without a foreground window
On Windows, there are two kinds of executables, console ones and
non-console ones. Git's executables are all console ones.

When launching the former e.g. in a scheduled task, a CMD window pops
up. This is not what we want for the tasks installed via the `git
maintenance` command.

To work around this, let's introduce `headless-git.exe`, which is a
non-console program that does _not_ pop up any window. All it does is to
re-launch `git.exe`, suppressing that console window, passing through
all command-line arguments as-are.

Helped-by: Carlo Marcelo Arenas Belón <carenas@gmail.com>
Helped-by: Yuyi Wang <Strawberry_Str@hotmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
2022-08-30 22:21:32 +02:00
Johannes Schindelin
90f1b5b447 vcxproj: handle GUI programs, too
So far, we only built Console programs, but we are about to introduce a
program that targets the Windows subsystem (i.e. it is a so-called "GUI"
program).

Let's handle this preemptively in the script that generates the Visual
Studio files.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2022-08-30 22:21:32 +02:00
Johannes Schindelin
920266a51d vcxproj: ignore -fno-stack-protector and -fno-common
An upcoming commit will introduce those compile options; MSVC does not
understand them, so let's suppress them when generating the Visual
Studio project files.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2022-08-30 22:21:32 +02:00
Johannes Schindelin
15e8d38565 vcxproj: handle resource files, too
On Windows, we also compile a "resource" file, which is similar to
source code, but contains metadata (such as the program version).

So far, we did not compile it in `MSVC` mode, only when compiling Git
for Windows with the GNU C Compiler.

In preparation for including it also when compiling with MS Visual C,
let's teach our `vcxproj` generator to handle those sort of files, too.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2022-08-30 22:21:32 +02:00
Johannes Schindelin
c49614d0aa buildsystems: remove duplicate clause
This seems to have been there since 259d87c354 (Add scripts to
generate projects for other buildsystems (MSVC vcproj, QMake),
2009-09-16), i.e. since the beginning of that file.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2022-08-30 22:21:32 +02:00
Johannes Schindelin
ef31fd2696 vcxproj: unclash project directories with build outputs
It already caused problems with the test suite that the directory
containing `git.vcxproj` is called the same as the Git executable
without its file extension: `./git` is ambiguous, it could refer both to
the directory `git/` as well as to `git.exe`.

Now there is one more problem: when our GitHub workflow runs on the
`vs/master` branch, it fails in all but the Windows builds, as they want
to write the file `git` but there is already a directory in the way.

Let's just go ahead and append `.proj` to all of those directories, e.g.
`git.proj/` instead of `git/`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2022-08-30 22:21:30 +02:00
Carlo Marcelo Arenas Belón
476e54b1c6 cmake: support local installations of git
At least in systems where the user is local and not an administrator
git will install in a subdirectory of %APPDATALOCAL%, so it makes
sense to also look there for the shell needed by the cmake integration
with Visual Studio.

Signed-off-by: Carlo Marcelo Arenas Belón <carenas@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-07-27 08:57:33 -07:00
Junio C Hamano
9e496fffc8 Merge branch 'jh/builtin-fsmonitor-part3'
More fsmonitor--daemon.

* jh/builtin-fsmonitor-part3: (30 commits)
  t7527: improve implicit shutdown testing in fsmonitor--daemon
  fsmonitor--daemon: allow --super-prefix argument
  t7527: test Unicode NFC/NFD handling on MacOS
  t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd
  t/helper/hexdump: add helper to print hexdump of stdin
  fsmonitor: on macOS also emit NFC spelling for NFD pathname
  t7527: test FSMonitor on case insensitive+preserving file system
  fsmonitor: never set CE_FSMONITOR_VALID on submodules
  t/perf/p7527: add perf test for builtin FSMonitor
  t7527: FSMonitor tests for directory moves
  fsmonitor: optimize processing of directory events
  fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed
  fsm-health-win32: force shutdown daemon if worktree root moves
  fsm-health-win32: add polling framework to monitor daemon health
  fsmonitor--daemon: stub in health thread
  fsmonitor--daemon: rename listener thread related variables
  fsmonitor--daemon: prepare for adding health thread
  fsmonitor--daemon: cd out of worktree root
  fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS
  unpack-trees: initialize fsmonitor_has_run_once in o->result
  ...
2022-06-10 15:04:15 -07:00
Jeff Hostetler
d06055501b fsmonitor--daemon: stub in health thread
Create another thread to watch over the daemon process and
automatically shut it down if necessary.

This commit creates the basic framework for a "health" thread
to monitor the daemon and/or the file system.  Later commits
will add platform-specific code to do the actual work.

The "health" thread is intended to monitor conditions that
would be difficult to track inside the IPC thread pool and/or
the file system listener threads.  For example, when there are
file system events outside of the watched worktree root or if
we want to have an idle-timeout auto-shutdown feature.

This commit creates the health thread itself, defines the thread-proc
and sets up the thread's event loop.  It integrates this new thread
into the existing IPC and Listener thread models.

This commit defines the API to the platform-specific code where all of
the monitoring will actually happen.

The platform-specific code for MacOS is just stubs.  Meaning that the
health thread will immediately exit on MacOS, but that is OK and
expected.  Future work can define MacOS-specific monitoring.

The platform-specific code for Windows sets up enough of the
WaitForMultipleObjects() machinery to watch for system and/or custom
events.  Currently, the set of wait handles only includes our custom
shutdown event (sent from our other theads).  Later commits in this
series will extend the set of wait handles to monitor other
conditions.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-05-26 15:59:27 -07:00
Jeff Hostetler
a85ad67bbd fsmonitor-settings: stub in macOS-specific incompatibility checking
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-05-26 15:59:26 -07:00
Jeff Hostetler
d33c804dae fsmonitor-settings: stub in Win32-specific incompatibility checking
Extend generic incompatibility checkout with platform-specific
mechanism.  Stub in Win32 version.

In the existing fsmonitor-settings code we have a way to mark
types of repos as incompatible with fsmonitor (whether via the
hook and IPC APIs).  For example, we do this for bare repos,
since there are no files to watch.

Extend this exclusion mechanism for platform-specific reasons.
This commit just creates the framework and adds a stub for Win32.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-05-26 15:59:26 -07:00
Yuyi Wang
5ec7110822 cmake: remove (_)UNICODE def on Windows in CMakeLists.txt
`UNICODE` and `_UNICODE` are not required when building git on Windows.
Actually, they should not be predefined at all.

There're 2 evidences that `(_)UNICODE` is supposed to be nonexist:

compat/win32/trace2_win32_process_info.c:83: It uses jw_array_string
which accepts pe32.szExeFile as const char*.

t/helper/test-drop-caches.c:16: Calling to GetCurrentDirectory with
Buffer as char*.

The autotools build system never defines `UNICODE` and `_UNICODE` and
builds on Windows well.

Signed-off-by: Yuyi Wang <Strawberry_Str@hotmail.com>
Acked-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-05-24 16:06:02 -07:00
Yuyi Wang
80431510a2 cmake: add pcre2 support
Fix one of the TODOs listed in the CMakeLists.txt by adding support
for building with pcre2.

As pcre2 doesn't provide cmake find module, we find it with pkgconf.
This patch also works with vcpkg on Windows, with pkgconf and pcre2
installed.

Pkgconf and pcre2 is detected automatically just like curl, expat
and iconv. The output of CMake indicates whether pcre2 is found.

Signed-off-by: Yuyi Wang <Strawberry_Str@hotmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-05-24 16:05:21 -07:00
Yuyi Wang
a561962479 cmake: fix CMakeLists.txt on Linux
CMakeLists.txt didn't follow the grammar of `set`, and it will fail when
setting `USE_VCPKG` off on non-Windows platforms.

When the platform is Linux, the Makefile adds `compat/linux/procinfo.o`
to `COMPAT_OBJS`, but the CMakeLists.txt didn't add
`compat/linux/procinfo.c` to `compat_SOURCES`. It would cause linkage
error.

Signed-off-by: Yuyi Wang <Strawberry_Str@hotmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-05-24 16:05:00 -07:00
Junio C Hamano
439c1e6d5d Merge branch 'jh/builtin-fsmonitor-part2'
Built-in fsmonitor (part 2).

* jh/builtin-fsmonitor-part2: (30 commits)
  t7527: test status with untracked-cache and fsmonitor--daemon
  fsmonitor: force update index after large responses
  fsmonitor--daemon: use a cookie file to sync with file system
  fsmonitor--daemon: periodically truncate list of modified files
  t/perf/p7519: add fsmonitor--daemon test cases
  t/perf/p7519: speed up test on Windows
  t/perf/p7519: fix coding style
  t/helper/test-chmtime: skip directories on Windows
  t/perf: avoid copying builtin fsmonitor files into test repo
  t7527: create test for fsmonitor--daemon
  t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon
  help: include fsmonitor--daemon feature flag in version info
  fsmonitor--daemon: implement handle_client callback
  compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on MacOS
  compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent
  compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on Windows
  fsmonitor--daemon: create token-based changed path cache
  fsmonitor--daemon: define token-ids
  fsmonitor--daemon: add pathname classification
  fsmonitor--daemon: implement 'start' command
  ...
2022-04-04 10:56:24 -07:00
Jeff Hostetler
f67df2556f compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin
Stub in empty implementation of fsmonitor--daemon
backend for Darwin (aka MacOS).

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-25 16:04:15 -07:00
Jeff Hostetler
62c7367133 compat/fsmonitor/fsm-listen-win32: stub in backend for Windows
Stub in empty filesystem listener backend for fsmonitor--daemon on Windows.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-25 16:04:15 -07:00
Neeraj Singh
abf38abec2 core.fsyncmethod: add writeout-only mode
This commit introduces the `core.fsyncMethod` configuration
knob, which can currently be set to `fsync` or `writeout-only`.

The new writeout-only mode attempts to tell the operating system to
flush its in-memory page cache to the storage hardware without issuing a
CACHE_FLUSH command to the storage controller.

Writeout-only fsync is significantly faster than a vanilla fsync on
common hardware, since data is written to a disk-side cache rather than
all the way to a durable medium. Later changes in this patch series will
take advantage of this primitive to implement batching of hardware
flushes.

When git_fsync is called with FSYNC_WRITEOUT_ONLY, it may fail and the
caller is expected to do an ordinary fsync as needed.

On Apple platforms, the fsync system call does not issue a CACHE_FLUSH
directive to the storage controller. This change updates fsync to do
fcntl(F_FULLFSYNC) to make fsync actually durable. We maintain parity
with existing behavior on Apple platforms by setting the default value
of the new core.fsyncMethod option.

Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-10 15:10:22 -08:00
brian m. carlson
05cd988dce wrapper: add a helper to generate numbers from a CSPRNG
There are many situations in which having access to a cryptographically
secure pseudorandom number generator (CSPRNG) is helpful.  In the
future, we'll encounter one of these when dealing with temporary files.
To make this possible, let's add a function which reads from a system
CSPRNG and returns some bytes.

We know that all systems will have such an interface.  A CSPRNG is
required for a secure TLS or SSH implementation and a Git implementation
which provided neither would be of little practical use.  In addition,
POSIX is set to standardize getentropy(2) in the next version, so in the
(potentially distant) future we can rely on that.

For systems which lack one of the other interfaces, we provide the
ability to use OpenSSL's CSPRNG.  OpenSSL is highly portable and
functions on practically every known OS, and we know it will have access
to some source of cryptographically secure randomness.  We also provide
support for the arc4random in libbsd for folks who would prefer to use
that.

Because this is a security sensitive interface, we take some
precautions.  We either succeed by filling the buffer completely as we
requested, or we fail.  We don't return partial data because the caller
will almost never find that to be a useful behavior.

Specify a makefile knob which users can use to specify one or more
suitable CSPRNGs, and turn the multiple string options into a set of
defines, since we cannot match on strings in the preprocessor.  We allow
multiple options to make the job of handling this in autoconf easier.

The order of options is important here.  On systems with arc4random,
which is most of the BSDs, we use that, since, except on MirBSD and
macOS, it uses ChaCha20, which is extremely fast, and sits entirely in
userspace, avoiding a system call.  We then prefer getrandom over
getentropy, because the former has been available longer on Linux, and
then OpenSSL. Finally, if none of those are available, we use
/dev/urandom, because most Unix-like operating systems provide that API.
We prefer options that don't involve device files when possible because
those work in some restricted environments where device files may not be
available.

Set the configuration variables appropriately for Linux and the BSDs,
including macOS, as well as Windows and NonStop.  We specifically only
consider versions which receive publicly available security support
here.  For the same reason, we don't specify getrandom(2) on Linux,
because CentOS 7 doesn't support it in glibc (although its kernel does)
and we don't want to resort to making syscalls.

Finally, add a test helper to allow this to be tested by hand and in
tests.  We don't add any tests, since invoking the CSPRNG is not likely
to produce interesting, reproducible results.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-01-17 14:17:48 -08:00
Junio C Hamano
a4bbd13be3 Merge branch 'hn/reftable'
The "reftable" backend for the refs API, without integrating into
the refs subsystem, has been added.

* hn/reftable:
  Add "test-tool dump-reftable" command.
  reftable: add dump utility
  reftable: implement stack, a mutable database of reftable files.
  reftable: implement refname validation
  reftable: add merged table view
  reftable: add a heap-based priority queue for reftable records
  reftable: reftable file level tests
  reftable: read reftable files
  reftable: generic interface to tables
  reftable: write reftable files
  reftable: a generic binary tree implementation
  reftable: reading/writing blocks
  Provide zlib's uncompress2 from compat/zlib-compat.c
  reftable: (de)serialization for the polymorphic record type.
  reftable: add blocksource, an abstraction for random access reads
  reftable: utility functions
  reftable: add error related functionality
  reftable: add LICENSE
  hash.h: provide constants for the hash IDs
2021-12-15 09:39:45 -08:00
Junio C Hamano
d67fc4bf0b Merge branch 'bc/require-c99'
Weather balloon to break people with compilers that do not support
C99.

* bc/require-c99:
  git-compat-util: add a test balloon for C99 support
2021-12-10 14:35:14 -08:00
brian m. carlson
7bc341e21b git-compat-util: add a test balloon for C99 support
The C99 standard was released in January 1999, now 22 years ago.  It
provides a variety of useful features, including variadic arguments for
macros, declarations after statements, designated initializers, and a
wide variety of other useful features, many of which we already use.

We'd like to take advantage of these features, but we want to be
cautious.  As far as we know, all major compilers now support C99 or a
later C standard, such as C11 or C17.  POSIX has required C99 support as
a requirement for the 2001 revision, so we can safely assume any POSIX
system which we are interested in supporting has C99.

Even MSVC, long a holdout against modern C, now supports both C11 and
C17 with an appropriate update.  Moreover, even if people are using an
older version of MSVC on these systems, they will generally need some
implementation of the standard Unix utilities for the testsuite, and GNU
coreutils, the most common option, has required C99 since 2009.
Therefore, we can safely assume that a suitable version of GCC or clang
is available to users even if their version of MSVC is not sufficiently
capable.

Let's add a test balloon to git-compat-util.h to see if anyone is using
an older compiler.  We'll add a comment telling people how to enable
this functionality on GCC and Clang, even though modern versions of both
will automatically do the right thing, and ask people still experiencing
a problem to report that to us on the list.

Note that C89 compilers don't provide the __STDC_VERSION__ macro, so we
use a well-known hack of using "- 0".  On compilers with this macro, it
doesn't change the value, and on C89 compilers, the macro will be
replaced with nothing, and our value will be 0.

For sparse, we explicitly request the gnu99 style because we've
traditionally taken advantage of some GCC- and clang-specific extensions
when available and we'd like to retain the ability to do that.  sparse
also defaults to C89 without it, so things will fail for us if we don't.

Update the cmake configuration to require C11 for MSVC.  We do this
because this will make MSVC to use C11, since it does not explicitly
support C99.  We do this with a compiler options because setting the
C_STANDARD option does not work in our CI on MSVC and at the moment, we
don't want to require C11 for Unix compilers.

In the Makefile, don't set any compiler flags for the compiler itself,
since on some systems, such as FreeBSD, we actually need C11, and asking
for C99 causes things to fail to compile.  The error message should make
it obvious what's going wrong and allow a user to set the appropriate
option when building in the event they're using a Unix compiler that
doesn't support it by default.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-12-01 14:50:01 -08:00
Ævar Arnfjörð Bjarmason
ebeb39faad git-sh-setup: remove "sane_grep", it's not needed anymore
Remove the sane_grep() shell function in git-sh-setup. The two reasons
for why it existed don't apply anymore:

1. It was added due to GNU grep supporting GREP_OPTIONS. See
   e1622bfcba (Protect scripted Porcelains from GREP_OPTIONS insanity,
   2009-11-23).

   Newer versions of GNU grep ignore that, but even on older versions
   its existence won't matter, none of these sane_grep() uses care
   about grep's output, they're merely using it to check if a string
   exists in a file or stream. We also don't care about the "LC_ALL=C"
   that "sane_grep" was using, these greps for fixed or ASCII strings
   will behave the same under any locale.

2. The SANE_TEXT_GREP added in 71b401032b (sane_grep: pass "-a" if
   grep accepts it, 2016-03-08) isn't needed either, none of these grep
   uses deal with binary data.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-21 16:17:57 -07:00
Han-Wen Nienhuys
ef8a6c6268 reftable: utility functions
This commit provides basic utility classes for the reftable library.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-08 10:45:48 -07:00
Ævar Arnfjörð Bjarmason
cfe853e66b hook-list.h: add a generated list of hooks, like config-list.h
Make githooks(5) the source of truth for what hooks git supports, and
punt out early on hooks we don't know about in find_hook(). This
ensures that the documentation and the C code's idea about existing
hooks doesn't diverge.

We still have Perl and Python code running its own hooks, but that'll
be addressed by Emily Shaffer's upcoming "git hook run" command.

This resolves a long-standing TODO item in bugreport.c of there being
no centralized listing of hooks, and fixes a bug with the bugreport
listing only knowing about 1/4 of the p4 hooks. It didn't know about
the recent "reference-transaction" hook either.

We could make the find_hook() function die() or BUG() out if the new
known_hook() returned 0, but let's make it return NULL just as it does
when it can't find a hook of a known type. Making it die() is overly
anal, and unlikely to be what we need in catching stupid typos in the
name of some new hook hardcoded in git.git's sources. By making this
be tolerant of unknown hook names, changes in a later series to make
"git hook run" run arbitrary user-configured hook names will be easier
to implement.

I have not been able to directly test the CMake change being made
here. Since 4c2c38e800 (ci: modification of main.yml to use cmake for
vs-build job, 2020-06-26) some of the Windows CI has a hard dependency
on CMake, this change works there, and is to my eyes an obviously
correct use of a pattern established in previous CMake changes,
namely:

 - 061c2240b1 (Introduce CMake support for configuring Git,
    2020-06-12)
 - 709df95b78 (help: move list_config_help to builtin/help,
    2020-04-16)
 - 976aaedca0 (msvc: add a Makefile target to pre-generate the Visual
   Studio solution, 2019-07-29)

The LC_ALL=C is needed because at least in my locale the dash ("-") is
ignored for the purposes of sorting, which results in a different
order. I'm not aware of anything in git that has a hard dependency on
the order, but e.g. the bugreport output would end up using whatever
locale was in effect when git was compiled.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Helped-by: René Scharfe <l.s.r@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-27 09:44:54 -07:00
Junio C Hamano
bc34e5227b Merge branch 'js/gfw-system-config-loc-fix'
Update the location of system-side configuration file on Windows.

* js/gfw-system-config-loc-fix:
  config: normalize the path of the system gitconfig
  cmake(windows): set correct path to the system Git config
  mingw: move Git for Windows' system config where users expect it
2021-07-16 17:42:46 -07:00
Junio C Hamano
c3c0b71f9a Merge branch 'mr/cmake'
CMake update.

* mr/cmake:
  cmake: add warning for ignored MSGFMT_EXE
  cmake: create compile_commands.json by default
  cmake: add knob to disable vcpkg
2021-07-13 16:52:51 -07:00
Dennis Ameling
50101b93ca cmake(windows): set correct path to the system Git config
Currently, when Git for Windows is built with CMake, the system Git config is
expected in a different location than when building via `make`: the former
expects it to be in `<runtime-prefix>/mingw64/etc/gitconfig`, the latter in
`<runtime-prefix>/etc/gitconfig`.

Because of this, things like `git clone` do not work correctly (because cURL is
no longer able to find its certificate bundle that it needs to validate HTTPS
certificates). See the full bug report and discussion here:
https://github.com/git-for-windows/git/issues/3071#issuecomment-789261386.

This commit aligns the CMake-based build by mimicking what is already done in
`config.mak.uname`.

This closes https://github.com/git-for-windows/git/issues/3071.

Signed-off-by: Dennis Ameling <dennis@dennisameling.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-28 20:11:50 -07:00
Matthew Rogers
ce24797d38 cmake: add warning for ignored MSGFMT_EXE
It does not make sense to attempt to set MSGFMT_EXE when NO_GETTEXT is
configured, as such add a check for NO_GETTEXT before attempting to set
it.

Suggested-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Matthew Rogers <mattr94@gmail.com>
Acked-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-11 15:23:24 +09:00
Matthew Rogers
409047a2b3 cmake: create compile_commands.json by default
Some users have expressed interest in a more "batteries included" way of
building via CMake[1], and a big part of that is providing easier access
to tooling external tools.

A straightforward way to accomplish this is to make it as simple as
possible is to enable the generation of the compile_commands.json file,
which is supported by many tools such as: clang-tidy, clang-format,
sourcetrail, etc.

This does come with a small run-time overhead during the configuration
step (~6 seconds on my machine):

    Time to configure with CMAKE_EXPORT_COMPILE_COMMANDS=TRUE

    real    1m9.840s
    user    0m0.031s
    sys     0m0.031s

    Time to configure with CMAKE_EXPORT_COMPILE_COMMANDS=FALSE

    real    1m3.195s
    user    0m0.015s
    sys     0m0.015s

This seems like a small enough price to pay to make the project more
accessible to newer users.  Additionally there are other large projects
like llvm [2] which has had this enabled by default for >6 years at the
time of this writing, and no real negative consequences that I can find
with my search-skills.

NOTE: That the compile_commands.json is currently produced only when
using the Ninja and Makefile generators.  See The CMake documentation[3]
for more info.

1: https://lore.kernel.org/git/CAOjrSZusMSvs7AS-ZDsV8aQUgsF2ZA754vSDjgFKMRgi_oZAWw@mail.gmail.com/
2: 2c5712051b
3: https://cmake.org/cmake/help/latest/variable/CMAKE_EXPORT_COMPILE_COMMANDS.html

Signed-off-by: Matthew Rogers <mattr94@gmail.com>
Acked-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-11 15:23:17 +09:00
Matthew Rogers
cd0a852981 cmake: add knob to disable vcpkg
When building on windows users have the option to use vcpkg to provide
the dependencies needed to compile.  Previously, this was used only when
using the Visual Studio generator which was not ideal because:

  - Not all users who want to use vcpkg use the Visual Studio
    generators.

  - Some versions of Visual Studio 2019 moved away from using the
    VS 2019  generator by default, making it impossible for Visual
    Studio to configure the project in the likely event that it couldn't
    find the dependencies.

  - Inexperienced users of CMake are very likely to get tripped up by
    the errors caused by a lack of vcpkg, making the above bullet point
    both annoying and hard to debug.

As such, let's make using vcpkg the default on windows.  Users who want
to avoid using vcpkg can disable it by passing -DNO_VCPKG=TRUE.

Signed-off-by: Matthew Rogers <mattr94@gmail.com>
Acked-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-06-11 15:23:00 +09:00
Junio C Hamano
6aae0e2ad2 Merge branch 'jh/simple-ipc-sans-pthread'
The "simple-ipc" did not compile without pthreads support, but the
build procedure was not properly account for it.

* jh/simple-ipc-sans-pthread:
  simple-ipc: correct ifdefs when NO_PTHREADS is defined
2021-05-22 18:29:01 +09:00
Jeff Hostetler
6aac70a870 simple-ipc: correct ifdefs when NO_PTHREADS is defined
Simple IPC always requires threads (in addition to various
platform-specific IPC support).  Fix the ifdefs in the Makefile
to define SUPPORTS_SIMPLE_IPC when appropriate.

Previously, the Unix version of the code would only verify that
Unix domain sockets were available.

This problem was reported here:
https://lore.kernel.org/git/YKN5lXs4AoK%2FJFTO@coredump.intra.peff.net/T/#m08be8f1942ea8a2c36cfee0e51cdf06489fdeafc

Reported-by: Randall S. Becker <rsbecker@nexbridge.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-21 07:55:00 +09:00
Junio C Hamano
a548f3e0ad Merge branch 'js/cmake-vsbuild'
CMake update for vsbuild.

* js/cmake-vsbuild:
  cmake(install): include vcpkg dlls
  cmake: add a preparatory work-around to accommodate `vcpkg`
  cmake(install): fix double .exe suffixes
  cmake: support SKIP_DASHED_BUILT_INS
2021-04-07 16:54:08 -07:00