This patch introduces support to set special NTFS attributes that are
interpreted by the Windows Subsystem for Linux as file mode bits, UID
and GID.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
With this patch, Git for Windows works as intended on mounted APFS
volumes (where renaming read-only files would fail).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
These are Git for Windows' Git GUI and gitk patches. We will have to
decide at some point what to do about them, but that's a little lower
priority (as Git GUI seems to be unmaintained for the time being, and
the gitk maintainer keeps a very low profile on the Git mailing list,
too).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Previously, we did not install any handler for Ctrl+C, but now we really
want to because the MSYS2 runtime learned the trick to call the
ConsoleCtrlHandler when Ctrl+C was pressed.
With this, hitting Ctrl+C while `git log` is running will only terminate
the Git process, but not the pager. This finally matches the behavior on
Linux and on macOS.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The Windows Subsystem for Linux (WSL) version 2 allows to use `chmod` on
NTFS volumes provided that they are mounted with metadata enabled (see
https://devblogs.microsoft.com/commandline/chmod-chown-wsl-improvements/
for details), for example:
$ chmod 0755 /mnt/d/test/a.sh
In order to facilitate better collaboration between the Windows
version of Git and the WSL version of Git, we can make the Windows
version of Git also support reading and writing NTFS file modes
in a manner compatible with WSL.
Since this slightly slows down operations where lots of files are
created (such as an initial checkout), this feature is only enabled when
`core.WSLCompat` is set to true. Note that you also have to set
`core.fileMode=true` in repositories that have been initialized without
enabling WSL compatibility.
There are several ways to enable metadata loading for NTFS volumes
in WSL, one of which is to modify `/etc/wsl.conf` by adding:
```
[automount]
enabled = true
options = "metadata,umask=027,fmask=117"
```
And reboot WSL.
It can also be enabled temporarily by this incantation:
$ sudo umount /mnt/c &&
sudo mount -t drvfs C: /mnt/c -o metadata,uid=1000,gid=1000,umask=22,fmask=111
It's important to note that this modification is compatible with, but
does not depend on WSL. The helper functions in this commit can operate
independently and functions normally on devices where WSL is not
installed or properly configured.
Signed-off-by: xungeng li <xungeng@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
On Windows, the current working directory is pretty much guaranteed to
contain a colon. If we feed that path to CVS, it mistakes it for a
separator between host and port, though.
This has not been a problem so far because Git for Windows uses MSYS2's
Bash using a POSIX emulation layer that also pretends that the current
directory is a Unix path (at least as long as we're in a shell script).
However, that is rather limiting, as Git for Windows also explores other
ports of other Unix shells. One of those is BusyBox-w32's ash, which is
a native port (i.e. *not* using any POSIX emulation layer, and certainly
not emulating Unix paths).
So let's just detect if there is a colon in $PWD and punt in that case.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
To verify that the symlink is resolved correctly, we use the fact that
`git.exe` is a native Win32 program, and that `git.exe config -f <path>`
therefore uses the native symlink resolution.
Signed-off-by: Bert Belder <bertbelder@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
At least on _some_ APFS network shares, Git fails to rename the object
files because they are marked as read-only, because that has the effect
of setting the uchg flag on APFS, which then means the file can't be
renamed or deleted.
To work around that, when a rename failed, and the read-only flag is
set, try to turn it off and on again.
This fixes https://github.com/git-for-windows/git/issues/4482
Signed-off-by: David Lomas <dl3@pale-eds.co.uk>
Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
... even if they may look like them.
As looking up the target of the "symbolic link" (just to see whether it
starts with `/ContainerMappedDirectories/`) is pretty expensive, we
do it when we can be *really* sure that there is a possibility that this
might be the case.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: JiSeop Moon <zcube@zcube.kr>
To support Git Bash running in a MinTTY, we use a dirty trick to access
the MSYS2 pseudo terminal: we execute a Bash snippet that accesses
/dev/tty.
The idea was to fall back to writing to/reading from CONOUT$/CONIN$ if
that Bash call failed because Bash was not found.
However, we should fall back even in other error conditions, because we
have not successfully read the user input. Let's make it so.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
On Windows, git repositories may have extra files which need cleaned
(e.g., a build directory) that may be arbitrarily deep. Suggest using
`core.longPaths` if such situations are encountered.
Fixes: #2715
Signed-off-by: Ben Boeckel <mathstuf@gmail.com>
The TerminateProcess() function does not actually leave the child
processes any chance to perform any cleanup operations. This is bad
insofar as Git itself expects its signal handlers to run.
A symptom is e.g. a left-behind .lock file that would not be left behind
if the same operation was run, say, on Linux.
To remedy this situation, we use an obscure trick: we inject a thread
into the process that needs to be killed and to let that thread run the
ExitProcess() function with the desired exit status. Thanks J Wyman for
describing this trick.
The advantage is that the ExitProcess() function lets the atexit
handlers run. While this is still different from what Git expects (i.e.
running a signal handler), in practice Git sets up signal handlers and
atexit handlers that call the same code to clean up after itself.
In case that the gentle method to terminate the process failed, we still
fall back to calling TerminateProcess(), but in that case we now also
make sure that processes spawned by the spawned process are terminated;
TerminateProcess() does not give the spawned process a chance to do so
itself.
Please note that this change only affects how Git for Windows tries to
terminate processes spawned by Git's own executables. Third-party
software that *calls* Git and wants to terminate it *still* need to make
sure to imitate this gentle method, otherwise this patch will not have
any effect.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Git for Windows uses MSYS2's Bash to run the test suite, which comes
with benefits but also at a heavy price: on the plus side, MSYS2's
POSIX emulation layer allows us to continue pretending that we are on a
Unix system, e.g. use Unix paths instead of Windows ones, yet this is
bought at a rather noticeable performance penalty.
There *are* some more native ports of Unix shells out there, though,
most notably BusyBox-w32's ash. These native ports do not use any POSIX
emulation layer (or at most a *very* thin one, choosing to avoid
features such as fork() that are expensive to emulate on Windows), and
they use native Windows paths (usually with forward slashes instead of
backslashes, which is perfectly legal in almost all use cases).
And here comes the problem: with a $PWD looking like, say,
C:/git-sdk-64/usr/src/git/t/trash directory.t5813-proto-disable-ssh
Git's test scripts get quite a bit confused, as their assumptions have
been shattered. Not only does this path contain a colon (oh no!), it
also does not start with a slash.
This is a problem e.g. when constructing a URL as t5813 does it:
ssh://remote$PWD. Not only is it impossible to separate the "host" from
the path with a $PWD as above, even prefixing $PWD by a slash won't
work, as /C:/git-sdk-64/... is not a valid path.
As a workaround, detect when $PWD does not start with a slash on
Windows, and simply strip the drive prefix, using an obscure feature of
Windows paths: if an absolute Windows path starts with a slash, it is
implicitly prefixed by the drive prefix of the current directory. As we
are talking about the current directory here, anyway, that strategy
works.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
On Windows, symbolic links have a type: a "file symlink" must point at
a file, and a "directory symlink" must point at a directory. If the
type of symlink does not match its target, it doesn't work.
Git does not record the type of symlink in the index or in a tree. On
checkout it'll guess the type, which only works if the target exists
at the time the symlink is created. This may often not be the case,
for example when the link points at a directory inside a submodule.
By specifying `symlink=file` or `symlink=dir` the user can specify what
type of symlink Git should create, so Git doesn't have to rely on
unreliable heuristics.
Signed-off-by: Bert Belder <bertbelder@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
In preparation for making this function a bit more complicated (to allow
for special-casing the `ContainerMappedDirectories` in Windows
containers, which look like a symbolic link, but are not), let's move it
out of the header.
Signed-off-by: JiSeop Moon <zcube@zcube.kr>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Accessing the Windows console through the special CONIN$ / CONOUT$ devices
doesn't work properly for non-ASCII usernames an passwords.
It also doesn't work for terminal emulators that hide the native console
window (such as mintty), and 'TERM=xterm*' is not necessarily a reliable
indicator for such terminals.
The new shell_prompt() function, on the other hand, works fine for both
MSys1 and MSys2, in native console windows as well as mintty, and properly
supports Unicode. It just needs bash on the path (for 'read -s', which is
bash-specific).
On Windows, try to use the shell to read from the terminal. If that fails
with ENOENT (i.e. bash was not found), use CONIN/OUT as fallback.
Note: To test this, create a UTF-8 credential file with non-ASCII chars,
e.g. in git-bash: 'echo url=http://täst.com > cred.txt'. Then in git-cmd,
'git credential fill <cred.txt' works (shell version), while calling git
without the git-wrapper (i.e. 'mingw64\bin\git credential fill <cred.txt')
mangles non-ASCII chars in both console output and input.
Signed-off-by: Karsten Blees <blees@dcon.de>
Update wchar_t buffers to use MAX_LONG_PATH instead of MAX_PATH and call
xutftowcs_long_path() in the Win32 backend source files.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
When t5605 tries to verify that files are hardlinked (or that they are
not), it uses the `-links` option of the `find` utility.
BusyBox' implementation does not support that option, and BusyBox-w32's
lstat() does not even report the number of hard links correctly (for
performance reasons).
So let's just switch to a different method that actually works on
Windows.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
On Windows, symbolic links actually have a type depending on the target:
it can be a file or a directory.
In certain circumstances, this poses problems, e.g. when a symbolic link
is supposed to point into a submodule that is not checked out, so there
is no way for Git to auto-detect the type.
To help with that, we will add support over the course of the next
commits to specify that symlink type via the Git attributes. This
requires an index_state, though, something that Git for Windows'
`symlink()` replacement cannot know about because the function signature
is defined by the POSIX standard and not ours to change.
So let's introduce a helper function to create symbolic links that
*does* know about the index_state.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
It is a known issue that a rename() can fail with an "Access denied"
error at times, when copying followed by deleting the original file
works. Let's just fall back to that behavior.
Signed-off-by: JiSeop Moon <zcube@zcube.kr>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The `git_terminal_prompt()` function expects the terminal window to be
attached to a Win32 Console. However, this is not the case with terminal
windows other than `cmd.exe`'s, e.g. with MSys2's own `mintty`.
Non-cmd terminals such as `mintty` still have to have a Win32 Console
to be proper console programs, but have to hide the Win32 Console to
be able to provide more flexibility (such as being resizeable not only
vertically but also horizontally). By writing to that Win32 Console,
`git_terminal_prompt()` manages only to send the prompt to nowhere and
to wait for input from a Console to which the user has no access.
This commit introduces a function specifically to support `mintty` -- or
other terminals that are compatible with MSys2's `/dev/tty` emulation. We
use the `TERM` environment variable as an indicator for that: if the value
starts with "xterm" (such as `mintty`'s "xterm_256color"), we prefer to
let `xterm_prompt()` handle the user interaction.
The most prominent user of `git_terminal_prompt()` is certainly
`git-remote-https.exe`. It is an interesting use case because both
`stdin` and `stdout` are redirected when Git calls said executable, yet
it still wants to access the terminal.
When running inside a `mintty`, the terminal is not accessible to the
`git-remote-https.exe` program, though, because it is a MinGW program
and the `mintty` terminal is not backed by a Win32 console.
To solve that problem, we simply call out to the shell -- which is an
*MSys2* program and can therefore access `/dev/tty`.
Helped-by: nalla <nalla@hamal.uberspace.de>
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When trying to ensure that long paths are handled correctly, we
first normalize absolute paths as we encounter them.
However, if the path is a so-called "drive-less" absolute path, i.e. if
it is relative to the current drive but _does_ start with a directory
separator, we would want the normalized path to be such a drive-less
absolute path, too.
Let's do that, being careful to still include the drive prefix when we
need to go through the `\\?\` dance (because there, the drive prefix is
absolutely required).
This fixes https://github.com/git-for-windows/git/issues/4586.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
While it may seem super convenient to some old Unix hands to simpy
require Perl to be available when running the test suite, this is a
major hassle on Windows, where we want to verify that Perl is not,
actually, required in a NO_PERL build.
As a super ugly workaround, we "install" a script into /usr/bin/perl
reading like this:
#!/bin/sh
# We'd much rather avoid requiring Perl altogether when testing
# an installed Git. Oh well, that's why we cannot have nice
# things.
exec c:/git-sdk-64/usr/bin/perl.exe "$@"
The problem with that is that BusyBox assumes that the #! line in a
script refers to an executable, not to a script. So when it encounters
the line #!/usr/bin/perl in t5532's proxy-get-cmd, it barfs.
Let's help this situation by simply executing the Perl script with the
"interpreter" specified explicitly.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Windows paths are typically limited to MAX_PATH = 260 characters, even
though the underlying NTFS file system supports paths up to 32,767 chars.
This limitation is also evident in Windows Explorer, cmd.exe and many
other applications (including IDEs).
Particularly annoying is that most Windows APIs return bogus error codes
if a relative path only barely exceeds MAX_PATH in conjunction with the
current directory, e.g. ERROR_PATH_NOT_FOUND / ENOENT instead of the
infinitely more helpful ERROR_FILENAME_EXCED_RANGE / ENAMETOOLONG.
Many Windows wide char APIs support longer than MAX_PATH paths through the
file namespace prefix ('\\?\' or '\\?\UNC\') followed by an absolute path.
Notable exceptions include functions dealing with executables and the
current directory (CreateProcess, LoadLibrary, Get/SetCurrentDirectory) as
well as the entire shell API (ShellExecute, SHGetSpecialFolderPath...).
Introduce a handle_long_path function to check the length of a specified
path properly (and fail with ENAMETOOLONG), and to optionally expand long
paths using the '\\?\' file namespace prefix. Short paths will not be
modified, so we don't need to worry about device names (NUL, CON, AUX).
Contrary to MSDN docs, the GetFullPathNameW function doesn't seem to be
limited to MAX_PATH (at least not on Win7), so we can use it to do the
heavy lifting of the conversion (translate '/' to '\', eliminate '.' and
'..', and make an absolute path).
Add long path error checking to xutftowcs_path for APIs with hard MAX_PATH
limit.
Add a new MAX_LONG_PATH constant and xutftowcs_long_path function for APIs
that support long paths.
While improved error checking is always active, long paths support must be
explicitly enabled via 'core.longpaths' option. This is to prevent end
users from shooting themselves in the foot by checking out files that Windows
Explorer, cmd/bash or their favorite IDE cannot handle.
Test suite:
Test the case is when the full pathname length of a dir is close
to 260 (MAX_PATH).
Bug report and an original reproducer by Andrey Rogozhnikov:
https://github.com/msysgit/git/pull/122#issuecomment-43604199
[jes: adjusted test number to avoid conflicts, added support for
chdir(), etc]
Thanks-to: Martin W. Kirst <maki@bitkings.de>
Thanks-to: Doug Kelly <dougk.ff7@gmail.com>
Original-test-by: Andrey Rogozhnikov <rogozhnikov.andrey@gmail.com>
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Stepan Kasal <kasal@ucw.cz>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Josh Soref <jsoref@gmail.com>
At some stage, t5003-archive-zip wants to add a file that is not ASCII.
To that end, it uses /bin/sh. But that file may actually not exist (it
is too easy to forget that not all the world is Unix/Linux...)! Besides,
we already have perfectly fine binary files intended for use solely by
the tests. So let's use one of them instead.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
There is a problem in the way 9ac3f0e5b3 (pack-objects: fix
performance issues on packing large deltas, 2018-07-22) initializes that
mutex in the `packing_data` struct. The problem manifests in a
segmentation fault on Windows, when a mutex (AKA critical section) is
accessed without being initialized. (With pthreads, you apparently do
not really have to initialize them?)
This was reported in https://github.com/git-for-windows/git/issues/1839.
Signed-off-by: Doug Kelly <dougk.ff7@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When running with BusyBox, we will want to avoid calling executables on
the PATH that are implemented in BusyBox itself.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We already avoid traversing NTFS junction points in `git clean -dfx`.
With this topic branch, we do that when the FSCache is enabled, too.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The -W option is only understood by MSYS2 Bash's pwd command. We already
make sure to override `pwd` by `builtin pwd -W` for MINGW, so let's not
double the effort here.
This will also help when switching the shell to another one (such as
BusyBox' ash) whose pwd does *not* understand the -W option.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This brings substantial wins in performance because the FSCache is now
per-thread, being merged to the primary thread only at the end, so we do
not have to lock (except while merging).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The `git clean` command needs to enumerate plenty of files and
directories, and can therefore benefit from the FSCache.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Traditionally, Git for Windows' SDK uses Bash as its default shell.
However, other Unix shells are available, too. Most notably, the Win32
port of BusyBox comes with `ash` whose `pwd` command already prints
Windows paths as Git for Windows wants them, while there is not even a
`builtin` command.
Therefore, let's be careful not to override `pwd` unless we know that
the `builtin` command is available.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Using FindFirstFileExW() requires the OS to allocate a 64K buffer for each
directory and then free it when we call FindClose(). Update fscache to call
the underlying kernel API NtQueryDirectoryFile so that we can do the buffer
management ourselves. That allows us to allocate a single buffer for the
lifetime of the cache and reuse it for each directory.
This change improves performance of 'git status' by 18% in a repo with ~200K
files and 30k folders.
Documentation for NtQueryDirectoryFile can be found at:
https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/content/ntifs/nf-ntifs-ntquerydirectoryfilehttps://docs.microsoft.com/en-us/windows/desktop/FileIO/file-attribute-constantshttps://docs.microsoft.com/en-us/windows/desktop/fileio/reparse-point-tags
To determine if the specified directory is a symbolic link, inspect the
FileAttributes member to see if the FILE_ATTRIBUTE_REPARSE_POINT flag is
set. If so, EaSize will contain the reparse tag (this is a so far
undocumented feature, but confirmed by the NTFS developers). To
determine if the reparse point is a symbolic link (and not some other
form of reparse point), test whether the tag value equals the value
IO_REPARSE_TAG_SYMLINK.
The NtQueryDirectoryFile() call works best (and on Windows 8.1 and
earlier, it works *only*) with buffer sizes up to 64kB. Which is 32k
wide characters, so let's use that as our buffer size.
Signed-off-by: Ben Peart <benpeart@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When FSCache is active, we can cache the reparse tag and use it directly
to determine whether a path refers to an NTFS junction, without any
additional, costly I/O.
Note: this change only makes a difference with the next commit, which
will make use of the FSCache in `git clean` (contingent on
`core.fscache` set, of course).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
BusyBox-w32 is a true Win32 application, i.e. it does not come with a
POSIX emulation layer.
That also means that it does *not* use the Unix convention of separating
the entries in the PATH variable using colons, but semicolons.
However, there are also BusyBox ports to Windows which use a POSIX
emulation layer such as Cygwin's or MSYS2's runtime, i.e. using colons
as PATH separators.
As a tell-tale, let's use the presence of semicolons in the PATH
variable: on Unix, it is highly unlikely that it contains semicolons,
and on Windows (without POSIX emulation), it is virtually guaranteed, as
everybody should have both $SYSTEMROOT and $SYSTEMROOT/system32 in their
PATH.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The recent change to make fscache thread specific relied on fscache_enable()
being called first from the primary thread before being called in parallel
from worker threads. Make that more robust and protect it with a critical
section to avoid any issues.
Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Ben Peart <benpeart@microsoft.com>
We will use this in the next commit to implement an FSCache-aware
version of is_mount_point().
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The idea is to allow running the test suite on MinGit with BusyBox
installed in /mingw64/bin/sh.exe. In that case, we will want to exclude
sort & find (and other Unix utilities) from being bundled.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Now that the fscache is single threaded, take advantage of the mem_pool as
the allocator to significantly reduce the cost of allocations and frees.
With the reduced cost of free, in future patches, we can start freeing the
fscache at the end of commands instead of just leaking it.
Signed-off-by: Ben Peart <benpeart@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We already have a directory where we store files intended for use by
multiple test scripts. The same directory is a better home for the
test-binary-*.png files than t/.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
