Ensure that when passing a pipe, the gnulib poll replacement will not
return 0 before the timeout has passed.
Not obeying the timeout (and merely returning 0) causes pathological
behavior when preparing a packfile for a repository and taking a
long time to do so. If poll were to return 0 immediately, this would
cause keep-alives to get sent as quickly as possible until the packfile
was created. Such deviance from the standard would cause megabytes (or
more) of keep-alive packets to be sent.
GetTickCount is used as it is efficient, stable and monotonically
increasing. (Neither GetSystemTime nor QueryPerformanceCounter have
all three of these properties.)
Windows does not necessarily mean Cygwin, it could also be MSYS. The
latter ships with a version of "kill" that does not understand "-f". In
msysgit this was addressed shipping Cygwin's version of kill.
Properly fix this by using the stock Windows "taskkill" command instead,
which is available since Windows XP Professional.
Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
Our current version of bash 3.1.17(5) can not parse the following snippet
correctly
p=abcd
abspath=/$p
subdir="x$(echo "$p" | tail -c $((253 - ${#abspath})))"
as it returns
tail: cannot open `253' for reading: No such file or directory
This is fixed in bash 3.1.20(4), I did not check earlier versions.
Signed-off-by: Thomas Braun <thomas.braun@byte-physics.de>
Windows paths are typically limited to MAX_PATH = 260 characters, even
though the underlying NTFS file system supports paths up to 32,767 chars.
This limitation is also evident in Windows Explorer, cmd.exe and many
other applications (including IDEs).
Particularly annoying is that most Windows APIs return bogus error codes
if a relative path only barely exceeds MAX_PATH in conjunction with the
current directory, e.g. ERROR_PATH_NOT_FOUND / ENOENT instead of the
infinitely more helpful ERROR_FILENAME_EXCED_RANGE / ENAMETOOLONG.
Many Windows wide char APIs support longer than MAX_PATH paths through the
file namespace prefix ('\\?\' or '\\?\UNC\') followed by an absolute path.
Notable exceptions include functions dealing with executables and the
current directory (CreateProcess, LoadLibrary, Get/SetCurrentDirectory) as
well as the entire shell API (ShellExecute, SHGetSpecialFolderPath...).
Introduce a handle_long_path function to check the length of a specified
path properly (and fail with ENAMETOOLONG), and to optionally expand long
paths using the '\\?\' file namespace prefix. Short paths will not be
modified, so we don't need to worry about device names (NUL, CON, AUX).
Contrary to MSDN docs, the GetFullPathNameW function doesn't seem to be
limited to MAX_PATH (at least not on Win7), so we can use it to do the
heavy lifting of the conversion (translate '/' to '\', eliminate '.' and
'..', and make an absolute path).
Add long path error checking to xutftowcs_path for APIs with hard MAX_PATH
limit.
Add a new MAX_LONG_PATH constant and xutftowcs_long_path function for APIs
that support long paths.
While improved error checking is always active, long paths support must be
explicitly enabled via 'core.longpaths' option. This is to prevent end
users to shoot themselves in the foot by checking out files that Windows
Explorer, cmd/bash or their favorite IDE cannot handle.
Test suite:
Test the case is when the full pathname length of a dir is close
to 260 (MAX_PATH).
Bug report and an original reproducer by Andrey Rogozhnikov:
https://github.com/msysgit/git/pull/122#issuecomment-43604199
Thanks-to: Martin W. Kirst <maki@bitkings.de>
Thanks-to: Doug Kelly <dougk.ff7@gmail.com>
Signed-off-by: Karsten Blees <blees@dcon.de>
Original-test-by: Andrey Rogozhnikov <rogozhnikov.andrey@gmail.com>
Signed-off-by: Stepan Kasal <kasal@ucw.cz>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
If multiple threads access a directory that is not yet in the cache, the
directory will be loaded by each thread. Only one of the results is added
to the cache, all others are leaked. This wastes performance and memory.
On cache miss, add a future object to the cache to indicate that the
directory is currently being loaded. Subsequent threads register themselves
with the future object and wait. When the first thread has loaded the
directory, it replaces the future object with the result and notifies
waiting threads.
Signed-off-by: Karsten Blees <blees@dcon.de>
Checking the work tree status is quite slow on Windows, due to slow lstat
emulation (git calls lstat once for each file in the index). Windows
operating system APIs seem to be much better at scanning the status
of entire directories than checking single files.
Add an lstat implementation that uses a cache for lstat data. Cache misses
read the entire parent directory and add it to the cache. Subsequent lstat
calls for the same directory are served directly from the cache.
Also implement opendir / readdir / closedir so that they create and use
directory listings in the cache.
The cache doesn't track file system changes and doesn't plug into any
modifying file APIs, so it has to be explicitly enabled for git functions
that don't modify the working copy.
Note: in an earlier version of this patch, the cache was always active and
tracked file system changes via ReadDirectoryChangesW. However, this was
much more complex and had negative impact on the performance of modifying
git commands such as 'git checkout'.
Signed-off-by: Karsten Blees <blees@dcon.de>
Add a macro to mark code sections that only read from the file system,
along with a config option and documentation.
This facilitates implementation of relatively simple file system level
caches without the need to synchronize with the file system.
Enable read-only sections for 'git status' and preload_index.
Signed-off-by: Karsten Blees <blees@dcon.de>
Emulating the POSIX lstat API on Windows via GetFileAttributes[Ex] is quite
slow. Windows operating system APIs seem to be much better at scanning the
status of entire directories than checking single files. A caching
implementation may improve performance by bulk-reading entire directories
or reusing data obtained via opendir / readdir.
Make the lstat implementation pluggable so that it can be switched at
runtime, e.g. based on a config option.
Signed-off-by: Karsten Blees <blees@dcon.de>
Emulating the POSIX dirent API on Windows via FindFirstFile/FindNextFile is
pretty staightforward, however, most of the information provided in the
WIN32_FIND_DATA structure is thrown away in the process. A more
sophisticated implementation may cache this data, e.g. for later reuse in
calls to lstat.
Make the dirent implementation pluggable so that it can be switched at
runtime, e.g. based on a config option.
Define a base DIR structure with pointers to readdir/closedir that match
the opendir implementation (i.e. similar to vtable pointers in OOP).
Define readdir/closedir so that they call the function pointers in the DIR
structure. This allows to choose the opendir implementation on a
call-by-call basis.
Move the fixed sized dirent.d_name buffer to the dirent-specific DIR
structure, as d_name may be implementation specific (e.g. a caching
implementation may just set d_name to point into the cache instead of
copying the entire file name string).
Signed-off-by: Karsten Blees <blees@dcon.de>
Since commit 0c499ea60f the send-pack builtin uses the side-band-64k
capability if advertised by the server.
Unfortunately this breaks pushing over the dump git protocol if used
over a network connection.
The detailed reasons for this breakage are (by courtesy of Jeff Preshing,
quoted from ttps://groups.google.com/d/msg/msysgit/at8D7J-h7mw/eaLujILGUWoJ):
----------------------------------------------------------------------------
MinGW wraps Windows sockets in CRT file descriptors in order to mimic the
functionality of POSIX sockets. This causes msvcrt.dll to treat sockets as
Installable File System (IFS) handles, calling ReadFile, WriteFile,
DuplicateHandle and CloseHandle on them. This approach works well in simple
cases on recent versions of Windows, but does not support all usage patterns.
In particular, using this approach, any attempt to read & write concurrently
on the same socket (from one or more processes) will deadlock in a scenario
where the read waits for a response from the server which is only invoked after
the write. This is what send_pack currently attempts to do in the use_sideband
codepath.
----------------------------------------------------------------------------
The new config option "sendpack.sideband" allows to override the side-band-64k
capability of the server, and thus makes the dump git protocol work.
Other transportation methods like ssh and http/https still benefit from
the sideband channel, therefore the default value of "sendpack.sideband"
is still true.
Signed-off-by: Thomas Braun <thomas.braun@byte-physics.de>
The previous implementation said that the filesystem information on
Windows is not reliable to determine whether a file is executable.
To find gather this information it was peeking into the first two bytes
of a file to see whether it looks executable.
Apart from the fact that on Windows executables are usually defined as
such by their extension it lead to slow opening of help file in some
situations.
When you have virus scanner running calling open on an executable file
is a potentially expensive operation. See the following measurements (in
seconds) for example.
With virus scanner running (coldcache):
$ ./a.exe /libexec/git-core/
before open (git-add.exe): 0.000000
after open (git-add.exe): 0.412873
before open (git-annotate.exe): 0.000175
after open (git-annotate.exe): 0.397925
before open (git-apply.exe): 0.000243
after open (git-apply.exe): 0.399996
before open (git-archive.exe): 0.000147
after open (git-archive.exe): 0.397783
before open (git-bisect--helper.exe): 0.000160
after open (git-bisect--helper.exe): 0.397700
before open (git-blame.exe): 0.000160
after open (git-blame.exe): 0.399136
...
With virus scanner running (hotcache):
$ ./a.exe /libexec/git-core/
before open (git-add.exe): 0.000000
after open (git-add.exe): 0.000325
before open (git-annotate.exe): 0.000229
after open (git-annotate.exe): 0.000177
before open (git-apply.exe): 0.000167
after open (git-apply.exe): 0.000150
before open (git-archive.exe): 0.000154
after open (git-archive.exe): 0.000156
before open (git-bisect--helper.exe): 0.000132
after open (git-bisect--helper.exe): 0.000180
before open (git-blame.exe): 0.000718
after open (git-blame.exe): 0.000724
...
This test did just list the given directory and open() each file in it.
With this patch I get:
$ time git help git
Launching default browser to display HTML ...
real 0m8.723s
user 0m0.000s
sys 0m0.000s
and without
$ time git help git
Launching default browser to display HTML ...
real 1m37.734s
user 0m0.000s
sys 0m0.031s
both tests with cold cache and giving the machine some time to settle
down after restart.
Signed-off-by: Heiko Voigt <heiko.voigt@mahr.de>
7ebac8cb94 made launching of .exe
externals work when installed in Unicode paths. But it broke launching
of non-.exe externals, no matter where they were installed. We now
correctly maintain the UTF-8 and UTF-16 paths in tandem in lookup_prog.
This fixes t5526, among others.
Signed-off-by: Adam Roben <adam@roben.org>
If Git were installed in a path containing non-ASCII characters,
commands such as git-am and git-submodule, which are implemented as
externals, would fail to launch with the following error:
> fatal: 'am' appears to be a git command, but we were not
> able to execute it. Maybe git-am is broken?
This was due to lookup_prog not being Unicode-aware. It was somehow
missed in 2ee5a1a14a.
Note that the only problem in this function was calling
GetFileAttributes instead of GetFileAttributesW. The calls to access()
were fine because access() is a macro which resolves to mingw_access,
which already handles Unicode correctly. But I changed lookup_prog to
use _waccess directly so that we only convert the path to UTF-16 once.
Signed-off-by: Adam Roben <adam@roben.org>
On Windows XP3 in git bash
git clone git@github.com:octocat/Spoon-Knife.git
cd Spoon-Knife
git gui
menu Remote\Fetch from\origin
error: cannot spawn git: No such file or directory
error: could not run rev-list
if u run
git fetch --all
it worked normal in git bash or gitgui tools
In second version CreateProcess get 'C:\Git\libexec\git-core/git.exe' in
first version - C:/Git/libexec/git-core/git.exe and not executes (unix
slashes)
after fixing C:\Git\libexec\git-core\git.exe or
C:/Git/libexec/git-core\git.exe it works normal
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
After importing anything with fast-import, we should always let the
garbage collector do its job, since the objects are written to disk
inefficiently.
This brings down an initial import of http://selenic.com/hg from about
230 megabytes to about 14.
In the future, we may want to make this configurable on a per-remote
basis, or maybe teach fast-import about it in the first place.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When calling `git fast-export a..a b` when a and b refer to the same
commit, nothing would be exported, and an incorrect reset line would
be printed for b ('from :0').
Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
The CVS tests expect `pwd` to return a POSIX-style directory. Let's skip
our MinGW-specific override to let `pwd` output a Windows-style directory
for that reason.
Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: 마누엘 <nalla@hamal.uberspace.de>
There are some issues with the git-svn test cases when they are
being run on windows under a MINGW build. Some things are not
available like the changing of the execute flag of shell scripts
via the chmod command. Also there were problems with folder names
that end with a dot on windows.
Signed-off-by: 마누엘 <nalla@users.noreply.github.com>
As per https://msdn.microsoft.com/en-us/library/x99tb11d.aspx:
The set of available locale names, languages, country/region
codes, and code pages includes all those supported by the Windows
NLS API except code pages that require more than two bytes per
character, such as UTF-7 and UTF-8.
Therefore, MinGW gettext cannot cope with UTF-8 at all, because it uses
the Win32 API internally.
However, when the test asks `locale -a` it reports that is_US.utf8 is
available, because that `locale` is actually an *MSys2* program (and MSys2
can cope with UTF-8 alright).
Let's just skip this test for MinGW Git altogether.
Helped-by: 마누엘 <nalla@users.noreply.github.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
There is a MinGW gettext.exe, but still no MinGW locale.exe. Instead the
MSys2 locale.exe kicks in, which corresponds to the MSys2 gettext.exe,
however. Therefore some assumptions of t0200 cannot be fulfilled when
running inside MSys2.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
t0027 is marked expensive, but really, for MinGW we want to run these
tests always.
Suggested by Thomas Braun.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Without this patch, t0027 expects the native end-of-lines to be a single
line feed character. On Windows, however, we set it to a carriage return
character followed by a line feed character. Thus, we have to modify
t0027 to expect different warnings depending on the end-of-line markers.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
A string of the form "@/abcd" is considered a file path
by the msys layer and therefore translated to a windows path.
Here the trick is to double the slashes.
The msys patch translation can be studied with the following
test program:
#include <stdio.h>
#include <stdlib.h>
int main(int argc, char** argv)
{
unsigned int i;
for(i=1; i < argc; i++)
printf("argv[%d]=%s\n",i, argv[i]);
exit(0);
}
Signed-off-by: Thomas Braun <thomas.braun@byte-physics.de>
MSys works very hard to convert Unix-style paths into DOS-style ones.
*Very* hard.
So hard, indeed, that
git blame -L/hello/,/green/
is translated into something like
git blame -LC:/msysgit/hello/,C:/msysgit/green/
As seen in msys_p2w in src\msys\msys\rt\src\winsup\cygwin\path.cc, line
3204ff:
case '-':
//
// here we check for POSIX paths as attributes to a POSIX switch.
//
...
seemingly absolute POSIX paths in single-letter options get expanded by
msys.dll unless they contain '=' or ';'.
So a quick and very dirty fix is to use '-L/;*evil/'. (Using an equal sign
works only when it is before a comma, so in the above example, /=*green/
would still be converted to a DOS-style path.)
Commit-message-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The test separator char is a colon which means any absolute paths on windows
confuse the tests that use global_excludes.
Suggested-by: Karsten Blees <karsten.blees@gmail.com>
Signed-off-by: Pat Thoyts <patthoyts@users.sourceforge.net>
Apparently the signal handling is not quite correct in the fsckobject
handling (most likely we rely on a side effect that lets us still output
some message after receiving a signal 13 but in the BuildHive setup this
fails intermittently).
As a consequence, the push in t5504 does fail as expected, but fails to
output anything (unexpected). Since this is good enough for now, let's
handle an empty output as success, too.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This happens only when the corresponding commits are not exported in
the current fast-export run. This can happen either when the relevant
commit is already marked, or when the commit is explicitly marked
as UNINTERESTING with a negative ref by another argument.
This breaks fast-export basec remote helpers.
Signed-off-by: Sverre Rabbelier <srabbelier@gmail.com>
MSys2 has a slightly different notion of what constitutes a tty than
the Microsoft C runtime. The former knows whether stdin/stdout/stderr
was redirected or not, while the latter looks for a Win32 Console.
In particular when we want to know whether to spawn a pager or not, we
would rather want to know what MSys2 thinks.
We are about to introduce a change to the msys2-runtime that sets an
environment variable MSYS_TTY_HANDLES to a list of Win32 handles that
correspond to stdin/stdout/stderr, respectively, *but skips* handles that
MSys2 does not think are terminals.
This commit handles that input to augment the isatty() function to return
1 also when MSYS_TTY_HANDLES contains the corresponding handle.
The only time when Git needs to know whether a Console is attached or not
is when winansi.c is asked to Do Its Thing, therefore we refrain from
overriding isatty there.
Note: this was an issue with MSys1-based Git for Windows, too, hidden by
the fact that Git for Windows used `cmd.exe` as a terminal -- which is
backed by a real Win32 Console. Had MSys1 used, say, rxvt as its default
terminal, the symptom would have been that "git log" does not spawn a
pager by default but instead outputs the entire history (without color
coding, too). In MSys2, the default terminal is mintty, therefore we
finally could not avoid to address the issue.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
On Windows >= Vista, not having an application manifest with a
requestedExecutionLevel can cause several kinds of confusing behavior.
The first and more obvious behavior is "Installer Detection", where
Windows sometimes decides (by looking at things like the file name and
even sequences of bytes within the executable) that an executable is an
installer and should run elevated (causing the well-known popup dialog
to appear). In Git's context, subcommands such as "git patch-id" or "git
update-index" fall prey to this behavior.
The second and more confusing behavior is "File Virtualization". It
means that when files are written without having write permission, it
does not fail (as expected), but they are instead redirected to
somewhere else. When the files are read, the original contents are
returned, though, not the ones that were just written somewhere else.
Even more confusing, not all write accesses are redirected; Trying to
write to write-protected .exe files, for example, will fail instead of
redirecting.
In addition to being unwanted behavior, File Virtualization causes
dramatic slowdowns in Git (see for instance
http://code.google.com/p/msysgit/issues/detail?id=320).
There are two ways to prevent those two behaviors: Either you embed an
application manifest within all your executables, or you add an external
manifest (a file with the same name followed by .manifest) to all your
executables. Since Git's builtins are hardlinked (or copied), it is
simpler and more robust to embed a manifest.
A recent enough MSVC compiler should already embed a working internal
manifest, but for MinGW you have to do so by hand.
Very lightly tested on Wine, where like on Windows XP it should not make
any difference.
References:
- New UAC Technologies for Windows Vista
http://msdn.microsoft.com/en-us/library/bb756960.aspx
- Create and Embed an Application Manifest (UAC)
http://msdn.microsoft.com/en-us/library/bb756929.aspx
[js: simplified the embedding dramatically by reusing Git for Windows'
existing Windows resource file, removed the optional (and dubious)
processorArchitecture attribute of the manifest's assemblyIdentity
section.]
Signed-off-by: Cesar Eduardo Barros <cesarb@cesarb.net>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When the rename function tries to move a directory it fails if the target
directory exists. It should check if it can delete the (possibly empty)
target directory and then try again to move the directory.
Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: 마누엘 <nalla@users.noreply.github.com>
With MSys2, the sigset_t type is defined in sys/types.h, therefore we
need to #include said file.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This test assumed that there are no two equivalent directory separators.
However, on Windows, the back slash and the forward slash *are*
equivalent. Let's paper over this issue by converting the backward
slashes to forward ones in the test that fails with MSys2 otherwise.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When shell scripts access a $TMP variable containing backslashes, they
will be mistaken for escape characters. Let's not let that happen by
converting them to forward slashes.
This fixes t7800 with MSys2.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
There is a really useful debugging technique developed by Sverre
Rabbelier that inserts "bash &&" somewhere in the test scripts, letting
the developer interact at given points with the current state.
Another debugging technique, used a lot by this here coder, is to run
certain executables via gdb by guarding a "gdb -args" call in
bin-wrappers/git.
Both techniques were disabled by 781f76b1(test-lib: redirect stdin of
tests).
Let's reinstate the ability to run an interactive shell by making the
redirection optional: setting the TEST_NO_REDIRECT environment variable
will skip the redirection.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This test is susceptible to MSys2's posix-to-windows path mangling; Let's
just use POSIX paths throughout and let the tests pass.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
