I found multiple issues while investigating this:
* Render thread shutdown is racy, because it doesn't actually stop the
render thread.
* Lifetime management in `ControlCore` failed to account for the
circular dependency of render thread --> renderer --> render data -->
terminal --> renderer --> render thread. Fixed by reordering the
`ControlCore` members to ensure their correct destruction.
* Ensured that the connection setter calls close on the previous
connection.
(Hopefully) Closes#18598
## Validation Steps Performed
* Can't repro the original failure ❌
* Opening and closing tabs as fast as possible doesn't crash anymore ✅
* Detaching and reattaching a tab producing continuous output ✅
If `VtEngine` gets removed from conhost, we need to be able to run
without any renderer present whatsoever. To make this possible,
I've turned all `Renderer&` into `Renderer*`.
Part of #14000
`TextBuffer::GenHTML` and `TextBuffer::GenRTF` now read directly from
the TextBuffer.
- Since we're reading from the buffer, we can now read _all_ the
attributes saved in the buffer. Formatted copy now copies most (if not
all) font/color attributes in the requested format (RTF/HTML).
- Use `TextBuffer::CopyRequest` to pass all copy-related options into
text generation functions as one unit.
- Helper function `TextBuffer::CopyRequest::FromConfig()` generates a
copy request based on Selection mode and user configuration.
- Both formatted text generation functions now use `std::string` and
`fmt::format_to` to generate the required strings. Previously, we were
using `std::ostringstream` which is not recommended due to its potential
overhead.
- Reading attributes from `ROW`'s attribute RLE simplified the logic as
we don't have to track attribute change between the text.
- On the caller side, we do not have to rebuild the plain text string
from the vector of strings anymore. `TextBuffer::GetPlainText()` returns
the entire text as one `std::string`.
- Removed `TextBuffer::TextAndColors`.
- Removed `TextBuffer::GetText()`. `TextBuffer::GetPlainText()` took its
place.
This PR also fixes two bugs in the formatted copy:
- We were applying line breaks after each selected row, even though the
row could have been a Wrapped row. This caused the wrapped rows to break
when they shouldn't.
- We mishandled Unicode text (\uN) within the RTF copy. Every next
character that uses a surrogate pair or high codepoint was missing in
the copied text when pasted to MSWord. The command `\uc4` should have
been `\uc1`, which is used to tell how many fallback characters are used
for each Unicode codepoint (\u). We always use one `?` character as the
fallback.
Closes#16191
**References and Relevant Issues**
- #16270
**Validation Steps Performed**
- Casual copy-pasting from Terminal or OpenConsole to word editors works
as before.
- Verified HTML copy by copying the generated HTML string and running it
through an HTML viewer.
[Sample](https://codepen.io/tusharvickey/pen/wvNXbVN)
- Verified RTF copy by copy-pasting the generated RTF string into
MSWord.
- SingleLine mode works (<kbd>Shift</kbd>+ copy)
- BlockSelection mode works (<kbd>Alt</kbd> selection)
The ultimate goal of this PR was to use ICU for text search to
* Improve Unicode support
Previously we used `towlower` and only supported BMP glphs.
* Improve search performance (10-100x)
This allows us to search for all results in the entire text buffer
at once without having to do so asynchronously.
Unfortunately, this required some significant changes too:
* ICU's search facilities operate on text positions which we need to be
mapped back to buffer coordinates. This required the introduction of
`CharToColumnMapper` to implement sort of a reverse-`_charOffsets`
mapping. It turns text (character) positions back into coordinates.
* Previously search restarted every time you clicked the search button.
It used the current selection as the starting position for the new
search. But since ICU's `uregex` cannot search backwards we're
required to accumulate all results in a vector first and so we
need to cache that vector in between searches.
* We need to know when the cached vector became invalid and so we have
to track any changes made to `TextBuffer`. The way this commit solves
it is by splitting `GetRowByOffset` into `GetRowByOffset` for
`const ROW` access and `GetMutableRowByOffset` which increments a
mutation counter on each call. The `Search` instance can then compare
its cached mutation count against the previous mutation count.
Finally, this commit makes 2 semi-unrelated changes:
* URL search now also uses ICU, since it's closely related to regular
text search anyways. This significantly improves performance at
large window sizes.
* A few minor issues in `UiaTracing` were fixed. In particular
2 functions which passed strings as `wstring` by copy are now
using `wstring_view` and `TraceLoggingCountedWideString`.
Related to #6319 and #8000
## Validation Steps Performed
* Search upward/downward in conhost ✅
* Search upward/downward in WT ✅
* Searching for any of ß, ẞ, ss or SS matches any of the other ✅
* Searching for any of Σ, σ, or ς matches any of the other ✅
This is a minor cleanup to deduplicate the two ReadConsole methods
and will help with making changes to how `COOKED_READ_DATA` is called.
It additionally changes the initial data payload from a `string_view`
to a `wstring_view` as it is guaranteed to be `wchar_t`.
This matches the current `COOKED_READ_DATA` implementation which
blindly assumes that the initial data consists of `wchar_t`.
Closes#5618
This adds PR adds a couple foundational functions and classes to make
our TextBuffer more performant and allow us to improve our Unicode
correctness in the future, by getting rid of our dependence on
`OutputCellIterator`. In the future we can then replace the simple
UTF-16 code point iterator with a proper grapheme cluster iterator.
While my focus is technically on Unicode correctness, the ~4x VT
throughput increase in OpenConsole is pretty nice too.
This PR adds:
* A new, simpler ROW iterator (unused in this PR)
* Cursor movement functions (`NavigateToPrevious`, `NavigateToNext`)
They're based on functions that align the cursor to the start/end
of the _current_ cell, so such functions can be added as well.
* `ReplaceText` to write a raw string of text with the possibility to
specify a right margin.
* `CopyRangeFrom` will allow us to make reflow much faster, as it's able
to bulk-copy already measured strings without re-measuring them.
Related to #8000
## Validation Steps Performed
* enwik8.txt, zhwik8.txt, emoji-test.txt, all work with proper
wide glyph reflow at the end of a row ✅
* This produces "a 咪" where only "a" has a white background:
```sh
printf '\e7こん\e8\x1b[107ma\x1b[m\n'
```
* This produces "abん":
```sh
stdbuf -o0 printf '\x1b7こん\x1b8a'; printf 'b\n'
```
* This produces "xy" at the end of the line:
```sh
stdbuf -o0 printf '\e[999C\bこ\bx'; printf 'y\n'
```
* This produces red whitespace followed by "こ " in the default
background color at the end of the line, and "ん" on the next line:
```sh
printf '\e[41m\e[K\e[m\e[999C\e[2Dこん\n'
```
This commit makes the following project-wide changes/replacements as a
first step in cleaning up our use of `NTSTATUS`.
* Rewriting any uses of `NTSTATUS_FROM_WIN32` that were vulnerable to multiple evaluation bugs
* This is required because the macro version of `NTSTATUS_FROM_WIN32` evaluates its argument twice
* Removing all our local redefinitions of `NTSTATUS` in favor of that of `winternl.h`
* Because of this, we got to (had to?) remove some old transclusions from `conddkrefs.h`
* `NT_SUCCESS` (ours) -> `SUCCEEDED_NTSTATUS` (wil)
* `VERIFY_IS_TRUE(NT_SUCCESS())` (overly elaborate) -> `VERIFY_NT_SUCCESS` (WEX)
* `VERIFY_SUCCESS_NTSTATUS` (ours) -> `VERIFY_NT_SUCCESS` (WEX)
* `!SUCCEEDED_NTSTATUS` -> `FAILED_NTSTATUS`
* One bad use of S_OK as an NTSTATUS -> `STATUS_SUCCESS`
* Removing `NTSTATUS` from any projects that do not use it
This is a follow-up of #13025 to make the members of `til::point/size/rect`
uniform and consistent without the use of `unions`. The only file that has
any changes is `src/host/getset.cpp` where an if condition was simplified.
## Validation Steps Performed
* Host unit tests ✅
* Host feature tests ✅
* ControlCore feature tests ✅
This commit is a from-scratch rewrite of `ROW` with the primary goal to get
rid of the rather bodgy `UnicodeStorage` class and improve Unicode support.
Previously a 120x9001 terminal buffer would store a vector of 9001 `ROW`s
where each `ROW` stored exactly 120 `wchar_t`. Glyphs exceeding their
allocated space would be stored in the `UnicodeStorage` which was basically
a `hashmap<Coordinate, String>`. Iterating over the text in a `ROW` would
require us to check each glyph and fetch it from the map conditionally.
On newlines we'd have to invalidate all map entries that are now gone,
so for every invalidated `ROW` we'd iterate through all glyphs again and if
a single one was stored in `UnicodeStorage`, we'd then iterate through the
entire hashmap to remove all coordinates that were residing on that `ROW`.
All in all, this wasn't the most robust nor performant code.
The new implementation is simple (from a design perspective):
Store all text in a `ROW` in a regular string. Grow the string if needed.
The association between columns and text works by storing character offsets
in a column-wide array. This algorithm is <100 LOC and removes ~1000.
As an aside this PR does a few more things that go hand in hand:
* Remove most of `ROW` helper classes, which aren't needed anymore.
* Allocate backing memory in a single `VirtualAlloc` call.
* Rewrite `IsCursorDoubleWidth` to use `DbcsAttrAt` directly.
Improves overall performance by 10-20% and makes this implementation
faster than the previous NxM storage, despite the added complexity.
Part of #8000
## Validation Steps Performed
* Existing and new unit and feature tests complete ✅
* Printing Unicode completes without crashing ✅
* Resizing works without crashing ✅
Upgrade check-spelling to [v0.0.20](https://github.com/check-spelling/check-spelling/releases/tag/v0.0.20)
This upgrade includes a refresh of the workflow
key new features:
* the previous comment is collapsed
* duplicate words are flagged (see `alone` and `the`)
* forbidding patterns (see `nonexistent`, `preexisting`, and `greater than`)
Each of these features can be tuned
- comment collapsing is controlled by the `followup` bits in the workflow-- but I can't imagine why one would want to turn it off
- duplicate words can be masked in `patterns.txt` (see `Guid` and `that`)
- forbidding patterns (especially duplicates) is in `.github/actions/spelling/line_forbidden.patterns`
Fwiw, I'm slowly moving towards not using `.txt` in filenames, but it's a long term project and I have a bunch of other goals for the near term.
The refresh of advice is of course flexible -- I'm still evolving my default text. Note that the default now includes some `curl` and I'm still working on how I want to consume the output. I'm getting close to the point where I might be able to provide a tool that could reliably consume the output (including on Windows).
This code has been used internally for a while, but I tested it for this repository here:
https://github.com/check-spelling/terminal/pull/2
Previously this project used a great variety of types to present text buffer
coordinates: `short`, `unsigned short`, `int`, `unsigned int`, `size_t`,
`ptrdiff_t`, `COORD`/`SMALL_RECT` (aka `short`), and more.
This massive commit migrates almost all use of those types over to the
centralized types `til::point`/`size`/`rect`/`inclusive_rect` and their
underlying type `til::CoordType` (aka `int32_t`).
Due to the size of the changeset and statistics I expect it to contain bugs.
The biggest risk I see is that some code potentially, maybe implicitly, expected
arithmetic to be mod 2^16 and that this code now allows it to be mod 2^32.
Any narrowing into `short` later on would then throw exceptions.
## PR Checklist
* [x] Closes#4015
* [x] I work here
* [x] Tests added/passed
## Validation Steps Performed
Casual usage of OpenConsole and Windows Terminal. ✅
The x86 Conhost UTs were failing because--when run in a specific order
(SearchTests, then SelectionInputTests)--PrepareGlobalScreenBuffer would
attempt to enable painting on a renderer that was dead and gone and
pushing up daisies.
## Summary of the Pull Request
The purpose of the `IRenderTarget` interface was to support the concept
of multiple buffers in conhost. When a text buffer needed to trigger a
redraw, the render target implementation would be responsible for
deciding whether to forward that redraw to the renderer, depending on
whether the buffer was active or not.
This PR instead introduces a flag in the `TextBuffer` class to track
whether it is active or not, and it can then simply check the flag to
decide whether it needs to trigger a redraw or not. That way it can work
with the `Renderer` directly, and we can avoid a bunch of virtual calls
for the various redraw methods.
## PR Checklist
* [x] Closes#12551
* [x] CLA signed.
* [ ] Tests added/passed
* [ ] Documentation updated.
* [ ] Schema updated.
* [x] I've discussed this with core contributors already. Issue number
where discussion took place: #12551
## Detailed Description of the Pull Request / Additional comments
Anywhere that had previously been getting an `IRenderTarget` from the
`TextBuffer` class in order to trigger a redraw, will now just call the
`TriggerRedraw` method on the `TextBuffer` class itself, since that will
handle the active check which used to be the responsibility of the
render target. All the redraw methods that were in `IRenderTarget` are
now exposed in `TextBuffer` instead.
For this to work, though, the `Renderer` needed to be available before
the `TextBuffer` could be constructed, which required a change in the
conhost initialization order. In the `ConsoleInitializeConnectInfo`
function, I had to move the `Renderer` initialization up so it could be
constructed before the call to `SetUpConsole` (which initializes the
buffer). Once both are ready, the `Renderer::EnablePainting` method is
called to start the render thread.
The other catch is that the renderer used to setup its initial viewport
in the constructor, but with the new initialization order, the viewport
size would not be known at that point. I've addressed that problem by
moving the viewport initialization into the `EnablePainting` method,
since that will only be called after the buffer is setup.
## Validation Steps Performed
The changes in architecture required a number of tweaks to the unit
tests. Some were using a dummy `IRenderTarget` implementation which now
needed to be replaced with a full `Renderer` (albeit with mostly null
parameters). In the case of the scroll test, a mock `IRenderTarget` was
used to track the scroll delta, which now had to be replaced with a mock
`RenderEngine` instead.
Some tests previously relied on having just a `TextBuffer` but without a
`Renderer`, and now they require both. And tests that were constructing
the `TextBuffer` and `Renderer` themselves had to be updated to use the
new initialization order, i.e. with the renderer constructed first.
Semantically, though, the tests still essentially work the same way as
they did before, and they all still pass.
Related work items: MSFT-32957145
Retrieved from https://microsoft.visualstudio.com os.2020 OS official/rs_wdx_dxp_windev bdb25dc99dcb2f1ee483dffe883d0178ea9d18dc
Moving things out of CharRow into ROW helps us hide it as an implementation detail.
This is part one of many.
### CharRow: Hide ClearCell, use ROW::ClearColumn
### CharRow: Hide GetText, use ROW::GetText
### CharRowBaseTests: remove dead file (never used!)
### CharRow: Move DoubleBytePadded into ROW
### CharRow: Move WrapForced into ROW
### Char/AttrRow: Hide Reset, use ROW::Reset
### Remove RowCellIterator (dead code)
RCI was unused; it was replaced by TextBufferCellIterator shortly after its creation
### Move AttrRowTests to ut_textbuffer from ut_host
It had no reliance on the host.
This PR adds support for per-profile tab colors, in accordance with
#7134. This adds a single `tabColor` property, that when set, specifies
the background color for profile's tab. This color can be overridden by
the color picker, and clearing the color with the color picker will
revert to this default color set for the tab.
* Full theming is covered in #3327 & #5772
Validation: Played with setting this color, both on launch and via
hot-reload
Specified in #7134Closes#1337
## Summary of the Pull Request
Move `ICoreSettings` and `IControlSettings` from the TerminalSettings project to the TerminalCore and TerminalControl projects respectively. Also entirely removes the TerminalSettings project.
The purpose of these interfaces is unchanged. `ICoreSettings` is used to instantiate a terminal. `IControlSettings` (which requires an `ICoreSettings`) is used to instantiate a UWP terminal control.
## References
Closes#7140
Related Epic: #885
Related Spec: #6904
## PR Checklist
* [X] Closes#7140
* [X] CLA signed
* [X] Tests ~added~/passed (no additional tests necessary)
* [X] ~Documentation updated~
* [X] ~Schema updated~
## Detailed Description of the Pull Request / Additional comments
A lot of the work here was having to deal with winmd files across all of these projects. The TerminalCore project now outputs a Microsoft.Terminal.TerminalControl.winmd. Some magic happens in TerminalControl.vcxproj to get this to work properly.
## Validation Steps Performed
Deployed Windows Terminal and opened a few new tabs.
Essentially what this does is map the default legacy foreground and
background attributes (typically white on black) to the `IsDefault`
color type in the `TextColor` class. As a result, we can now initialize
the buffer for "legacy" shells (like PowerShell and cmd.exe) with
default colors, instead of white on black. This fixes the startup
rendering in conpty clients, which expect an initial default background
color. It also makes these colors update appropriately when the default
palette values change.
One complication in getting this to work, is that the console permits
users to change which color indices are designated as defaults, so we
can't assume they'll always be white on black. This means that the
legacy-to-`TextAttribute` conversion will need access to those default
values.
Unfortunately the defaults are stored in the conhost `Settings` class
(the `_wFillAttribute` field), which isn't easily accessible to all the
code that needs to construct a `TextAttribute` from a legacy value. The
`OutputCellIterator` is particularly problematic, because some iterator
types need to generate a new `TextAttribute` on every iteration.
So after trying a couple of different approaches, I decided that the
least worst option would be to add a pair of static properties for the
legacy defaults in the `TextAttribute` class itself, then refresh those
values from the `Settings` class whenever the defaults changed (this
only happens on startup, or when the conhost _Properties_ dialog is
edited).
And once the `TextAttribute` class had access to those defaults, it was
fairly easy to adapt the constructor to handle the conversion of default
values to the `IsDefault` color type. I could also then simplify the
`TextAttribute::GetLegacyAttributes` method which does the reverse
mapping, and which previously required the default values to be passed
in as a parameter
VALIDATION
I had to make one small change to the `TestRoundtripExhaustive` unit
test which assumed that all legacy attributes would convert to legacy
color types, which is no longer the case, but otherwise all the existing
tests passed as is. I added a new unit test verifying that the default
legacy attributes correctly mapped to default color types, and the
default color types were mapped back to the correct legacy attributes.
I've manually confirmed that this fixed the issue raised in #5952,
namely that the conhost screen is cleared with the correct default
colors, and also that it is correctly refreshed when changing the
palette from the properties dialog. And I've combined this PR with
#6506, and confirmed that the PowerShell and the cmd shell renderings in
Windows Terminal are at least improved, if not always perfect.
This is a prerequisite for PR #6506Closes#5952
Hide any commandline (cooked read) we have before we begin a resize, and
show it again after the resize.
## References
* I found #5618 while I was working on this.
## PR Checklist
* [x] Closes#1856
* [x] I work here
* [x] Tests added/passed
* [n/a] Requires documentation to be updated
## Detailed Description of the Pull Request / Additional comments
Basically, during a resize, we try to restore the viewport position
correctly, and part of that checks where the current commandline ends.
However, when we do that, the commandline's _current_ state still
reflects the _old_ buffer size, so resizing to be smaller can cause us
to throw an exception, when we find that the commandline doesn't fit in
the new viewport cleanly.
By hiding it, then redrawing it, we avoid this problem entirely. We
don't need to perform the check on the old commandline contents (since
they'll be empty), and we'll redraw it just fine for the new buffer size
## Validation Steps Performed
* ran tests
* checked resizing, snapping in conhost with a cooked read
* checked resizing, snapping in the Terminal with a cooked read
Generated by https://github.com/jsoref/spelling `f`; to maintain your repo, please consider `fchurn`
I generally try to ignore upstream bits. I've accidentally included some items from the `deps/` directory. I expect someone will give me a list of items to drop, I'm happy to drop whole files/directories, or to split the PR into multiple items (E.g. comments/locals/public).
Closes#4294
## Summary of the Pull Request
#4354 is a pretty complicated PR. It's got a bunch of conpty changes, but what it also has was some critical improvements to the roundtrip test suite. I'm working on some other bugfixes in the same area currently, and need these tests enhancements in those branches _now_. The rest of #4354 is complex enough that I don't trust it will get merged soon (if ever). However, these fixes _should_ be in regardless.
## PR Checklist
* [x] Taken directly from #4354
* [x] I work here
* [x] Tests added/passed
* [n/a] Requires documentation to be updated
## Detailed Description of the Pull Request / Additional comments
This is four main changes:
* Enable conpty to be fully enabled in unittests. Just setting up a VT renderer isn't enough to trick the host into being in conpty mode - it also needs to have some other flags set.
* Some minor changes to `CommonState` to better configure the common test state for conpty
* Move some of the verify helpers from `ConptyRoundtripTests` into their own helper class, to be shared in multiple tests
* Add a `TerminalBufferTests` class, for testing the Terminal buffer directly (without conpty).
This change is really easier than
would suggest, I promise.
## Summary of the Pull Request
This PR adds two tests:
* First, I started by writing a test where I could write output to the console host and inspect what output came out of conpty. This is the `ConptyOutputTests` in the host unit tests.
* Then I got crazy and thought _"what if I could take that output and dump it straight into the `Terminal`"_? Hence, the `ConptyRoundtripTests` were born, into the TerminalCore unit tests.
## References
Done in pursuit of #4200, but I felt this warranted it's own atomic PR
## PR Checklist
* [x] Doesn't close anything on it's own.
* [x] I work here
* [x] you better believe this adds tests
* [n/a] Requires documentation to be updated
## Detailed Description of the Pull Request / Additional comments
From the comment in `ConptyRoundtripTests`:
> This test class creates an in-proc conpty host as well as a Terminal, to
> validate that strings written to the conpty create the same resopnse on the
> terminal end. Tests can be written that validate both the contents of the
> host buffer as well as the terminal buffer. Everytime that
> `renderer.PaintFrame()` is called, the tests will validate the expected
> output, and then flush the output of the VtEngine straight to th
Also, some other bits had to be updated:
* The renderer needed to be able to survive without a thread, so I hadded a simple check that it actually had a thread before calling `pThread->NotifyPaint`
* Bits in `CommonState` used `NTSTATUS_FROM_HRESULT` which did _not_ work outside the host project. Since the `NTSTATUS` didn't seem that important, I replaced that with a `HRESULT`
* `CommonState` likes to initialize the console to some _weird_ defaults. I added an optional param to let us just use the defaults.
* Removed using namespace directive from header files and put these in cpp files where they are used
* Fixed tabbing issues by replacing them with spaces.
Also regrouped the using directives.
* Update src/host/exemain.cpp
Co-Authored-By: Mike Griese <migrie@microsoft.com>
* Update src/interactivity/win32/find.cpp
Co-Authored-By: Mike Griese <migrie@microsoft.com>
* Apply [[nodiscard]] to functions returning error codes
- applied [[nodiscard]] for all HRESULT, LRESULT, and NTSTATUS functions
- fixed IntelliSense declaration complaints leading to function not
implemented warnings
- deleted declared but never implemented functions
- fixed unused parameter warnings
How verified:
- bcz dbg
- opencon
- testcon
- VS2019 debug build
* - use LOG_IF_FAILED where applicable
- remove use of goto
- make MakeAltRasterFont return void
* - add missing [[nodiscard]]
- remove vestigal function declarations
- fix inconsistent function declaration
