microsoft/terminal
1
0
Fork 0
mirror of https://github.com/microsoft/terminal.git synced 2025-12-10 17:53:06 -06:00
Code Issues Packages Projects Releases 138 Wiki Activity

Remove CHAR_INFO munging for raster fonts (#17681)

Browse Source 
`RealUnicodeToFalseUnicode` was described as:
> This routine converts a unicode string into the correct characters
> for an OEM (cp 437) font. This code is needed because the gdi glyph
> mapper converts unicode to ansi using codepage 1252 to index font.
> This is how the data is stored internally.

In other words, it takes a UCS2 string, translates it to the current
codepage and translates it back to UCS2 in the US version of Windows.
In the "eastern" DBCS version it "reinterprets" the DBCS string as
`CP_USA` (a particularly weird quirk).

The original implementation used to do this translation at every
opportunity where text went into or out of conhost.
The translation was weird, but it was consistent.
In Windows 10 RS1 conhost got a new UCS2-aware text buffer and
this translation was removed from most places, as the text buffer
was converted to store proper UCS2. This broke the entire concept
of the translation though. Whatever data you previously wrote with
something like `WriteConsoleOutputCharacter` now came back with
something entirely else via `ReadConsoleOutput`.

In other words, I believe past RS1 there was technically never any
point in "munging" `CHAR_INFO`s, as this only covered 2 API functions.

Still, this does mean that this PR represents an API breaking change.
It's a minor one though, because it only affects 2 API functions.
And more importantly, it's a necessary breaking change as we move
further and further away from correlating codepoint and column counts.

## Validation Steps Performed
* Remaining tests pass ✅
This commit is contained in:
Leonard Hecker 2024-08-14 01:18:16 +02:00 committed by GitHub
parent 9074e9d6a8
commit 0fd8dc575f
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
9 changed files with 30 additions and 659 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
src/buffer/out/OutputCellIterator.cpp
View File

@ -274,7 +274,7 @@ OutputCellIterator& OutputCellIterator::operator++()
}
case Mode::CharInfo:
{
// Walk forward by one because charinfos are just the legacy version of cells and prealigned to columns
// Walk forward by one because char infos are just the legacy version of cells and prealigned to columns
_pos++;
if (operator bool())
{

51
src/host/dbcs.cpp
View File

@ -6,7 +6,6 @@
#include "misc.h"
#include "../types/inc/convert.hpp"
#include "../types/inc/GlyphWidth.hpp"
#include "../interactivity/inc/ServiceLocator.hpp"
@ -47,56 +46,6 @@ bool CheckBisectStringA(_In_reads_bytes_(cbBuf) PCHAR pchBuf, _In_ DWORD cbBuf,
return false;
}
// Routine Description:
// - This routine removes the double copies of characters used when storing DBCS/Double-wide characters in the text buffer.
// - It munges up Unicode cells that are about to be returned whenever there is DBCS data and a raster font is enabled.
// - This function is ONLY FOR COMPATIBILITY PURPOSES. Please do not introduce new usages.
// Arguments:
// - buffer - The buffer to walk and fix
// Return Value:
// - The length of the final modified buffer.
DWORD UnicodeRasterFontCellMungeOnRead(const std::span<CHAR_INFO> buffer)
{
// Walk through the source CHAR_INFO and copy each to the destination.
// EXCEPT for trailing bytes (this will de-duplicate the leading/trailing byte double copies of the CHAR_INFOs as stored in the buffer).
// Set up indices used for arrays.
DWORD iDst = 0;
// Walk through every CHAR_INFO
for (DWORD iSrc = 0; iSrc < buffer.size(); iSrc++)
{
// If it's not a trailing byte, copy it straight over, stripping out the Leading/Trailing flags from the attributes field.
auto& src{ til::at(buffer, iSrc) };
if (!WI_IsFlagSet(src.Attributes, COMMON_LVB_TRAILING_BYTE))
{
auto& dst{ til::at(buffer, iDst) };
dst = src;
WI_ClearAllFlags(dst.Attributes, COMMON_LVB_SBCSDBCS);
iDst++;
}
// If it was a trailing byte, we'll just walk past it and keep going.
}
// Zero out the remaining part of the destination buffer that we didn't use.
const auto cchDstToClear = gsl::narrow<DWORD>(buffer.size()) - iDst;
if (cchDstToClear > 0)
{
const auto pciDstClearStart = buffer.data() + iDst;
ZeroMemory(pciDstClearStart, cchDstToClear * sizeof(CHAR_INFO));
}
// Add the additional length we just modified.
iDst += cchDstToClear;
// now that we're done, we should have copied, left alone, or cleared the entire length.
FAIL_FAST_IF(iDst != buffer.size());
return iDst;
}
// Routine Description:
// - Checks if a char is a lead byte for a given code page.
// Arguments:

2
src/host/dbcs.h
View File

@ -22,8 +22,6 @@ Revision History:
bool CheckBisectStringA(_In_reads_bytes_(cbBuf) PCHAR pchBuf, _In_ DWORD cbBuf, const CPINFO* const pCPInfo);
DWORD UnicodeRasterFontCellMungeOnRead(const std::span<CHAR_INFO> buffer);
bool IsDBCSLeadByteConsole(const CHAR ch, const CPINFO* const pCPInfo);
BYTE CodePageToCharSet(const UINT uiCodePage);

73
src/host/directio.cpp
View File

@ -413,56 +413,6 @@ CATCH_RETURN();
CATCH_RETURN();
}
[[nodiscard]] static std::vector<CHAR_INFO> _ConvertCellsToMungedW(std::span<CHAR_INFO> buffer, const Viewport& rectangle)
{
std::vector<CHAR_INFO> result;
result.reserve(buffer.size());
const auto size = rectangle.Dimensions();
auto bufferIter = buffer.begin();
for (til::CoordType i = 0; i < size.height; i++)
{
for (til::CoordType j = 0; j < size.width; j++)
{
// Prepare a candidate charinfo on the output side copying the colors but not the lead/trail information.
auto candidate = *bufferIter;
WI_ClearAllFlags(candidate.Attributes, COMMON_LVB_SBCSDBCS);
// If the glyph we're given is full width, it needs to take two cells.
if (IsGlyphFullWidth(candidate.Char.UnicodeChar))
{
// If we're not on the final cell of the row...
if (j < size.width - 1)
{
// Mark that we're consuming two cells.
j++;
// Fill one cell with a copy of the color and character marked leading
WI_SetFlag(candidate.Attributes, COMMON_LVB_LEADING_BYTE);
result.push_back(candidate);
// Fill a second cell with a copy of the color marked trailing and a padding character.
WI_ClearFlag(candidate.Attributes, COMMON_LVB_LEADING_BYTE);
WI_SetFlag(candidate.Attributes, COMMON_LVB_TRAILING_BYTE);
}
else
{
// If we're on the final cell, this won't fit. Replace with a space.
candidate.Char.UnicodeChar = UNICODE_SPACE;
}
}
// Push our candidate in.
result.push_back(candidate);
// Advance to read the next item.
++bufferIter;
}
}
return result;
}
[[nodiscard]] HRESULT ReadConsoleOutputWImplHelper(const SCREEN_INFORMATION& context,
std::span<CHAR_INFO> targetBuffer,
const Viewport& requestRectangle,
@ -547,16 +497,7 @@ CATCH_RETURN();
try
{
RETURN_IF_FAILED(ReadConsoleOutputWImplHelper(context, buffer, sourceRectangle, readRectangle));
if (!context.GetActiveBuffer().GetCurrentFont().IsTrueTypeFont())
{
// For compatibility reasons, we must maintain the behavior that munges the data if we are writing while a raster font is enabled.
// This can be removed when raster font support is removed.
UnicodeRasterFontCellMungeOnRead(buffer);
}
return S_OK;
return ReadConsoleOutputWImplHelper(context, buffer, sourceRectangle, readRectangle);
}
CATCH_RETURN();
}
@ -684,17 +625,7 @@ CATCH_RETURN();
writer.BackupCursor();
}
if (!context.GetActiveBuffer().GetCurrentFont().IsTrueTypeFont())
{
// For compatibility reasons, we must maintain the behavior that munges the data if we are writing while a raster font is enabled.
// This can be removed when raster font support is removed.
auto translated = _ConvertCellsToMungedW(buffer, requestRectangle);
RETURN_IF_FAILED(WriteConsoleOutputWImplHelper(context, translated, requestRectangle.Width(), requestRectangle, writtenRectangle));
}
else
{
RETURN_IF_FAILED(WriteConsoleOutputWImplHelper(context, buffer, requestRectangle.Width(), requestRectangle, writtenRectangle));
}
RETURN_IF_FAILED(WriteConsoleOutputWImplHelper(context, buffer, requestRectangle.Width(), requestRectangle, writtenRectangle));
if (writer)
{

466
src/host/ft_host/CJK_DbcsTests.cpp
View File

@ -62,14 +62,12 @@ namespace DbcsWriteRead
};
void TestRunner(_In_opt_ WORD* const pwAttrOverride,
const bool fUseTrueType,
const DbcsWriteRead::WriteMode WriteMode,
const UnicodeMode fWriteInUnicode,
const DbcsWriteRead::ReadMode ReadMode,
const bool fReadWithUnicode);
bool Setup(_In_ bool fIsTrueType,
_Out_ HANDLE* const phOut,
bool Setup(_Out_ HANDLE* const phOut,
_Out_ WORD* const pwAttributes);
void SendOutput(const HANDLE hOut,
@ -88,7 +86,6 @@ namespace DbcsWriteRead
const WORD wAttrWritten,
const DbcsWriteRead::WriteMode WriteMode,
const DbcsWriteRead::UnicodeMode fWriteWithUnicode,
const bool fIsTrueTypeFont,
const DbcsWriteRead::ReadMode ReadMode,
const bool fReadWithUnicode,
CharInfoPattern& expected);
@ -96,13 +93,11 @@ namespace DbcsWriteRead
const CharInfoPattern& PrepReadConsoleOutput(
const DbcsWriteRead::WriteMode WriteMode,
const UnicodeMode fWriteWithUnicode,
const bool fIsTrueTypeFont,
const bool fReadWithUnicode);
const CharInfoPattern& PrepReadConsoleOutputCharacter(
const DbcsWriteRead::WriteMode WriteMode,
const UnicodeMode fWriteWithUnicode,
const bool fIsTrueTypeFont,
const bool fReadWithUnicode);
};
@ -120,7 +115,6 @@ class DbcsTests
TEST_METHOD(TestMultibyteInputCoalescing);
BEGIN_TEST_METHOD(TestDbcsWriteRead)
TEST_METHOD_PROPERTY(L"Data:fUseTrueTypeFont", L"{true, false}")
TEST_METHOD_PROPERTY(L"Data:WriteMode", L"{0, 1, 2, 3}")
TEST_METHOD_PROPERTY(L"Data:fWriteInUnicode", L"{0, 1, 2}")
TEST_METHOD_PROPERTY(L"Data:ReadMode", L"{0, 1}")
@ -173,8 +167,7 @@ bool DbcsTests::DbcsTestSetup()
return true;
}
bool DbcsWriteRead::Setup(_In_ bool fIsTrueType,
_Out_ HANDLE* const phOut,
bool DbcsWriteRead::Setup(_Out_ HANDLE* const phOut,
_Out_ WORD* const pwAttributes)
{
const auto hOut = GetStdOutputHandle();
@ -186,20 +179,8 @@ bool DbcsWriteRead::Setup(_In_ bool fIsTrueType,
// Now set up the font. Many of these APIs are oddly dependent on font, so set as appropriate.
CONSOLE_FONT_INFOEX cfiex = { 0 };
cfiex.cbSize = sizeof(cfiex);
if (!fIsTrueType)
{
// We use Terminal as the raster font name always.
wcscpy_s(cfiex.FaceName, L"Terminal");
// Use default raster font size from Japanese system.
cfiex.dwFontSize.X = 8;
cfiex.dwFontSize.Y = 18;
}
else
{
wcscpy_s(cfiex.FaceName, L"MS Gothic");
cfiex.dwFontSize.Y = 16;
}
wcscpy_s(cfiex.FaceName, L"MS Gothic");
cfiex.dwFontSize.Y = 16;
VERIFY_WIN32_BOOL_SUCCEEDED_RETURN(OneCoreDelay::SetCurrentConsoleFontEx(hOut, FALSE, &cfiex));
@ -441,49 +422,6 @@ namespace PrepPattern
}
}
// Receive Output Table:
// attr | wchar (char) | symbol
// ------------------------------------
// 0x029 | 0x0051 (0x51) | Q
// 0x029 | 0x3044 (0x44) | Hiragana I
// 0x029 | 0x304B (0x4B) | Hiragana KA
// 0x029 | 0x306A (0x6A) | Hiragana NA
// 0x029 | 0x005A (0x5A) | Z
// 0x029 | 0x0059 (0x59) | Y
// 0x029 | 0x0058 (0x58) | X
// 0x029 | 0x0057 (0x57) | W
// 0x029 | 0x0056 (0x56) | V
// 0x029 | 0x0055 (0x55) | U
// 0x029 | 0x0054 (0x54) | T
// 0x029 | 0x306B (0x6B) | Hiragana NI
// 0x000 | 0x0000 (0x00) | <null>
// 0x000 | 0x0000 (0x00) | <null>
// 0x000 | 0x0000 (0x00) | <null>
// 0x000 | 0x0000 (0x00) | <null>
// ...
// "Null Padded" means any unused data in the buffer will be filled with null and null attribute.
// "Dedupe" means that any full-width characters in the buffer (despite being stored doubled inside the buffer)
// will be returned as single copies.
// "W" means that we intend Unicode data to be browsed in the resulting struct (even though wchar and char are unioned.)
static constexpr CharInfoPattern NullPaddedDedupeW{
makeCharInfo(0x0051, colored),
makeCharInfo(0x3044, colored),
makeCharInfo(0x304b, colored),
makeCharInfo(0x306a, colored),
makeCharInfo(0x005a, colored),
makeCharInfo(0x0059, colored),
makeCharInfo(0x0058, colored),
makeCharInfo(0x0057, colored),
makeCharInfo(0x0056, colored),
makeCharInfo(0x0055, colored),
makeCharInfo(0x0054, colored),
makeCharInfo(0x306b, colored),
makeCharInfo(0x0000, zeroed),
makeCharInfo(0x0000, zeroed),
makeCharInfo(0x0000, zeroed),
makeCharInfo(0x0000, zeroed),
};
// Receive Output Table:
// attr | wchar (char) | symbol
// ------------------------------------
@ -570,135 +508,6 @@ namespace PrepPattern
makeCharInfo(0x0020, white),
};
// Receive Output Table:
// attr | wchar (char) | symbol
// ------------------------------------
// 0x029 | 0x0051 (0x51) | Q
// 0x029 | 0x3044 (0x44) | Hiragana I
// 0x029 | 0x304B (0x4B) | Hiragana KA
// 0x029 | 0x306A (0x6A) | Hiragana NA
// 0x029 | 0x005A (0x5A) | Z
// 0x029 | 0x0059 (0x59) | Y
// 0x029 | 0x0058 (0x58) | X
// 0x029 | 0x0057 (0x57) | W
// 0x029 | 0x0056 (0x56) | V
// 0x007 | 0x0020 (0x20) | <space>
// 0x007 | 0x0020 (0x20) | <space>
// 0x007 | 0x0020 (0x20) | <space>
// 0x007 | 0x0020 (0x20) | <space>
// 0x000 | 0x0000 (0x00) | <null>
// 0x000 | 0x0000 (0x00) | <null>
// 0x000 | 0x0000 (0x00) | <null>
// ...
// "Space Padded" means most of the unused data in the buffer will be filled with spaces and the default attribute.
// "Dedupe" means that any full-width characters in the buffer (despite being stored doubled inside the buffer)
// will be returned as single copies.
// "W" means that we intend Unicode data to be browsed in the resulting struct (even though wchar and char are unioned.)
// "Truncated" means that this pattern trims off some of the end of the buffer with NULLs.
static constexpr CharInfoPattern SpacePaddedDedupeTruncatedW{
makeCharInfo(0x0051, colored),
makeCharInfo(0x3044, colored),
makeCharInfo(0x304b, colored),
makeCharInfo(0x306a, colored),
makeCharInfo(0x005a, colored),
makeCharInfo(0x0059, colored),
makeCharInfo(0x0058, colored),
makeCharInfo(0x0057, colored),
makeCharInfo(0x0056, colored),
makeCharInfo(0x0020, white),
makeCharInfo(0x0020, white),
makeCharInfo(0x0020, white),
makeCharInfo(0x0020, white),
makeCharInfo(0x0000, zeroed),
makeCharInfo(0x0000, zeroed),
makeCharInfo(0x0000, zeroed),
};
// Receive Output Table:
// attr | wchar (char) | symbol
// ------------------------------------
// 0x029 | 0x0051 (0x51) | Q
// 0x029 | 0x3044 (0x44) | Hiragana I
// 0x029 | 0x3044 (0x44) | Hiragana I
// 0x029 | 0x304B (0x4B) | Hiragana KA
// 0x029 | 0x304B (0x4B) | Hiragana KA
// 0x029 | 0x306A (0x6A) | Hiragana NA
// 0x029 | 0x306A (0x6A) | Hiragana NA
// 0x029 | 0x005A (0x5A) | Z
// 0x029 | 0x0059 (0x59) | Y
// 0x029 | 0x0058 (0x58) | X
// 0x000 | 0x0000 (0x00) | <null>
// 0x000 | 0x0000 (0x00) | <null>
// 0x000 | 0x0000 (0x00) | <null>
// 0x000 | 0x0000 (0x00) | <null>
// 0x000 | 0x0000 (0x00) | <null>
// 0x000 | 0x0000 (0x00) | <null>
// ...
// "Doubled" means that any full-width characters in the buffer are returned twice.
// "Truncated" means that this pattern trims off some of the end of the buffer with NULLs.
// "W" means that we intend Unicode data to be browsed in the resulting struct (even though wchar and char are unioned.)
static constexpr CharInfoPattern DoubledTruncatedW{
makeCharInfo(0x0051, colored),
makeCharInfo(0x3044, colored),
makeCharInfo(0x3044, colored),
makeCharInfo(0x304b, colored),
makeCharInfo(0x304b, colored),
makeCharInfo(0x306a, colored),
makeCharInfo(0x306a, colored),
makeCharInfo(0x005a, colored),
makeCharInfo(0x0059, colored),
makeCharInfo(0x0058, colored),
makeCharInfo(0x0000, zeroed),
makeCharInfo(0x0000, zeroed),
makeCharInfo(0x0000, zeroed),
makeCharInfo(0x0000, zeroed),
makeCharInfo(0x0000, zeroed),
makeCharInfo(0x0000, zeroed),
};
// Receive Output Table:
// attr | wchar (char) | symbol
// ------------------------------------
// 0x029 | 0x0051 (0x51) | Q
// 0x129 | 0x0082 (0x82) | Hiragana I Shift-JIS Codepage 932 Lead Byte
// 0x229 | 0x00A2 (0xA2) | Hiragana I Shift-JIS Codepage 932 Trail Byte
// 0x129 | 0x0082 (0x82) | Hiragana KA Shift-JIS Codepage 932 Lead Byte
// 0x229 | 0x00A9 (0xA9) | Hiragana KA Shift-JIS Codepage 932 Trail Byte
// 0x129 | 0x0082 (0x82) | Hiragana NA Shift-JIS Codepage 932 Lead Byte
// 0x229 | 0x00C8 (0xC8) | Hiragana NA Shift-JIS Codepage 932 Trail Byte
// 0x029 | 0x005A (0x5A) | Z
// 0x029 | 0x0059 (0x59) | Y
// 0x029 | 0x0058 (0x58) | X
// 0x029 | 0x0057 (0x57) | W
// 0x029 | 0x0056 (0x56) | V
// 0x007 | 0x0020 (0x20) | <space>
// 0x007 | 0x0020 (0x20) | <space>
// 0x007 | 0x0020 (0x20) | <space>
// 0x007 | 0x0020 (0x20) | <space>
// ...
// "Space Padded" means most of the unused data in the buffer will be filled with spaces and the default attribute.
// "Dedupe" means that any full-width characters in the buffer (despite being stored doubled inside the buffer)
// will be returned as single copies.
// "A" means that we intend in-codepage (char) data to be browsed in the resulting struct (even though wchar and char are unioned.)
static constexpr CharInfoPattern SpacePaddedDedupeA{
makeCharInfo(0x0051, colored),
makeCharInfo(0x0082, colored | leading),
makeCharInfo(0x00a2, colored | trailing),
makeCharInfo(0x0082, colored | leading),
makeCharInfo(0x00a9, colored | trailing),
makeCharInfo(0x0082, colored | leading),
makeCharInfo(0x00c8, colored | trailing),
makeCharInfo(0x005a, colored),
makeCharInfo(0x0059, colored),
makeCharInfo(0x0058, colored),
makeCharInfo(0x0057, colored),
makeCharInfo(0x0056, colored),
makeCharInfo(0x0020, white),
makeCharInfo(0x0020, white),
makeCharInfo(0x0020, white),
makeCharInfo(0x0020, white),
};
// Receive Output Table:
// attr | wchar (char) | symbol
// ------------------------------------
@ -781,47 +590,6 @@ namespace PrepPattern
makeCharInfo(0x00c9, colored | trailing),
};
// Receive Output Table:
// attr | wchar (char) | symbol
// ------------------------------------
// 0x029 | 0x0051 (0x51) | Q
// 0x129 | 0x0082 (0x82) | Hiragana I Shift-JIS Codepage 932 Lead Byte
// 0x229 | 0x00A2 (0xA2) | Hiragana I Shift-JIS Codepage 932 Trail Byte
// 0x129 | 0x0082 (0x82) | Hiragana I Shift-JIS Codepage 932 Lead Byte
// 0x229 | 0x00A2 (0xA2) | Hiragana I Shift-JIS Codepage 932 Trail Byte
// 0x129 | 0x0082 (0x82) | Hiragana KA Shift-JIS Codepage 932 Lead Byte
// 0x229 | 0x00A9 (0xA9) | Hiragana KA Shift-JIS Codepage 932 Trail Byte
// 0x129 | 0x0082 (0x82) | Hiragana KA Shift-JIS Codepage 932 Lead Byte
// 0x229 | 0x00A9 (0xA9) | Hiragana KA Shift-JIS Codepage 932 Trail Byte
// 0x129 | 0x0082 (0x82) | Hiragana NA Shift-JIS Codepage 932 Lead Byte
// 0x229 | 0x00C8 (0xC8) | Hiragana NA Shift-JIS Codepage 932 Trail Byte
// 0x129 | 0x0082 (0x82) | Hiragana NA Shift-JIS Codepage 932 Lead Byte
// 0x229 | 0x00C8 (0xC8) | Hiragana NA Shift-JIS Codepage 932 Trail Byte
// 0x029 | 0x005A (0x5A) | Z
// 0x029 | 0x0059 (0x59) | Y
// 0x029 | 0x0058 (0x58) | X
// ...
// "Doubled" means that any full-width characters in the buffer are returned twice.
// "A" means that we intend in-codepage (char) data to be browsed in the resulting struct.
static constexpr CharInfoPattern DoubledA{
makeCharInfo(0x0051, colored),
makeCharInfo(0x0082, colored | leading),
makeCharInfo(0x00a2, colored | trailing),
makeCharInfo(0x0082, colored | leading),
makeCharInfo(0x00a2, colored | trailing),
makeCharInfo(0x0082, colored | leading),
makeCharInfo(0x00a9, colored | trailing),
makeCharInfo(0x0082, colored | leading),
makeCharInfo(0x00a9, colored | trailing),
makeCharInfo(0x0082, colored | leading),
makeCharInfo(0x00c8, colored | trailing),
makeCharInfo(0x0082, colored | leading),
makeCharInfo(0x00c8, colored | trailing),
makeCharInfo(0x005a, colored),
makeCharInfo(0x0059, colored),
makeCharInfo(0x0058, colored),
};
// Receive Output Table:
// attr | wchar (char) | symbol
// ------------------------------------
@ -864,94 +632,6 @@ namespace PrepPattern
makeCharInfo(0x0000, colored | trailing),
};
// Receive Output Table:
// attr | wchar (char) | symbol
// ------------------------------------
// 0x029 | 0x0051 (0x51) | Q
// 0x129 | 0x3044 (0x44) | Hiragana I
// 0x229 | 0x3044 (0x44) | Hiragana I
// 0x129 | 0x304B (0x4B) | Hiragana KA
// 0x229 | 0x304B (0x4B) | Hiragana KA
// 0x129 | 0x306A (0x6A) | Hiragana NA
// 0x229 | 0x306A (0x6A) | Hiragana NA
// 0x129 | 0x005A (0x5A) | Z
// 0x229 | 0x0059 (0x59) | Y
// 0x129 | 0x0058 (0x58) | X
// 0x229 | 0x0000 (0x00) | <null>
// 0x129 | 0x0000 (0x00) | <null>
// 0x229 | 0x0000 (0x00) | <null>
// 0x029 | 0x0000 (0x00) | <null>
// 0x029 | 0x0000 (0x00) | <null>
// 0x029 | 0x0000 (0x00) | <null>
// ...
// "Doubled" means that any full-width characters in the buffer are returned twice.
// "Truncated" means that this pattern trims off some of the end of the buffer with NULLs.
// "W" means that we intend Unicode data to be browsed in the resulting struct (even though wchar and char are unioned.)
static constexpr CharInfoPattern DoubledTruncatedCoverAChar{
makeCharInfo(0x0051, colored),
makeCharInfo(0x3044, colored | leading),
makeCharInfo(0x3044, colored | trailing),
makeCharInfo(0x304b, colored | leading),
makeCharInfo(0x304b, colored | trailing),
makeCharInfo(0x306a, colored | leading),
makeCharInfo(0x306a, colored | trailing),
makeCharInfo(0x005a, colored | leading),
makeCharInfo(0x0059, colored | trailing),
makeCharInfo(0x0058, colored | leading),
makeCharInfo(0x0000, colored | trailing),
makeCharInfo(0x0000, colored | leading),
makeCharInfo(0x0000, colored | trailing),
makeCharInfo(0x0000, colored),
makeCharInfo(0x0000, colored),
makeCharInfo(0x0000, colored),
};
// Receive Output Table:
// attr | wchar (char) | symbol
// ------------------------------------
// 0x029 | 0x0051 (0x51) | Q
// 0x129 | 0x3044 (0x44) | Hiragana I
// 0x229 | 0x304B (0x4B) | Hiragana KA
// 0x129 | 0x306A (0x6A) | Hiragana NA
// 0x229 | 0x005A (0x5A) | Z
// 0x129 | 0x0059 (0x59) | Y
// 0x229 | 0x0058 (0x58) | X
// 0x029 | 0x0057 (0x57) | W
// 0x029 | 0x0056 (0x56) | V
// 0x029 | 0x0020 (0x20) | <space>
// 0x029 | 0x0020 (0x20) | <space>
// 0x029 | 0x0020 (0x20) | <space>
// 0x007 | 0x0020 (0x20) | <space>
// 0x007 | 0x0000 (0x00) | <null>
// 0x007 | 0x0000 (0x00) | <null>
// 0x007 | 0x0000 (0x00) | <null>
// ...
// "Space Padded" means most of the unused data in the buffer will be filled with spaces and the default attribute.
// "Dedupe" means that any full-width characters in the buffer (despite being stored doubled inside the buffer)
// will be returned as single copies.
// "W" means that we intend Unicode data to be browsed in the resulting struct (even though wchar and char are unioned.)
// "Truncated" means that this pattern trims off some of the end of the buffer with NULLs.
// "A Cover Attr" means that after all the other operations, we will finally run through and cover up the attributes
// again with what they would have been for multi-byte data (leading and trailing flags)
static constexpr CharInfoPattern ACoverAttrSpacePaddedDedupeTruncatedW{
makeCharInfo(0x0051, colored),
makeCharInfo(0x3044, colored | leading),
makeCharInfo(0x304b, colored | trailing),
makeCharInfo(0x306a, colored | leading),
makeCharInfo(0x005a, colored | trailing),
makeCharInfo(0x0059, colored | leading),
makeCharInfo(0x0058, colored | trailing),
makeCharInfo(0x0057, colored),
makeCharInfo(0x0056, colored),
makeCharInfo(0x0020, colored),
makeCharInfo(0x0020, colored),
makeCharInfo(0x0020, colored),
makeCharInfo(0x0020, white),
makeCharInfo(0x0000, white),
makeCharInfo(0x0000, white),
makeCharInfo(0x0000, white),
};
// Receive Output Table:
// attr | wchar (char) | symbol
// ------------------------------------
@ -1002,7 +682,6 @@ namespace PrepPattern
const CharInfoPattern& DbcsWriteRead::PrepReadConsoleOutput(
const DbcsWriteRead::WriteMode WriteMode,
const UnicodeMode fWriteWithUnicode,
const bool fIsTrueTypeFont,
const bool fReadWithUnicode)
{
switch (WriteMode)
@ -1013,80 +692,34 @@ const CharInfoPattern& DbcsWriteRead::PrepReadConsoleOutput(
case UnicodeMode::UnicodeSingle:
if (fReadWithUnicode)
{
if (fIsTrueTypeFont)
{
// When written with WriteConsoleOutputW and read back with ReadConsoleOutputW when the font is TrueType,
// we will get a deduplicated set of Unicode characters with no lead/trailing markings and space padded at the end.
return PrepPattern::SpacePaddedDedupeW;
}
else
{
// When written with WriteConsoleOutputW and read back with ReadConsoleOutputW when the font is Raster,
// we will get a deduplicated set of Unicode characters with no lead/trailing markings and space padded at the end...
// ... except something weird happens with truncation (TODO figure out what)
return PrepPattern::SpacePaddedDedupeTruncatedW;
}
// When written with WriteConsoleOutputW and read back with ReadConsoleOutputW when the font is TrueType,
// we will get a deduplicated set of Unicode characters with no lead/trailing markings and space padded at the end.
return PrepPattern::SpacePaddedDedupeW;
}
else
{
if (fIsTrueTypeFont)
{
// Normally this would be SpacePaddedDedupeA (analogous to the SpacePaddedDedupeW above), but since the narrow
// unicode chars can't be represented as narrow DBCS (since those don't exist) we get SpacePaddedDedupeInvalidA.
return PrepPattern::SpacePaddedDedupeInvalidA;
}
else
{
// When written with WriteConsoleOutputW and read back with ReadConsoleOutputA under Raster font, we will get the
// double-byte sequences stomped on top of a Unicode filled CHAR_INFO structure that used -1 for trailing bytes.
return PrepPattern::SpacePaddedDedupeA;
}
// Since the narrow unicode chars can't be represented as narrow DBCS (since those don't exist) we get SpacePaddedDedupeInvalidA.
return PrepPattern::SpacePaddedDedupeInvalidA;
}
break;
case UnicodeMode::UnicodeDoubled:
if (fReadWithUnicode)
{
if (fIsTrueTypeFont)
{
// In a TrueType font, we will get back Unicode characters doubled up and marked with leading and trailing bytes.
return PrepPattern::DoubledW;
}
else
{
// We get the same as SpacePaddedDedupeTruncatedW above, but due to the unicode chars being doubled, we get DoubledTruncatedW.
return PrepPattern::DoubledTruncatedW;
}
// In a TrueType font, we will get back Unicode characters doubled up and marked with leading and trailing bytes.
return PrepPattern::DoubledW;
}
else
{
if (fIsTrueTypeFont)
{
// In a TrueType font, we will get back Unicode characters doubled up and marked with leading and trailing bytes.
return PrepPattern::A;
}
else
{
// When written with WriteConsoleOutputW and read back with ReadConsoleOutputA under Raster font,
// we will get the double-byte sequences doubled up, because each narrow cell is written as a DBCS separately.
return PrepPattern::DoubledA;
}
// In a TrueType font, we will get back Unicode characters doubled up and marked with leading and trailing bytes.
return PrepPattern::A;
}
break;
default:
if (fReadWithUnicode)
{
if (fIsTrueTypeFont)
{
// When written with WriteConsoleOutputW and read back with ReadConsoleOutputA when the font is TrueType,
// we will get back Unicode characters doubled up and marked with leading and trailing bytes.
return PrepPattern::DoubledW;
}
else
{
// When written with WriteConsoleOutputA and read back with ReadConsoleOutputW when the font is Raster,
// we will get back de-duplicated Unicode characters with no lead / trail markings.The extra array space will remain null.
return PrepPattern::NullPaddedDedupeW;
}
// When written with WriteConsoleOutputW and read back with ReadConsoleOutputA when the font is TrueType,
// we will get back Unicode characters doubled up and marked with leading and trailing bytes.
return PrepPattern::DoubledW;
}
else
{
@ -1103,17 +736,8 @@ const CharInfoPattern& DbcsWriteRead::PrepReadConsoleOutput(
// Writing with the CRT down here.
if (fReadWithUnicode)
{
// If we wrote with the CRT and are reading back with the W functions, the font does matter.
if (fIsTrueTypeFont)
{
// In a TrueType font, we will get back Unicode characters doubled up and marked with leading and trailing bytes.
return PrepPattern::DoubledW;
}
else
{
// In a Raster font, we will get back de-duplicated Unicode characters with no lead/trail markings. The extra array space will remain null.
return PrepPattern::NullPaddedDedupeW;
}
// In a TrueType font, we will get back Unicode characters doubled up and marked with leading and trailing bytes.
return PrepPattern::DoubledW;
}
else
{
@ -1131,7 +755,6 @@ const CharInfoPattern& DbcsWriteRead::PrepReadConsoleOutput(
const CharInfoPattern& DbcsWriteRead::PrepReadConsoleOutputCharacter(
const DbcsWriteRead::WriteMode WriteMode,
const UnicodeMode fWriteWithUnicode,
const bool fIsTrueTypeFont,
const bool fReadWithUnicode)
{
if (DbcsWriteRead::WriteMode::WriteConsoleOutputFunc == WriteMode)
@ -1141,49 +764,21 @@ const CharInfoPattern& DbcsWriteRead::PrepReadConsoleOutputCharacter(
case UnicodeMode::UnicodeSingle:
if (fReadWithUnicode)
{
if (fIsTrueTypeFont)
{
return PrepPattern::SpacePaddedDedupeW;
}
else
{
return PrepPattern::ACoverAttrSpacePaddedDedupeTruncatedW;
}
return PrepPattern::SpacePaddedDedupeW;
}
else
{
if (fIsTrueTypeFont)
{
return PrepPattern::TrueTypeCharANullWithAttrs;
}
else
{
return PrepPattern::SpacePaddedDedupeA;
}
return PrepPattern::TrueTypeCharANullWithAttrs;
}
break;
case UnicodeMode::UnicodeDoubled:
if (fReadWithUnicode)
{
if (fIsTrueTypeFont)
{
return PrepPattern::WNullCoverAChar;
}
else
{
return PrepPattern::DoubledTruncatedCoverAChar;
}
return PrepPattern::WNullCoverAChar;
}
else
{
if (fIsTrueTypeFont)
{
return PrepPattern::A;
}
else
{
return PrepPattern::DoubledA;
}
return PrepPattern::A;
}
break;
default:
@ -1214,7 +809,6 @@ const CharInfoPattern& DbcsWriteRead::PrepReadConsoleOutputCharacter(
void DbcsWriteRead::PrepExpected(const WORD wAttrWritten,
const DbcsWriteRead::WriteMode WriteMode,
const DbcsWriteRead::UnicodeMode fWriteWithUnicode,
const bool fIsTrueTypeFont,
const DbcsWriteRead::ReadMode ReadMode,
const bool fReadWithUnicode,
CharInfoPattern& expected)
@ -1223,12 +817,12 @@ void DbcsWriteRead::PrepExpected(const WORD wAttrWritten,
{
case DbcsWriteRead::ReadMode::ReadConsoleOutputFunc:
{
expected = DbcsWriteRead::PrepReadConsoleOutput(WriteMode, fWriteWithUnicode, fIsTrueTypeFont, fReadWithUnicode);
expected = DbcsWriteRead::PrepReadConsoleOutput(WriteMode, fWriteWithUnicode, fReadWithUnicode);
break;
}
case DbcsWriteRead::ReadMode::ReadConsoleOutputCharacterFunc:
{
expected = DbcsWriteRead::PrepReadConsoleOutputCharacter(WriteMode, fWriteWithUnicode, fIsTrueTypeFont, fReadWithUnicode);
expected = DbcsWriteRead::PrepReadConsoleOutputCharacter(WriteMode, fWriteWithUnicode, fReadWithUnicode);
break;
}
default:
@ -1343,7 +937,6 @@ void DbcsWriteRead::Verify(std::span<CHAR_INFO> rgExpected, std::span<CHAR_INFO>
}
void DbcsWriteRead::TestRunner(_In_opt_ WORD* const pwAttrOverride,
const bool fUseTrueType,
const DbcsWriteRead::WriteMode WriteMode,
const UnicodeMode fWriteInUnicode,
const DbcsWriteRead::ReadMode ReadMode,
@ -1354,7 +947,7 @@ void DbcsWriteRead::TestRunner(_In_opt_ WORD* const pwAttrOverride,
// used by default in the buffer (set during clearing as well).
HANDLE hOut;
WORD wAttributes;
if (!DbcsWriteRead::Setup(fUseTrueType, &hOut, &wAttributes))
if (!DbcsWriteRead::Setup(&hOut, &wAttributes))
{
// If we can't set up (setup will detect systems where this test cannot operate) then return early.
return;
@ -1373,7 +966,7 @@ void DbcsWriteRead::TestRunner(_In_opt_ WORD* const pwAttrOverride,
// Prepare the array of CHAR_INFO structs that we expect to receive back when we will call read in a moment.
// This can vary based on font, unicode/non-unicode (when reading AND writing), and codepage.
CharInfoPattern pciExpected;
DbcsWriteRead::PrepExpected(wAttributes, WriteMode, fWriteInUnicode, fUseTrueType, ReadMode, fReadWithUnicode, pciExpected);
DbcsWriteRead::PrepExpected(wAttributes, WriteMode, fWriteInUnicode, ReadMode, fReadWithUnicode, pciExpected);
// Now call the appropriate READ API for this test.
CharInfoPattern pciActual{};
@ -1385,9 +978,6 @@ void DbcsWriteRead::TestRunner(_In_opt_ WORD* const pwAttrOverride,
void DbcsTests::TestDbcsWriteRead()
{
bool fUseTrueTypeFont;
VERIFY_SUCCEEDED(TestData::TryGetValue(L"fUseTrueTypeFont", fUseTrueTypeFont));
int iWriteMode;
VERIFY_SUCCEEDED(TestData::TryGetValue(L"WriteMode", iWriteMode));
auto WriteMode = (DbcsWriteRead::WriteMode)iWriteMode;
@ -1441,8 +1031,7 @@ void DbcsTests::TestDbcsWriteRead()
VERIFY_FAIL(L"Read mode not supported");
}
auto testInfo = NoThrowString().Format(L"\r\n\r\n\r\nUse '%s' font. Write with %s '%s'%s. Check Read with %s '%s' API. Use %d codepage.\r\n",
fUseTrueTypeFont ? L"TrueType" : L"Raster",
auto testInfo = NoThrowString().Format(L"\r\n\r\n\r\nWrite with %s '%s'%s. Check Read with %s '%s' API. Use %d codepage.\r\n",
pwszWriteMode,
fWriteInUnicode ? L"W" : L"A",
fWriteInUnicode == DbcsWriteRead::UnicodeMode::UnicodeDoubled ? L" (doubled)" : L"",
@ -1461,7 +1050,6 @@ void DbcsTests::TestDbcsWriteRead()
}
DbcsWriteRead::TestRunner(wAttributes != 0 ? &wAttributes : nullptr,
fUseTrueTypeFont,
WriteMode,
fWriteInUnicode,
ReadMode,

90
src/host/ut_host/DbcsTests.cpp
View File

@ -1,90 +0,0 @@
// Copyright (c) Microsoft Corporation.
// Licensed under the MIT license.
#include "precomp.h"
#include "WexTestClass.h"
#include "../../inc/consoletaeftemplates.hpp"
#include "CommonState.hpp"
#include "globals.h"
#include "../buffer/out/textBuffer.hpp"
#include "dbcs.h"
#include "input.h"
using namespace WEX::Logging;
using namespace WEX::TestExecution;
class DbcsTests
{
TEST_CLASS(DbcsTests);
TEST_METHOD(TestUnicodeRasterFontCellMungeOnRead)
{
const size_t cchTestSize = 20;
// create test array of 20 characters
CHAR_INFO rgci[cchTestSize];
// pick a color to use for attributes to ensure it's preserved.
const auto wAttrTest = FOREGROUND_BLUE | BACKGROUND_GREEN | BACKGROUND_INTENSITY;
// target array will look like
// abcdeLTLTLTLTLTpqrst
// where L is a leading half of a double-wide char sequence
// and T is the trailing half of a double-wide char sequence
// fill ASCII characters first by looping and
// incrementing. we'll cover up the middle later
auto wch = L'a';
for (size_t i = 0; i < ARRAYSIZE(rgci); i++)
{
rgci[i].Char.UnicodeChar = wch;
rgci[i].Attributes = wAttrTest;
wch++;
}
// we're going to do katakana KA, GA, KI, GI, KU
// for the double wide characters.
WCHAR wchDouble = 0x30AB;
for (size_t i = 5; i < 15; i += 2)
{
rgci[i].Char.UnicodeChar = wchDouble;
rgci[i].Attributes = COMMON_LVB_LEADING_BYTE | wAttrTest;
rgci[i + 1].Char.UnicodeChar = wchDouble;
rgci[i + 1].Attributes = COMMON_LVB_TRAILING_BYTE | wAttrTest;
wchDouble++;
}
const std::span<CHAR_INFO> buffer(rgci, ARRAYSIZE(rgci));
// feed it into UnicodeRasterFontCellMungeOnRead to confirm that it is working properly.
// do it in-place to confirm that it can operate properly in the common case.
auto dwResult = UnicodeRasterFontCellMungeOnRead(buffer);
// the final length returned should be the same as the length we started with
if (VERIFY_ARE_EQUAL(ARRAYSIZE(rgci), dwResult, L"Ensure the length claims that we are the same before and after."))
{
Log::Comment(L"Ensure the letters are now as expected.");
// the expected behavior is to reduce the LEADING/TRAILING double copies into a single copy
WCHAR wchExpected[]{ 'a', 'b', 'c', 'd', 'e', 0x30AB, 0x30AC, 0x30AD, 0x30AE, 0x30AF, 'p', 'q', 'r', 's', 't' };
for (size_t i = 0; i < ARRAYSIZE(wchExpected); i++)
{
VERIFY_ARE_EQUAL(wchExpected[i], rgci[i].Char.UnicodeChar);
// and simultaneously strip the LEADING/TRAILING attributes
// no other attributes should be affected (test against color flags we set).
VERIFY_ARE_EQUAL(wAttrTest, rgci[i].Attributes);
}
// and all extra portions of the array should be zeroed.
for (auto i = ARRAYSIZE(wchExpected); i < ARRAYSIZE(rgci); i++)
{
VERIFY_ARE_EQUAL(rgci[i].Char.UnicodeChar, 0);
VERIFY_ARE_EQUAL(rgci[i].Attributes, 0);
}
}
}
};

1
src/host/ut_host/Host.UnitTests.vcxproj
View File

@ -15,7 +15,6 @@
<ClCompile Include="ApiRoutinesTests.cpp" />
<ClCompile Include="ClipboardTests.cpp" />
<ClCompile Include="ConsoleArgumentsTests.cpp" />
<ClCompile Include="DbcsTests.cpp" />
<ClCompile Include="HistoryTests.cpp" />
<ClCompile Include="InitTests.cpp" />
<ClCompile Include="ObjectTests.cpp" />

3
src/host/ut_host/Host.UnitTests.vcxproj.filters
View File

@ -21,9 +21,6 @@
<ClCompile Include="ConsoleArgumentsTests.cpp">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="DbcsTests.cpp">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="ScreenBufferTests.cpp">
<Filter>Source Files</Filter>
</ClCompile>

1
src/host/ut_host/sources
View File

@ -20,7 +20,6 @@ SOURCES = \
HistoryTests.cpp \
UtilsTests.cpp \
ConsoleArgumentsTests.cpp \
DbcsTests.cpp \
ScreenBufferTests.cpp \
TextBufferIteratorTests.cpp \
TextBufferTests.cpp \