Commit Graph

455 Commits

Author SHA1 Message Date
dependabot[bot]
7e7f9505b9 build(deps): bump tmp from 0.2.6 to 0.2.7 in /test/automation (#323938)
Bumps [tmp](https://github.com/raszi/node-tmp) from 0.2.6 to 0.2.7.
- [Changelog](https://github.com/raszi/node-tmp/blob/master/CHANGELOG.md)
- [Commits](https://github.com/raszi/node-tmp/compare/v0.2.6...v0.2.7)

---
updated-dependencies:
- dependency-name: tmp
  dependency-version: 0.2.7
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-01 21:13:46 +00:00
Robo
4b6f5e55bb chore: bump electron@42.5.0 (#321629)
* chore: bump electron@42.4.0

* chore: apply temp dir workaround for short paths

* chore: use 24.15.x for CI node

* chore: update nodejs build

* chore: bump electron@42.5.0

* fix: unblock playwright install on node 24.17

Node 24.16+ made Readable pause()/resume() a no-op on destroyed streams
which makes yauzl 2.x / extract-zip 2.x and older playwright extraction
hang forever.

- extensions/copilot: add "yauzl": "^3.3.1" override (was missed by #318682)
  so electron and @vscode/vsce no longer resolve the broken yauzl 2.10, fixing the
  hung `npm ci` in the Copilot and Extract chat-lib pipelines.
- extensions/copilot: bump electron ^39.8.5 -> ^42.5.0 so its install
  script uses the native @electron-internal/extract-zip instead of extract-zip.
- bump @playwright/test ^1.56.1 -> ^1.61.1 so `playwright install`
  uses the fixed extractor, unblocking the "Download Electron and
  Playwright" step in all electron test pipelines.

* chore: update build

* agentHost: fix macOS sandbox smoke sentinel parsing

On macOS CI, the AgentHost sandbox smoke test resolves the shell to
/bin/sh, which uses the sentinel-based completion path. In that path, the
parser could consume the echoed sentinel command text
(`<<<COPILOT_SENTINEL_..._EXIT_$?>>>`) before the real numeric marker
arrived, causing a false `Exit code: -1` failure even though the command
later completed successfully.

Harden the sentinel parser to ignore echoed/non-numeric sentinel text
and use the latest complete numeric marker instead. Also force the
macOS AgentHost sandbox smoke test to use /bin/sh and assert that in the
suite log so local runs exercise the same path as CI.

Adds a regression test for echoed sentinel command text.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chore: update screenshot baseline after playwright bump

* chore: bump distro

* chore: fix typecheck

* chore: bump distro

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-06-29 13:53:48 +00:00
Paul
12149dcfef Fix model config smoke test flake (#323304) 2026-06-28 02:51:22 +00:00
Paul
da4387232c Add smoke tests for model configs (#323205) 2026-06-27 17:51:29 +00:00
Alexandru Dima
f3176866ad test: always log failed requests and console errors in web smoke tests (#323020)
Move `requestfailed` (context + page) and console error/warning logging out
of the `--verbose` block in the Playwright browser driver so they are always
captured. Without this, a stalled or aborted module fetch during web workbench
bootstrap can prevent `.monaco-workbench` from rendering without producing a
page error, crash or HTTP error response, leaving the failure (a 20s timeout)
impossible to root cause from CI logs. The chatty lifecycle events stay gated
behind `--verbose`.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-06-25 22:35:05 +00:00
Alexandru Dima
18a52c8a29 smoke: fail fast with actionable error when Electron crashes during launch (#322905)
* smoke: fail fast with actionable error when Electron crashes during launch

The per-suite `before all` hook launches Electron via Playwright's
`_electron.launch`. This was called with `timeout: 0`, so when the
Electron process crashed during startup (e.g. a native SIGSEGV in the
Chromium browser-process/sandbox init path, which leaves a minidump),
the launch promise never resolved or rejected. It simply hung until
Mocha's 120s `before all` hook timeout fired, producing an opaque
"Timeout of 120000ms exceeded" with no hint that a native crash had
occurred.

Give the launch (and the first-window wait) a finite timeout well below
the Mocha hook timeout and wrap them so a failure is re-thrown as an
actionable error pointing at the crash-dump directory and Playwright
trace. This turns a 120s opaque hang into a faster, clearly-diagnosable
failure.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-06-25 14:15:01 +00:00
Alex Ross
71292ea46e Try fix Copilot CLI editor flake (#322722)
* Try fix Copilot CLI editor flake

* Smoke test probes

* Remove probe
2026-06-24 08:56:43 -07:00
Alexandru Dima
de07516770 smoke: add remote server process exit diagnostics (#322697)
Remote smoke tests intermittently fail with "Cannot reconnect. Please reload
the window." The root cause is the remote server process exiting unexpectedly
shortly after a client connects; because the server logs nothing on the way out,
the client reconnects against a freshly spawned server with a new connection
token and hits "Unknown reconnection token (never seen)".

Add best-effort process exit diagnostics in server-main.ts that distinguish a
self-exit (beforeExit), an external kill (signal) and a crash
(uncaughtExceptionMonitor), and log the test resolver child's exit/close signal.
Gated behind the VSCODE_SERVER_EXIT_DIAGNOSTICS env var, which the remote smoke
launcher sets, so it adds no product noise.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-06-24 12:47:37 +02:00
Alex Ross
6577439bdf Reenable 2nd turn of Copilot CLI test (#322056)
* Reenable 2nd turn of Copilot CLI test

* test running on azure dev ops

* Restore title change guard

* Run even if failure

* Try fix missing chat session

* Try to prevent session from getting disposed

* Try to fix another flake

* Remove flake probe
2026-06-23 18:37:41 +05:00
Alexandru Dima
16c0598ca1 smoke: fix flaky Agents Window Copilot CLI session activation (#322170)
The "Agents Window > Test Copilot CLI session" smoke test flakily timed
out at `activateSessionByLabel`, which located the just-completed session
row by its title and expected it to equal the mocked reply
`MOCKED_COPILOT_RESPONSE`. That title is set asynchronously by a utility
model after the first turn and races the untitled->committed session
swap, so it is non-deterministically either the user's prompt (the
synchronous fallback) or the generated reply. The test passed on macOS
(title-gen landed) and failed on Linux (it didn't) in the same run.

Decouple row identification from response verification:
`activateSessionByLabel` now accepts one or several row substrings and
matches a row containing ANY of them, while a separate `responseLabel`
is verified against the active session's response bubble. The smoke test
passes both the first prompt and the reply, so activation is
deterministic regardless of when title generation completes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-06-20 01:15:46 +02:00
Alex Ross
d00a7238ac Try to fix another Copilot CLI bug (#321935) 2026-06-18 16:51:01 +02:00
Alex Ross
193b1196d8 Add a second turn to the chat smoke tests (#319942)
* Reduce duplicate code

* Add a second turn to the chat smoke tests

* Fix tests

* "fix" sandbox test

* Make things work

* Get tests passing

* CCR feedback

* Simplify diff

* Undo chatView.ts changes

* Missing Claude warm up
2026-06-18 12:22:13 +02:00
Giuseppe Cianci
8637799a55 Merge remote-tracking branch 'origin/main' into agents/codex-harness-improvement-plan
# Conflicts:
#	src/vs/platform/agentHost/node/codex/codexAgent.ts
2026-06-17 19:00:37 +02:00
Giuseppe Cianci
8e32735940 Codex smoke: route shared CAPI client at the mock so the turn actually runs
The Codex smoke test previously could only skip: when the Codex session
type WAS available, the agent host's shared CopilotApiService discovered
CAPI endpoints by calling a hardcoded `api.github.com/copilot_internal/user`,
which 401s with the fake smoke token (the other harnesses avoid this —
Copilot CLI uses the @github/copilot SDK pointed at COPILOT_API_URL, and
the renderer-side Claude test uses the extension token manager). So no
agent-host model path had ever been mocked.

- copilotApiService.ts: honor a test-only `VSCODE_AGENT_HOST_CAPI_URL_OVERRIDE`
  env. When set, skip the api.github.com discovery and route every CAPI
  request (models/responses/messages) at the override. Never set in
  production, so per-token discovery is unchanged.
- agentsWindow.test.ts (setupAgentHostSuite): set that override to the mock
  server URL, so the codex harness's model list + /responses traffic reach
  the mock.
- agentsWindow.ts (isSessionTypeAvailable): the picker dropdown is a one-shot
  snapshot, so a provider that registers a few seconds after load (codex
  spawns a native app-server first) was missed when the dropdown opened too
  early. Re-open the dropdown on each poll and use a generous timeout so the
  gate reliably runs when codex is present and only skips when it is absent.

Verified locally end-to-end: with codex available the test selects Codex,
the real app-server drives a turn through the proxy to the mock /responses
endpoint (codex's native payload), and MOCKED_CODEX_RESPONSE renders. With
codex unavailable it skips. 71 codex + 83 copilotApiService unit tests pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-06-17 18:44:24 +02:00
Giuseppe Cianci
e6967d8b45 Codex: add Agents-Window smoke test (real app-server, mock LLM)
Adds an end-to-end smoke test for the codex harness that uses the REAL
codex app-server and fakes only the LLM. It reuses the existing
Agents-Window smoke harness, which already drives:

  real Agents-Window UI -> agent host -> real spawned `codex` binary ->
  real CodexProxyService -> mock LLM server

The mock LLM server (scripts/chat-simulation/common/mock-llm-server.ts)
already speaks codex's OpenAI `/responses` API and ships a
`gpt-5.3-codex` `/responses` chat-default model, so no LLM-side changes
are needed.

- test/smoke/.../agentsWindow.test.ts: new `Agents Window (Codex)` suite
  via the existing setupAgentHostSuite helper with
  `chat.agentHost.codexAgent.enabled` (codex is off by default). The test
  selects the `Codex` session type, submits a scenario prompt, and
  asserts the mocked reply plus a `chat/turnStarted` frame in the AHP
  transcript (proving the codex harness, not a renderer fallback, served
  it). Adds a warmUpCodexModel helper mirroring warmUpClaudeModel.
- test/automation/src/agentsWindow.ts: new isSessionTypeAvailable(label)
  used to skip gracefully when codex isn't registered.

Codex only registers when its SDK is resolvable (product.agentSdks.codex
in packaged builds, or VSCODE_AGENT_HOST_CODEX_SDK_ROOT in dev); when
absent the test skips rather than fails.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-06-17 18:14:08 +02:00
Sandeep Somavarapu
d67d6aadf2 sessions: show provider names as section headers in session type picker (#321597)
* sessions: show provider names as section headers in session type picker

The session type picker (harness picker) showed provider names like
"Copilot Chat" and "Local Agent Host" as inline descriptors to the right
of the first row in each group, which read like a label on that row and
made it look like there were duplicate/broken entries.

Render the provider name as a proper section header above each group,
with a divider between sections, when more than one provider contributes
session types. Session types are grouped by their provider's display
label (preserving first-seen order) so each header is shown once even
when providers are interleaved or share a label. The same grouping is
applied to the mobile bottom-sheet variant.

Also rename the local provider label from "Local Chat" to "Copilot Chat".

Fixes #321535

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* sessions: give session type picker a minimum width

Set a minimum width on the session type picker dropdown so it does not
collapse to a narrow popup when labels are short.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* sessions: set session type picker minimum width to 200

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* sessions: fix AgentHost smoke test selection and picker a11y for section headers

The session type picker now renders provider names as non-clickable
header rows. The smoke automation's selectSessionType matched the
provider label ("Local Agent Host") on an action row's inline group
title and clicked it; with headers that text is on a header row, so the
click did nothing and AgentHost sessions never started (CI timeouts).

- Update selectSessionType to select the first actionable row beneath a
  matching section header, while still matching session type labels
  directly on action rows.
- Include the provider label in the action item ARIA label when section
  headers are shown, so same-named types (e.g. "Claude") are
  disambiguated for screen readers since header rows are skipped by list
  navigation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-06-16 08:35:46 -07:00
Alexandru Dima
53ff304e90 Fix Agents Window smoke test: use runCommand instead of keybinding (#321225)
The NewChatInSessionsWindowAction keybinding (Ctrl+L) is gated on
the chat editor silently does nothing. This caused the 'Test Claude
session' and 'Test Local session' smoke tests to time out waiting for the
new-session view.

- Add `f1: true` to NewChatInSessionsWindowAction so it appears in the
  command palette (only registered in the Agents Window context)
- Change `startNewSession()` in the automation helper to use
  `quickaccess.runCommand()` instead of `dispatchKeybinding('ctrl+l')`,
  bypassing the focus gate entirely

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-06-12 22:14:56 +00:00
Alex Ross
0b18363732 Fix more flake (#321123) 2026-06-12 14:02:34 +02:00
Alex Ross
446cd9331f Try to fix flake by retrying command palette (#321105) 2026-06-12 11:50:46 +02:00
Lee Murray
644acb0d33 Refactor chat action icons for consistency (#320752)
* refactor: update chat action icons from arrowUp to newLine for consistency

* update chat send icon from arrowUp to newLine in chat input and widget fixtures

* update chat send button icon from arrowUp to newLine

---------

Co-authored-by: mrleemurray <mrleemurray@users.noreply.github.com>
2026-06-10 09:25:42 -07:00
dependabot[bot]
055061c84e Bump shell-quote from 1.7.3 to 1.8.4 in /test/automation (#320621)
Bumps [shell-quote](https://github.com/ljharb/shell-quote) from 1.7.3 to 1.8.4.
- [Changelog](https://github.com/ljharb/shell-quote/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ljharb/shell-quote/compare/v1.7.3...v1.8.4)

---
updated-dependencies:
- dependency-name: shell-quote
  dependency-version: 1.8.4
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-09 17:09:48 +00:00
Christof Marti
7d08c0750c Add local agent host test (#317981) 2026-06-08 16:07:05 +02:00
Christof Marti
837d9b877e Add Copilot CLI sandbox test (#317981) 2026-06-04 15:12:55 -07:00
Alex Ross
616eab07e3 Add "hello" tests to VS Code smoke tests (#319719)
* Add "hello" tests to VS Code smoke tests

* Address CCR feedback
2026-06-03 15:29:51 +02:00
dependabot[bot]
97dfdd4d6f build(deps): bump tmp from 0.2.4 to 0.2.6 in /test/automation (#318673)
Bumps [tmp](https://github.com/raszi/node-tmp) from 0.2.4 to 0.2.6.
- [Changelog](https://github.com/raszi/node-tmp/blob/master/CHANGELOG.md)
- [Commits](https://github.com/raszi/node-tmp/compare/v0.2.4...v0.2.6)

---
updated-dependencies:
- dependency-name: tmp
  dependency-version: 0.2.6
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-28 17:03:16 +00:00
Robo
8748be1f1a feat: update to Electron v42 (#316661)
* feat: update to Electron v42

* chore: drop support for arm 32-bit server

* chore: update types/node to v24.x

* chore: temporarily lock the target version for build/

* chore: update v8-source-location.patch

* chore: fix clippy

* chore: cleanup armhf server ci config

* fix: broken lock file

* fix: c++ version requirement for sysroot builds

* fix: msvc compilation of native modules

* fix: handle rejections for fire-and-forget loadurl

* fix: windows build

* ci: fix teardown of daemon process on windows

```
2026-05-15T20:55:09.7717127Z Assertion failed: !(handle->flags & UV_HANDLE_CLOSING), file src\win\async.c, line 76
```

* chore: update deb and rpm dependencies

* chore: update version info

* spec: improve reliablity of offscreencanvas tests

* spec: retry EPERM failures on teardown

* chore: update x86_64 rpm deps

* ci: exclude server binskim for armhf

* temp: bump distro

* test: ignore deprecation warnings treated as errors

* chore: update lockfile

* fix: externalize electron from auth extension bundles

 Error: Cannot find module 'c:\Users\cloudtest\AppData\Local\Temp\vscode-sanityQvCaze\vscode-server-win32-x64-web\extensions\github-authentication\dist\install.js'
     at Module._resolveFilename (node:internal/modules/cjs/loader:1476:15)
     at wrapResolveFilename (node:internal/modules/cjs/loader:1049:27)
     at defaultResolveImplForCJSLoading (node:internal/modules/cjs/loader:1073:10)
     at resolveForCJSWithHooks (node:internal/modules/cjs/loader:1094:12)
     at Module._load (node:internal/modules/cjs/loader:1262:25)
     at wrapModuleLoad (node:internal/modules/cjs/loader:255:19)
     at Module.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:154:5)
     at node:internal/main/run_main_module:33:47 {
   code: 'MODULE_NOT_FOUND',
   requireStack: []
 }

 Node.js v24.15.0

* test: make wsl sanity tests reliable

* chore: bump electron@42.1.0

* temp: bump distro

* chore: bump electron@42.2.0

* chore: bump distro

* chore: update debian dependencies

* Revert "test: make wsl sanity tests reliable"

This reverts commit b3f2b63e83.

* test: do not fail for deprecation warnings

* chore: patch node24 server binary for wsl1

* chore: address review feedback

* chore: revert global navigation error handler in browserview

* chore: bump distro
2026-05-27 14:57:41 +02:00
Anthony Kim
3827a8b38a Smoke test for Copilot CLI via chat extension in editor window (#317140)
* Copilot CLI sanity testing

* Make things better

* see if 1.0.48 correctly fails

* Try to be smarter with git auth

* Test  if copilot cli sanity test auth correctly.

* Test if copilot cli sanity test FAILS correctly

* Try to get sanity tests pass?

* Copilot CLI from chat extension sanity test

* Try more smoke test for copilot cli

* More smoke test related things?

* title is messing with chat disabled..?

* Why is there timeout for .editor-instance .interactive-session

* add copilot cli ui smoke auth diagnostics

* Be more descriptive when copilot cli smoke test fail

* remove integration test, upgrade smoke test

* Clean up Copilot CLI smoke test diagnostics

* Get even more inspiration from agent smoke test
2026-05-21 16:00:35 -07:00
Sandeep Somavarapu
8f51c4a890 fix null-safety in LocalNewSession._resolveGitState (#317645)
* fix null-safety in LocalNewSession._resolveGitState

When resolveWorkspace() creates a folder with gitRepository: undefined,
the autorun in _resolveGitState crashed spreading undefined with the
non-null assertion. Build a fallback ISessionGitRepository from the
folder root so the git state update works even when the workspace was
not pre-populated with repository metadata.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* smoke: skip Claude session test pending CI investigation

The Claude session test consistently times out on macOS CI — the
claude-code session controller never starts. Skip it while we
investigate what blocks createNewChatSessionItem on CI builds.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* hoist fallback gitRepository outside autorun

Avoid recreating the fallback ISessionGitRepository and its
constObservable on every git state change by moving it before the
autorun closure.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* smoke: retry send button click if new-session view persists

The send button click can silently fail on CI if the button moved or
an overlay intercepted the event. After clicking, verify the new-session
homepage disappears. If it's still visible after 3 seconds, retry the
click up to 3 times.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-21 00:02:51 +00:00
Sandeep Somavarapu
2e8b995ab1 agents: add smoke test for Agents Window with mocked LLM server (#317545)
* agents: add smoke test for Agents Window with mocked LLM server

Adds a smoke test that opens the Agents Window, creates a new session
on a workspace folder, sends a 'hello world' prompt, and verifies the
request reaches a local mock LLM server that returns a canned response.

The test exercises two session types: Copilot CLI and Claude Code. A
third Local-session test is included but marked `it.skip` for now.

Key pieces:

* `test/smoke/src/areas/agents/agents.test.ts` — new smoke suite. Starts
  `scripts/chat-simulation/common/mock-llm-server.js` on a random port,
  registers a per-test scenario with a distinct reply, and injects
  `VSCODE_COPILOT_CHAT_TOKEN` / `GITHUB_PAT` / `IS_SCENARIO_AUTOMATION`
  env vars so the Copilot extension's token manager picks up a fake
  token whose endpoints.api/proxy point at the mock server.

* `test/automation/src/agents.ts` — new `Agents` workbench helper with
  `openCurrentFolderInAgentsWindow`, `switchToAgentsWindow`,
  `startNewSession`, `selectSessionType`, `submitNewSessionPrompt`,
  `waitForAssistantText`.

* `test/automation/src/code.ts` + `electron.ts` — `LaunchOptions` now
  accepts an `extraEnv` map that is merged on top of `process.env`
  when spawning the Electron child, so tests can inject env-based mocks
  without going through a custom launcher.

* `src/vs/sessions/browser/sessionsSetUpService.ts` —
  `shouldSkipSessionsWelcome` now returns `true` whenever
  `enableSmokeTestDriver` is set, so the welcome/auth dialog does not
  block smoke runs.

* `scripts/chat-simulation/common/mock-llm-server.js` — adds two
  models to `EXTRA_MODELS` (`gpt-5.3-codex` for Copilot CLI default,
  `claude-sonnet-4.5` for Claude Code), and routes `/v1/messages` to a
  new `handleMessagesApi` that streams Anthropic-format SSE
  (`message_start` / `content_block_delta` / `message_stop`) which the
  Claude Code session type's messages-API parser expects.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* fix copilot cli test

* skip claude test

* sessions: fix new-session removal regression and rename smoke test

1. Revert the `_refreshSessionCache` filter change from d5747b31c4
   back to `adapter instanceof AgentSessionAdapter`. The broadened
   `adapter !== this._currentNewSession` check raced with the
   unconditional `this._currentNewSession = undefined` in the
   `_sendFirstChat*` paths: a late callback from a previous session's
   commit would wipe the pointer and the next refresh would evict the
   new session's temp adapter, navigating the Agents Window back to the
   homepage mid-request.

2. Rename the Agents Window smoke test infrastructure for clarity:
   - `test/automation/src/agents.ts` -> `agentsWindow.ts`
     (class `Agents` -> `AgentsWindow`,
     `workbench.agents` -> `workbench.agentsWindow`)
   - `test/smoke/src/areas/agents/agents.test.ts` ->
     `areas/agentsWindow/agentsWindow.test.ts`

Verified with 10 consecutive smoke-test loops: 8/10 fully green
(Copilot CLI + Claude + Local), 2 intermittent UI timing flakes
unrelated to the navigation regression.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* agents smoke: address PR #317545 review feedback

- switchToAgentsWindow: replace fixed 2s sleep with waitForElement on
  the Agents workbench DOM (`.agent-sessions-workbench`) so the helper
  returns as soon as the new window is interactable.
- resolveElectronConfiguration: apply `extraEnv` last, after the
  TESTRESOLVER_* assignments in the remote branch, so caller-provided
  env vars truly have final precedence.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* agents smoke: skip Agents Window tests for OSS quality

The Copilot extension is not built in OSS CI (it's in excludedExtensions
and its dist/extension.js is only produced by its own esbuild pipeline).
Without it all three session-type providers fail to activate, causing
every Agents Window test to time out.

Skip the suite when quality is OSS, matching the pattern used by
setupExtensionTests and setupLocalizationTests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* copilotcli: add proxy endpoint to getAuthInfo for mock server routing

Without `endpoints.proxy`, the SDK's model-fetch calls
(`/models/session`, `/copilot_internal/v2/token`) fall back to
the real GitHub API which rejects the fake HMAC with a 401. This
caused intermittent smoke test failures (1 in 10) because the
Copilot CLI language models never registered, making the chat-setup
readiness gate depend on Claude's model registration timing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* copilotcli: don't cache failed model fetches

When `getAvailableModels` throws (e.g. transient network failure or
HMAC validation error with a proxy), the empty result was permanently
cached in `_availableModels`. Subsequent calls to `getModels()` would
return the cached empty array without retrying, leaving the Copilot CLI
language model provider with zero models for the rest of the session.

Clear `_availableModels` on error so the next call retries the fetch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* agents smoke: reset workspace before opening Agents Window

Earlier smoke test suites (e.g. Tasks) modify .vscode/tasks.json and
leave uncommitted changes. A dirty workspace prevents worktree creation
and triggers the uncommitted-changes confirmation flow which aborts the
Copilot CLI session on builds.

Reset via `git checkout . --quiet` in the before hook, matching the
pattern used by notebook and search smoke tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Sandeep Somavarapu <sandy081@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-05-20 13:34:54 -07:00
Rob Lourens
6fdbdc2548 Fix flaky statusbar smoke test (TOCTOU race in click) (#313141)
* fix: avoid TOCTOU race in smoke test click by using page.click

The smoke test for statusbar 'quick input' was flaky because click() in
PlaywrightDriver used two separate operations:
  1. getElementXY( CDP round-trip to get {x, y}selector)
  2. page.mouse.click(x,  another CDP commandy)

Between these two steps, the page's JS event loop is free to process
queued tasks (~1ms window). In the failing case, the markdown-language-
features extension registers a Language Status '{}' item in the status
bar during this window, shifting the EOL item's x position. The click
then lands on '{}' instead of EOL.

Clicking '{}' invokes ShowTooltipCommand which calls hover.show(true),
trapping keyboard focus in a hover  so the EOL quick input neverwidget
opens and the test times out.

Fix: use page.click(selector) for the common no-offset case. Playwright's
native click waits for the element's bounding box to be stable across
multiple frames before dispatching the click, eliminating the race.

Validated with 50 consecutive runs (0 failures vs. ~1-in-22 before).

Also delete ELECTRON_RUN_AS_NODE from the environment before launching
the test binary. When this env var is set, Electron acts as a plain
Node.js binary and rejects Chromium flags like --remote-debugging-port
that Playwright injects, preventing dev-mode smoke tests from launching.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* address review: fix double-offset bug and reduce page.click timeout

- Fix pre-existing double offset: getElementXY already incorporates both
  xoffset/yoffset into the returned coordinates, so don't add them again
  in mouse.click when both offsets are provided.
- Reduce page.click timeout from 2000ms to 100ms so each poll attempt
  in waitAndClick's retry loop (200  100ms = 20s budget) stays within
  its intended time budget.
- Use xoffset ?? 0 / yoffset ?? 0 instead of falsy checks.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: fall back to legacy click for elements with degenerate bounding box

The previous attempt to use page.click() for all clicks broke smoke tests that
target elements like Monaco's hidden .native-edit-context contenteditable, which
Playwright refuses to interact with (even with force: true) because the element
has a degenerate bounding box. Try page.click first (preserves the TOCTOU fix
for status bar items), then fall back to getElementXY + mouse.click on failure.

(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: don't skip Playwright actionability checks on click

Removing 'force: true' from page.click - the actionability checks (specifically
the 'stable' check that waits for the bounding box to be unchanged across 2
frames) are exactly what fixes the original TOCTOU race. Skipping them caused
clicks to fire during layout shifts and not register their handlers properly,
breaking unrelated tests like the Problems View statusbar click.

The fallback to legacy getElementXY + mouse.click still handles elements that
genuinely fail actionability checks (e.g. Monaco's hidden .native-edit-context).

(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: narrow TOCTOU guard in statusbar.clickOn instead of broad click() change

Revert the playwrightDriver.click() change (which regressed multiple unrelated CI
tests) and instead add a narrowly-scoped fix in statusbar.clickOn(): for
editor-area status items, poll getElementXY until two consecutive samples are
identical before clicking. This guards against the language status ({}) item
shifting EOL/encoding/etc. between position lookup and click dispatch, without
affecting any other test surface.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: use page.click with stable-coords fallback to eliminate statusbar TOCTOU

Use Playwright's page.click (which re-verifies elementFromPoint immediately
before dispatching, closing the TOCTOU window) for editor-area status bar
items. Fall back to a stable-coordinates click when page.click's actionability
check  the known case is Monaco's .native-edit-context overlayfails
(z-index: -10) which elementFromPoint returns instead of the intended target.
Before the fallback, input state is reset (mouse.move to 0,0 + small wait)
so partial hover/mousedown from the failed attempt don't corrupt subsequent
events.

The stable-coordinates fallback polls getElementXY until two consecutive
samples match, then clicks at those exact  minimising (thoughcoordinates
not eliminating) the TOCTOU window in the fallback path too.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* address review: narrow fallback to pointer-intercept errors, use wait() helper, clarify robustClick docs

- robustClick now only falls back on 'intercepts pointer events' errors;
  all other errors are rethrown so real failures aren't silently masked
- Wrap fallback error to preserve both the original and fallback messages
- Replace raw setTimeout with existing wait() helper
- Clarify Code.robustClick JSDoc: not a polling replacement for waitAndClick

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* simplify: remove unnecessary input-state reset before fallback click

The 'intercepts pointer events' error fires at Playwright's actionability
check phase, before any mouse events are dispatched to the  so therepage
is no partial hover/mousedown state to reset.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-30 00:07:57 +00:00
Rob Lourens
995d3dce7c Try to make multiroot smoke test less flaky (#311127)
* Try to make multiroot smoke test less flaky

The 'Multiroot > shows results from all folders' smoke test on macOS
browser was failing because openFileQuickAccessAndWait() retries 9 times
in ~160ms with no backoff when it sees 'No matching results'. On a fresh
remote/browser server start, the workspace folders may not have been
attached yet (extension host still starting), so file search legitimately
returns 'No matching results' and all 9 retries fire before the workspace
is ready.

Add an incremental backoff between retries so slow workspace/file-search
initialization on CI has a chance to catch up.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update test/automation/src/quickaccess.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-17 18:10:47 -07:00
Raymond Zhao
c1f3775929 chore: run npm audit fix (#306509) 2026-03-30 16:32:21 -07:00
dependabot[bot]
094fdac406 Bump picomatch in /test/automation (#304919)
Bumps  and [picomatch](https://github.com/micromatch/picomatch). These dependencies needed to be updated together.

Updates `picomatch` from 2.3.1 to 2.3.2
- [Release notes](https://github.com/micromatch/picomatch/releases)
- [Changelog](https://github.com/micromatch/picomatch/blob/master/CHANGELOG.md)
- [Commits](https://github.com/micromatch/picomatch/compare/2.3.1...2.3.2)

Updates `picomatch` from 4.0.3 to 4.0.4
- [Release notes](https://github.com/micromatch/picomatch/releases)
- [Changelog](https://github.com/micromatch/picomatch/blob/master/CHANGELOG.md)
- [Commits](https://github.com/micromatch/picomatch/compare/2.3.1...2.3.2)

---
updated-dependencies:
- dependency-name: picomatch
  dependency-version: 2.3.2
  dependency-type: indirect
- dependency-name: picomatch
  dependency-version: 4.0.4
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Raymond Zhao <7199958+rzhao271@users.noreply.github.com>
2026-03-26 21:49:26 +00:00
dependabot[bot]
393b54fc07 Bump minimatch from 3.1.2 to 3.1.5 in /test/automation (#300521)
Bumps [minimatch](https://github.com/isaacs/minimatch) from 3.1.2 to 3.1.5.
- [Changelog](https://github.com/isaacs/minimatch/blob/main/changelog.md)
- [Commits](https://github.com/isaacs/minimatch/compare/v3.1.2...v3.1.5)

---
updated-dependencies:
- dependency-name: minimatch
  dependency-version: 3.1.5
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-11 16:55:45 +00:00
Megan Rogge
b40b1dbc3f improve specificity for chat selector to fix smoketest (#292922) 2026-02-04 12:36:26 -08:00
Robo
d0e516655a fix: rename product executable on macOS (#291948)
* fix: rename product executable name on macOS

* chore: update test/automation/src/electron.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* chore: update test/automation/src/electron.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* chore: rename in additional places

* chore: rename in code-perf.js

* chore: create symlink for backwards compatibility

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-02-03 14:50:21 -08:00
Peng Lyu
5366853409 Support multi window scenarios in vscode mcp server (#289250)
* Add window management tools for multi-window support

* Refactor window management tools to remove title references and simplify responses

* Refactor PlaywrightDriver to use readonly page property and remove _currentPage references

* Update test/mcp/src/automationTools/windows.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix URL matching and CDP session handling in multi-window support (#289261)

* Initial plan

* Address code review feedback: improve URL matching and CDP session handling

- Improve URL matching in switchToWindow to prefer exact matches before substring matches
- Clear CDP session when switching windows to prevent using stale sessions
- Add comprehensive documentation explaining the matching strategy and CDP session behavior

Co-authored-by: rebornix <876920+rebornix@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: rebornix <876920+rebornix@users.noreply.github.com>

* Add automation tools for window interactions and update multiplex server configuration

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: rebornix <876920+rebornix@users.noreply.github.com>
2026-02-02 16:11:12 -08:00
Benjamin Pasero
1951d88056 Chat Anonymous: can send a chat message with anonymous access (fix #292296) (#292341) 2026-02-02 17:14:52 +00:00
Matt Bierner
e6f0ffde9c Merge pull request #291937 from mjbvz/dev/mjbvz/severe-buzzard
Update build ts versions
2026-01-31 15:34:15 -08:00
Copilot
b31c729a39 Add smoke test for anonymous chat access (#291953)
* Initial plan

* Add smoke test for anonymous chat access

Co-authored-by: bpasero <900690+bpasero@users.noreply.github.com>

* fix

* .

* .

* .

* .

* .

* .

* .

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: bpasero <900690+bpasero@users.noreply.github.com>
Co-authored-by: Benjamin Pasero <benjamin.pasero@microsoft.com>
2026-01-31 16:33:11 +00:00
Matt Bierner
ee01088b74 Update roots 2026-01-30 13:12:58 -08:00
Tyler James Leonhardt
a69f194713 A semi-working way to send a message in chat (#290309)
* A semi-working way to send a message in chat

The old way stopped working... idk why... and this way isn't perfect - it has to be written on 1 line.... but I guess it's something...

* add this extra check that other places use
2026-01-26 07:13:20 +01:00
Tyler James Leonhardt
52bc869c5d Allow passing in an extension to debug (#290306)
So that the automation can debug an extension
2026-01-26 01:50:20 +00:00
Benjamin Pasero
e2144d22df refactor - improve workspace path handling in smoke tests (#287706) 2026-01-14 06:53:34 +00:00
Peng Lyu
e64d58389d vscode mcp: support custom workspace path; enable restart tool (#286297)
* vscode mcp: support specifying workspace path; enable restart tool

* fix: include workspace path in launch options for browser

* fix: ensure workspace path is set for tests requiring a workspace

* Address PR review comments: support optional workspace path

* fix: standardize workspace path variable naming in tests

* fix: fallback to rootPath for workspacePath in CI environments
2026-01-13 16:38:31 -08:00
Megan Rogge
9986468dd2 add accessibility tests for more of the workbench (#286446) 2026-01-08 21:33:47 +01:00
Benjamin Pasero
851852ceca Smoke test failure: verifies that "hot exit" works for dirty files (fix #286330) (#286333) 2026-01-08 06:36:43 +01:00
Ladislau Szomoru
dd65f0c5ef Engineering - switch from npm-run-all to npm-run-all2 (#284087)
* Introduce compilation error

* Engineering - limit the tasks that we run

* Limit available memory to simulate an OOM

* Try to update the task

* Remove the use of npm-run-all

* Fix script

* Another try

* Try npm-run-all2

* Restore tasks, keep npm-run-all2

* Switch from npm-run-all to npm-run-all2

* Revert changes that were used for testing
2026-01-05 15:49:42 +01:00
Raymond Zhao
41997d514d chore: npm audit fix (#279437)
* chore: npm audit fix

* chore: npm audit fix in extensions

* chore: npm audit fix in tests

* Catch last few
2025-11-25 20:29:09 +00:00
Robo
4c5bfb172a feat: create versioned resources for windows setup (#263998)
* feat: create versioned resources for windows setup

* chore: use inno_updater to remove old installation

* chore: remove old installation as part of setup

* chore: update explorer-command

* chore: prefer session-end

* chore: uninst delete updating_version

* chore: make session-ending write synchronous

* chore: cleanup updateService.win32.ts

* chore: invoke inno_updater gc path for non background update

* chore: move session-end path to runtime

* chore: use commit for updating_version

* chore: fix invalid string

* chore: set appUpdate path

* chore: update inno_updater

* chore: empty commit for testing

* chore: some cleanups

1) Check for session-ending flag in appx and tunnel callsites
2) Move gc for background update to cleanup phase in updateservice
3) Set update state to ready when there is a running inno_setup

* chore: disallow same version update

* chore: disallow application launch in the middle of update

* chore: empty commit for testing

* chore: bump inno_updater

* chore: empty commit for testing

* chore: move gc to update startup

* chore: move feature behind insider only check

* chore: bump inno_updater

* chore: bump explorer-command

* fix: build

* fix: gc for background update in system setup

* chore: create separate cli entrypoints for build

* fix: check for setup mutex created by inno

* chore: remove problematic updatingVersionPath deletion

* chore: remove redundant update check

* chore: bump inno_updater

* chore: fix build

* chore: bump inno updater
2025-11-24 21:32:12 +09:00