Chat OTel: drop sprint plan, tighten comments and docs

This commit is contained in:
Zhichao Li
2026-05-23 23:40:57 -07:00
parent 48cde2511d
commit ec79deaf7c
9 changed files with 19 additions and 68 deletions

View File

@@ -85,7 +85,7 @@ Three namespaces coexist on extension-emitted spans:
| Namespace | Purpose | Status |
|---|---|---|
| `gen_ai.*` | OTel GenAI Semantic Conventions. Use whenever a standard key exists. | Canonical |
| `github.copilot.*` | Copilot-specific vendor namespace, shared with the [Copilot CLI runtime](https://github.com/github/copilot-agent-runtime) (see `src/core/otel/otelGenAI.ts`). | **Preferred — new attributes go here.** |
| `github.copilot.*` | Copilot-specific vendor namespace. | **Preferred — new attributes go here.** |
| `copilot_chat.*` | Original VS Code-only namespace. Several keys remain for backwards compatibility. | **Legacy — keep emitting; do not add new keys here.** |
### Dual-emit rules
@@ -95,12 +95,6 @@ Three namespaces coexist on extension-emitted spans:
- Mark the legacy row in [agent_monitoring.md](../../../extensions/copilot/docs/monitoring/agent_monitoring.md) with **Legacy** in the "Requirement" column and a pointer to the preferred key. No sunset date — legacy keys live on indefinitely.
- Hash sensitive identifiers (e.g., MCP server names) with `hashTelemetryValue` from [`util/node/crypto.ts`](../../../extensions/copilot/src/util/node/crypto.ts). Emit hashes unconditionally; raw values only when `captureContent` is enabled.
### Cross-surface enum divergence
A few enums emitted under `github.copilot.*` differ between VS Code and the CLI runtime — flag these explicitly in docs and dashboards:
- `github.copilot.skill.source`: VS Code emits the raw `PromptFileSource` enum value (`github-workspace`, `copilot-personal`, `agents-workspace`, `extension-contribution`, …). The CLI emits its own enum (`project`, `personal-copilot`, `plugin`, `builtin`, `remote`, …). Reconcile server-side if joining the two surfaces.
## 4. Service Layer & Selection
`IOTelService` ([otelService.ts](../../../extensions/copilot/src/platform/otel/common/otelService.ts)) is the only abstraction consumers should depend on — never import the OTel SDK directly outside `node/otelServiceImpl.ts`. Three implementations:

View File

@@ -119,12 +119,10 @@ OTel is **off by default** with zero overhead. It activates when:
> Copilot Chat emits OTel attributes under three namespaces:
>
> - **`gen_ai.*`** — [OTel GenAI Semantic Conventions](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/gen-ai/). Use these whenever a standard key exists.
> - **`github.copilot.*`** — The canonical Copilot-specific namespace, shared with the [Copilot CLI runtime](https://github.com/github/copilot-agent-runtime). Prefer this for new dashboards and alerts.
> - **`copilot_chat.*`** — The original VS Code extension namespace. Several keys (notably `copilot_chat.repo.*` and `gen_ai.usage.reasoning_tokens`) are now **dual-emitted alongside the `github.copilot.*` equivalents**. Tables below mark these rows as **Legacy** with a pointer to the preferred key.
> - **`github.copilot.*`** — Canonical Copilot-specific namespace. Prefer this for new dashboards and alerts.
> - **`copilot_chat.*`** — Original VS Code extension namespace. Several keys (notably `copilot_chat.repo.*` and `gen_ai.usage.reasoning_tokens`) are now **dual-emitted alongside the `github.copilot.*` equivalents**. Tables below mark these rows as **Legacy** with a pointer to the preferred key.
>
> Legacy keys continue to emit indefinitely so existing collectors, dashboards, and downstream consumers (Agent Debug Log, Chronicle, SQLite span store) keep working without changes. There is no sunset date.
>
> **Cross-surface compatibility caveat:** A few enums diverge between VS Code and the CLI runtime (e.g. `skill.source` carries VS Code's `PromptFileSource` value on extension-emitted spans and the CLI's own enum on CLI-emitted spans). Where this matters, the divergence is called out on the relevant attribute row.
### Traces

View File

@@ -1,36 +0,0 @@
# OTel CLI Parity Sprint — branch `zhichli/oteladdition`
Reference: copilot-agent-runtime CLI PR #8037 (`e8a24076831b09a17f7f641b33885ddd5fcb4b3e`).
Goal: bring VS Code's extension-side OTel up to the new `github.copilot.*` shape introduced in the CLI, without breaking existing `copilot_chat.*` consumers (Agent Debug Log, Chronicle, OTLP exporters, SQLite span store).
## Locked design decisions
| # | Decision |
|---|---|
| 1 | MCP server names hashed by default (SHA-256 hex). Raw name only when `captureContent=true`. |
| 2 | `agent.type` derived from existing `modeInstructions2.isBuiltin` flag (`builtin` / `custom`). A richer registry-source field is a follow-up. |
| 3 | Skill source emitted verbatim as VS Code's `PromptFileSource` enum value (no CLI mapping). Deferred — skill emission not part of this PR. |
| 4 | Indefinite dual-emit; legacy `copilot_chat.*` keys keep emitting, marked **Legacy** in doc, no sunset. |
## Tasks (atomic commits)
1. **feat(otel): add `github.copilot.*` attribute constants and hash helper**`genAiAttributes.ts`
2. **feat(otel): dual-emit git context under `github.copilot.git.*`**`workspaceOTelMetadata.ts`
3. **feat(otel): rename reasoning tokens key with dual-emit**`chatMLFetcher.ts`, `otelSqliteStore.ts`
4. **feat(otel): stamp `github.copilot.agent.type` on `invoke_agent`**`toolCallingLoop.ts`
5. **feat(otel): structured `github.copilot.tool.parameters.*` on `execute_tool`**`toolsService.ts`
6. **feat(otel): enrich `execute_hook` with `decision` / `tool_names` / `duration_seconds`**`chatHookService.ts`
7. **docs(otel): document `github.copilot.*` attributes and mark `copilot_chat.repo.*` legacy**`agent_monitoring.md`
8. **docs(otel): add dual-emit policy section to OTel skill**`.github/skills/otel/SKILL.md`
## Deferred to follow-up PRs
- `github.copilot.skill.invoked` span event with raw `PromptFileSource`.
- `github.copilot.mcp.server.lifecycle` event + `connection.count` counter.
- `github.copilot.context.{skills,mcp_server_names,custom_agent_names}` snapshot attrs on `invoke_agent`.
- Mode/agent registry `source` field for the richer `agent.type` enum.
## Hiccups & Notes
(Filled in during execution.)

View File

@@ -186,7 +186,6 @@ export class ChatHookService implements IChatHookService {
: 'error';
span.setAttribute(CopilotChatAttr.HOOK_RESULT_KIND, resultKind);
// Map to CLI's `decision` enum for cross-surface dashboards.
const hookDecision = commandResult.kind === HookCommandResultKind.Error
? 'block'
: commandResult.kind === HookCommandResultKind.NonBlockingError
@@ -218,7 +217,7 @@ export class ChatHookService implements IChatHookService {
// If stopReason is set (including empty string for "stop without message"), stop processing remaining hooks
if (result.stopReason !== undefined) {
// Stop signals from a successful hook still flip the decision to `block` for CLI parity.
// A stop signal from a successful hook still counts as a block.
if (hookDecision === 'pass') {
span.setAttribute(GitHubCopilotAttr.HOOK_DECISION, 'block');
}

View File

@@ -738,8 +738,6 @@ export abstract class ToolCallingLoop<TOptions extends IToolCallingLoopOptions =
// Extract custom mode name for debug logging (kept separate from agentName to avoid metric cardinality)
const modeInstructions = (this.options.request as { modeInstructions2?: { name?: string; isBuiltin?: boolean } }).modeInstructions2;
const customModeName = modeInstructions?.name && !modeInstructions.isBuiltin ? modeInstructions.name : undefined;
// `builtin` for default/built-in modes, `custom` for user/workspace-contributed modes.
// `plugin` is reserved for extension-contributed modes once that lands in the registry.
const agentType: 'builtin' | 'custom' = modeInstructions && modeInstructions.isBuiltin === false ? 'custom' : 'builtin';
// If this is a subagent request, look up the parent trace context stored by the parent agent's execute_tool span

View File

@@ -153,9 +153,8 @@ export class ToolsService extends BaseToolsService {
} catch { /* swallow serialization errors */ }
}
// Structured `github.copilot.tool.parameters.*` (CLI parity). Always-safe
// attrs (hashes, edit_type) emit unconditionally; raw paths/commands/MCP
// names are gated on captureContent.
// Structured `github.copilot.tool.parameters.*`. Hashes and edit_type emit
// unconditionally; raw paths, commands, and MCP names are gated.
try {
const { attrs: paramAttrs, gatedAttrs: gatedParamAttrs } = extractToolParameters(String(name), options.input);
for (const [k, v] of Object.entries(paramAttrs)) {

View File

@@ -193,8 +193,7 @@ export const CopilotCliSdkAttr = {
} as const;
/**
* Canonical `github.copilot.*` attribute namespace shared with the Copilot CLI
* runtime (see copilot-agent-runtime `src/core/otel/otelGenAI.ts`). These
* Canonical `github.copilot.*` attribute namespace for Copilot Chat. These
* attributes are dual-emitted alongside the legacy `copilot_chat.*` keys; new
* dashboards should prefer this namespace.
*/
@@ -243,7 +242,7 @@ export type AgentType = 'builtin' | 'plugin' | 'custom';
export type HookDecision = 'block' | 'approve' | 'non_blocking_error' | 'pass';
export type EditOperationType = 'create' | 'update' | 'str_replace' | 'insert';
/** Max length for the `tool.parameters.command` attribute (matches CLI). */
/** Max length for the `tool.parameters.command` attribute. */
export const TOOL_PARAM_COMMAND_MAX_LEN = 256;
/** Tool names treated as shell-command tools for parameter extraction. */
@@ -256,12 +255,12 @@ export const SHELL_TOOL_NAMES: ReadonlySet<string> = new Set([
]);
/**
* Tool names treated as file tools for parameter extraction. Includes both
* Copilot CLI names (camelCase / Claude-style) and VS Code's `ToolName` enum
* values (snake_case, see `extension/tools/common/toolNames.ts`).
* Tool names treated as file tools for parameter extraction. Covers VS Code's
* `ToolName` enum values (snake_case, see `extension/tools/common/toolNames.ts`)
* and the camelCase / Claude-style variants seen on external surfaces.
*/
export const FILE_TOOL_NAMES: ReadonlySet<string> = new Set([
// Copilot CLI / Claude-style names
// camelCase / Claude-style names
'view',
'create',
'edit',

View File

@@ -53,7 +53,7 @@ function buildWorkspaceMetadata(repoContext: RepoContext, fileUri?: URI): Worksp
/**
* Convert workspace metadata to OTel attributes, omitting undefined values.
* Emits both the legacy `copilot_chat.repo.*` namespace and the canonical
* `github.copilot.git.*` namespace (CLI parity, see PR #8037).
* `github.copilot.git.*` namespace.
*/
export function workspaceMetadataToOTelAttributes(
metadata?: WorkspaceOTelMetadata,

View File

@@ -24,9 +24,10 @@ export interface ToolParameterAttributes {
}
/**
* Port of the CLI's `extractToolParameters` (copilot-agent-runtime PR #8037,
* `src/core/otel/otelGenAI.ts`). Produces `github.copilot.tool.parameters.*`
* attributes for shell, file, skill, and MCP tool calls.
* Produces structured `github.copilot.tool.parameters.*` attributes for shell,
* file, skill, and MCP tool calls. `gatedAttrs` are content-sensitive (raw
* paths, commands, MCP server names) and emit only when `captureContent` is
* enabled; `attrs` are always safe (hashes, fixed enums).
*/
export function extractToolParameters(toolName: string, input: unknown): ToolParameterAttributes {
const attrs: Record<string, string> = {};
@@ -63,9 +64,8 @@ export function extractToolParameters(toolName: string, input: unknown): ToolPar
attrs[GitHubCopilotAttr.TOOL_PARAM_SKILL_NAME] = skillName;
}
// MCP-style tool names: VS Code/CLI use `mcp_<server>_<tool>`; Anthropic-style
// references use `mcp__<server>__<tool>`. Accept both: try double-underscore
// first, then fall back to single.
// MCP-style tool names: VS Code emits `mcp_<server>_<tool>`; Anthropic-style
// references use `mcp__<server>__<tool>`. Accept both.
if (toolName.startsWith('mcp__')) {
const rest = toolName.slice('mcp__'.length);
const sep = rest.indexOf('__');