{ "version": "Notebook/1.0", "$schema": "https://github.com/Microsoft/Application-Insights-Workbooks/blob/master/schema/workbook.json", "items": [ { "type": 1, "content": { "json": "## Metergrade Economic Control Check\n\nRead-only. Every query on this page runs inside your own Log Analytics workspace. Nothing on this page is transmitted to Metergrade, and the workbook makes no outbound call.\n\nSelect your workspace and a time range, then work down the page. **Start with Preflight** — it reports whether the diagnostic data these queries need is present, which is the difference between an empty result and a misleading one." }, "name": "intro" }, { "type": 9, "content": { "version": "KqlParameterItem/1.0", "parameters": [ { "id": "f2a67877-3501-d52a-2b48-3019d4ac0577", "version": "KqlParameterItem/1.0", "name": "Workspace", "label": "Log Analytics workspace", "type": 5, "isRequired": true, "multiSelect": false, "typeSettings": { "resourceTypeFilter": { "microsoft.operationalinsights/workspaces": true }, "additionalResourceOptions": [] } }, { "id": "f7eda8e3-92d1-3fbf-906e-510ce03f4d15", "version": "KqlParameterItem/1.0", "name": "TimeRange", "label": "Time range", "type": 4, "isRequired": true, "value": { "durationMs": 2592000000 }, "typeSettings": { "selectableValues": [ { "durationMs": 604800000 }, { "durationMs": 2592000000 }, { "durationMs": 7776000000 } ], "allowCustom": true } } ], "style": "pills", "queryType": 0, "resourceType": "microsoft.operationalinsights/workspaces" }, "name": "parameters" }, { "type": 1, "content": { "json": "### Preflight — can this workspace answer the question?\n\nConfirms both APIM tables exist and carry usable data in the selected window. Run this first: a missing diagnostic category is a configuration fact, not a broken query." }, "name": "heading-00-preflight" }, { "type": 3, "content": { "version": "KqlItem/1.0", "query": "// PREFLIGHT — run this first. It answers one question: can the Economic\n// Control Check produce evidence in this workspace, over this window?\n//\n// Read-only. Runs in your Log Analytics workspace; sends nothing anywhere.\n//\n// WHY THIS QUERY EXISTS. The two tables the check reads are created by APIM\n// diagnostic settings, and the LLM table is a SEPARATE category that is off\n// by default. A workspace that has never received that category does not\n// contain an empty table — it contains no table, and every later query\n// fails to resolve rather than returning zero rows. That failure looks like\n// a broken kit. It is a configuration fact, and it is knowable up front.\n//\n// `union isfuzzy=true` is what makes the question askable: it tolerates a\n// table reference that cannot be resolved instead of failing the query. The\n// empty typed datatable is not decoration — it pins the column schema so\n// the join below still resolves when BOTH tables are absent, which is\n// exactly the case this query exists to report.\n\nlet _startTime = todatetime('{TimeRange:start}');\nlet _endTime = todatetime('{TimeRange:end}');\nlet expected = datatable(Table:string, Provides:string)\n[\n 'ApiManagementGatewayLogs', 'Request identity, operation, status, latency',\n 'ApiManagementGatewayLlmLog', 'Model, deployment, token counts'\n];\nlet observed =\n union isfuzzy=true\n (datatable(Table:string, Records:long, Requests:long, TokenBearing:long, Earliest:datetime, Latest:datetime)[]),\n (ApiManagementGatewayLogs\n | where TimeGenerated between (_startTime .. _endTime)\n | summarize Records = count(), Requests = dcount(CorrelationId),\n TokenBearing = long(null),\n Earliest = min(TimeGenerated), Latest = max(TimeGenerated)\n | extend Table = 'ApiManagementGatewayLogs'\n | project Table, Records, Requests, TokenBearing, Earliest, Latest),\n (ApiManagementGatewayLlmLog\n | where TimeGenerated between (_startTime .. _endTime)\n // TokenBearing counts requests that actually reported usage.\n // A present table with no usage data is a third state, and it\n // is the one that silently produces empty economics later.\n | summarize Records = count(), Requests = dcount(CorrelationId),\n TokenBearing = dcountif(CorrelationId, isnotnull(TotalTokens) and TotalTokens > 0),\n Earliest = min(TimeGenerated), Latest = max(TimeGenerated)\n | extend Table = 'ApiManagementGatewayLlmLog'\n | project Table, Records, Requests, TokenBearing, Earliest, Latest);\nexpected\n| join kind=leftouter (observed) on Table\n| extend\n Status = case(\n isnull(Records),\n 'NOT PRESENT — this table does not exist in this workspace. Enable the matching APIM diagnostic setting category.',\n Records == 0,\n 'PRESENT BUT EMPTY — no records in the selected window. Widen the window, or confirm traffic reached this gateway.',\n Table == 'ApiManagementGatewayLlmLog' and TokenBearing == 0,\n 'PRESENT WITHOUT TOKEN DATA — records exist but none report usage. Economics cannot be established from this.',\n 'READY'),\n Records = coalesce(Records, long(0)),\n Requests = coalesce(Requests, long(0))\n| project Table, Provides, Status, Records, Requests, TokenBearing, Earliest, Latest\n| order by Table asc\n", "size": 0, "title": "Preflight — can this workspace answer the question?", "queryType": 0, "resourceType": "microsoft.operationalinsights/workspaces", "crossComponentResources": [ "{Workspace}" ], "visualization": "table" }, "name": "query-00-preflight" }, { "type": 1, "content": { "json": "### Observed consumption by day\n\nRequest volume and token consumption per day, collapsed to one row per request. Usage coverage states how much of the volume the totals rest on." }, "name": "heading-01-observed-spend" }, { "type": 3, "content": { "version": "KqlItem/1.0", "query": "// Observed AI request volume and token consumption, by day.\n// Read-only. Runs in your Log Analytics workspace; sends nothing anywhere.\n//\n// ONE ROW PER REQUEST, NOT PER LOG RECORD.\n//\n// ApiManagementGatewayLlmLog does not emit one record per request. Request,\n// response and streamed-chunk records are written separately under a single\n// CorrelationId. Counting records therefore overstates traffic — on a\n// streaming-heavy estate, by a multiple — and summing token columns across\n// those records double-counts any value reported more than once.\n//\n// So the table is collapsed to one row per CorrelationId first, and a token\n// value is taken only when the records AGREE on it: make_set(x, 2) collects\n// up to two distinct non-null values, so array_length() == 1 proves\n// agreement, == 0 means nothing was reported, and > 1 is a real conflict.\n// A conflict yields null — Metergrade declines the number rather than\n// choosing one. Summing null is not summing zero, which is why the coverage\n// column below exists: it states how much of the volume the totals rest on.\n//\n// A wrong number is worse than a missing one.\n\nlet _startTime = todatetime('{TimeRange:start}');\nlet _endTime = todatetime('{TimeRange:end}');\nlet requests =\n ApiManagementGatewayLlmLog\n | where TimeGenerated between (_startTime .. _endTime)\n | summarize\n TimeGenerated = min(TimeGenerated),\n PromptTokensSet = make_set(PromptTokens, 2),\n CompletionTokensSet = make_set(CompletionTokens, 2),\n TotalTokensSet = make_set(TotalTokens, 2)\n by CorrelationId\n | extend\n PromptTokens = iff(array_length(PromptTokensSet) == 1, tolong(PromptTokensSet[0]), long(null)),\n CompletionTokens = iff(array_length(CompletionTokensSet) == 1, tolong(CompletionTokensSet[0]), long(null)),\n TotalTokens = iff(array_length(TotalTokensSet) == 1, tolong(TotalTokensSet[0]), long(null));\nrequests\n| summarize\n Requests = count(),\n RequestsWithUsage = countif(isnotnull(TotalTokens)),\n InputTokens = sum(PromptTokens),\n OutputTokens = sum(CompletionTokens),\n TotalTokens = sum(TotalTokens)\n by Day = bin(TimeGenerated, 1d)\n// The share of requests the token totals are actually derived from. Below\n// 100%, the totals are a floor and not an estimate of the remainder.\n| extend UsageCoveragePct = round(100.0 * RequestsWithUsage / Requests, 1)\n//\n// KUSTO'S sum() RETURNS 0 FOR A GROUP WHERE EVERY VALUE IS NULL, not null.\n// That turns \"no request in this group reported usage\" into the confident\n// claim \"this group consumed nothing\" — the exact substitution the rest of\n// the kit refuses to make. A bar at zero reads as measured absence of spend;\n// a gap reads as absence of evidence, which is what it is. So a token total\n// resting on zero reporting requests is returned absent.\n| extend\n InputTokens = iff(RequestsWithUsage > 0, InputTokens, long(null)),\n OutputTokens = iff(RequestsWithUsage > 0, OutputTokens, long(null)),\n TotalTokens = iff(RequestsWithUsage > 0, TotalTokens, long(null))\n| project Day, Requests, RequestsWithUsage, UsageCoveragePct, InputTokens, OutputTokens, TotalTokens\n| order by Day asc\n", "size": 0, "title": "Observed consumption by day", "queryType": 0, "resourceType": "microsoft.operationalinsights/workspaces", "crossComponentResources": [ "{Workspace}" ], "visualization": "timechart" }, "name": "query-01-observed-spend" }, { "type": 1, "content": { "json": "### Where consumption concentrates\n\nRequests and tokens by model and deployment, with each row's share of total request volume." }, "name": "heading-02-model-distribution" }, { "type": 3, "content": { "version": "KqlItem/1.0", "query": "// Where the consumption concentrates, by model and deployment.\n// Read-only. Runs in your Log Analytics workspace; sends nothing anywhere.\n//\n// Collapsed to one row per CorrelationId before aggregation, for the reason\n// given in 01-observed-spend.kql: the LLM table emits several records per\n// logical request, so record-level counts and sums are not request-level\n// economics. Model and deployment are taken only where the records agree;\n// where they disagree, or were never reported, the row is grouped under\n// '(not reported)' rather than being dropped. Dropping it would shrink the\n// denominator and quietly overstate the share held by everything else.\n\nlet _startTime = todatetime('{TimeRange:start}');\nlet _endTime = todatetime('{TimeRange:end}');\nlet requests =\n ApiManagementGatewayLlmLog\n | where TimeGenerated between (_startTime .. _endTime)\n | summarize\n ModelNameSet = make_set(ModelName, 2),\n DeploymentNameSet = make_set(DeploymentName, 2),\n PromptTokensSet = make_set(PromptTokens, 2),\n CompletionTokensSet = make_set(CompletionTokens, 2),\n TotalTokensSet = make_set(TotalTokens, 2)\n by CorrelationId\n | extend\n ModelName = iff(array_length(ModelNameSet) == 1, tostring(ModelNameSet[0]), ''),\n DeploymentName = iff(array_length(DeploymentNameSet) == 1, tostring(DeploymentNameSet[0]), ''),\n PromptTokens = iff(array_length(PromptTokensSet) == 1, tolong(PromptTokensSet[0]), long(null)),\n CompletionTokens = iff(array_length(CompletionTokensSet) == 1, tolong(CompletionTokensSet[0]), long(null)),\n TotalTokens = iff(array_length(TotalTokensSet) == 1, tolong(TotalTokensSet[0]), long(null));\nlet totalRequests = toscalar(requests | count);\nrequests\n| extend\n ModelName = iff(isempty(ModelName), '(not reported)', ModelName),\n DeploymentName = iff(isempty(DeploymentName), '(not reported)', DeploymentName)\n| summarize\n Requests = count(),\n RequestsWithUsage = countif(isnotnull(TotalTokens)),\n InputTokens = sum(PromptTokens),\n OutputTokens = sum(CompletionTokens),\n TotalTokens = sum(TotalTokens)\n by ModelName, DeploymentName\n| extend\n ShareOfRequestsPct = round(100.0 * Requests / totalRequests, 1),\n UsageCoveragePct = round(100.0 * RequestsWithUsage / Requests, 1)\n//\n// KUSTO'S sum() RETURNS 0 FOR A GROUP WHERE EVERY VALUE IS NULL, not null.\n// That turns \"no request in this group reported usage\" into the confident\n// claim \"this group consumed nothing\" — the exact substitution the rest of\n// the kit refuses to make. A bar at zero reads as measured absence of spend;\n// a gap reads as absence of evidence, which is what it is. So a token total\n// resting on zero reporting requests is returned absent.\n| extend\n InputTokens = iff(RequestsWithUsage > 0, InputTokens, long(null)),\n OutputTokens = iff(RequestsWithUsage > 0, OutputTokens, long(null)),\n TotalTokens = iff(RequestsWithUsage > 0, TotalTokens, long(null))\n| project ModelName, DeploymentName, Requests, ShareOfRequestsPct, UsageCoveragePct, InputTokens, OutputTokens, TotalTokens\n| order by TotalTokens desc, Requests desc\n", "size": 0, "title": "Where consumption concentrates", "queryType": 0, "resourceType": "microsoft.operationalinsights/workspaces", "crossComponentResources": [ "{Workspace}" ], "visualization": "barchart" }, "name": "query-02-model-distribution" }, { "type": 1, "content": { "json": "### Attribution gaps\n\nHow much observed AI consumption cannot be assigned to an accountable API and operation. The unattributed share stays in the denominator." }, "name": "heading-03-attribution-gaps" }, { "type": 3, "content": { "version": "KqlItem/1.0", "query": "// How much observed AI consumption cannot be assigned to an accountable\n// workload. The unattributed share stays in the denominator — that is the\n// point of the query.\n//\n// Read-only. Runs in your Log Analytics workspace; sends nothing anywhere.\n//\n// WHY THIS QUERY JOINS TWO TABLES.\n//\n// ApiId and OperationId DO NOT EXIST on ApiManagementGatewayLlmLog. That\n// table carries model, deployment and token counts, and identifies the\n// request only by CorrelationId; the closest thing it holds is\n// OperationName, which is not the operation identity used for attribution.\n// Reading ApiId from it does not return empty — it fails to resolve, and\n// the query does not run at all.\n//\n// Request identity lives in ApiManagementGatewayLogs. Attribution is\n// therefore a property of the JOIN, and this query is the only place in the\n// check where the two tables have to meet.\n//\n// Both sides are collapsed to one row per CorrelationId before joining, so\n// the join cannot multiply one request into several. Identity fields are\n// taken only where the gateway records agree; disagreement reads as absent\n// rather than being resolved by picking.\n//\n// LEFT OUTER, anchored on the LLM side: a model call with no matching\n// gateway record is the most unattributed thing in the estate, and an inner\n// join would silently delete exactly that population from the result.\n\nlet _startTime = todatetime('{TimeRange:start}');\nlet _endTime = todatetime('{TimeRange:end}');\nlet llm =\n ApiManagementGatewayLlmLog\n | where TimeGenerated between (_startTime .. _endTime)\n | summarize\n PromptTokensSet = make_set(PromptTokens, 2),\n CompletionTokensSet = make_set(CompletionTokens, 2),\n TotalTokensSet = make_set(TotalTokens, 2)\n by CorrelationId\n | extend\n PromptTokens = iff(array_length(PromptTokensSet) == 1, tolong(PromptTokensSet[0]), long(null)),\n CompletionTokens = iff(array_length(CompletionTokensSet) == 1, tolong(CompletionTokensSet[0]), long(null)),\n TotalTokens = iff(array_length(TotalTokensSet) == 1, tolong(TotalTokensSet[0]), long(null))\n | project CorrelationId, PromptTokens, CompletionTokens, TotalTokens;\nlet gateway =\n ApiManagementGatewayLogs\n | where TimeGenerated between (_startTime .. _endTime)\n | summarize\n ApiIdSet = make_set(ApiId, 2),\n OperationIdSet = make_set(OperationId, 2),\n ProductIdSet = make_set(ProductId, 2)\n by CorrelationId\n | extend\n ApiId = iff(array_length(ApiIdSet) == 1, tostring(ApiIdSet[0]), ''),\n OperationId = iff(array_length(OperationIdSet) == 1, tostring(OperationIdSet[0]), ''),\n ProductId = iff(array_length(ProductIdSet) == 1, tostring(ProductIdSet[0]), ''),\n HasGatewayRecord = true\n | project CorrelationId, ApiId, OperationId, ProductId, HasGatewayRecord;\nlet joined =\n llm\n | join kind=leftouter (gateway) on CorrelationId\n | extend\n HasGatewayRecord = coalesce(HasGatewayRecord, false),\n ApiId = coalesce(ApiId, ''),\n OperationId = coalesce(OperationId, ''),\n ProductId = coalesce(ProductId, '')\n | extend Attribution = case(\n not(HasGatewayRecord),\n 'UNATTRIBUTED — no gateway record for this model call',\n isempty(ApiId) and isempty(OperationId),\n 'UNATTRIBUTED — gateway record carries no API or operation identity',\n isempty(OperationId),\n 'PARTIAL — API known, operation not identified',\n isempty(ProductId),\n 'ATTRIBUTED — API and operation, no product',\n 'ATTRIBUTED — API, operation and product');\nlet totalRequests = toscalar(joined | count);\njoined\n| summarize\n Requests = count(),\n InputTokens = sum(PromptTokens),\n OutputTokens = sum(CompletionTokens),\n TotalTokens = sum(TotalTokens)\n by Attribution\n| extend ShareOfRequestsPct = round(100.0 * Requests / totalRequests, 1)\n| project Attribution, Requests, ShareOfRequestsPct, InputTokens, OutputTokens, TotalTokens\n| order by Requests desc\n", "size": 0, "title": "Attribution gaps", "queryType": 0, "resourceType": "microsoft.operationalinsights/workspaces", "crossComponentResources": [ "{Workspace}" ], "visualization": "table" }, "name": "query-03-attribution-gaps" }, { "type": 1, "content": { "json": "### Failure and retry waste\n\nConsumption spent on requests that did not succeed, and the throttling and server errors that drive client retries." }, "name": "heading-04-retry-and-failure-waste" }, { "type": 3, "content": { "version": "KqlItem/1.0", "query": "// Measured consumption spent on work that did not succeed.\n// Read-only. Runs in your Log Analytics workspace; sends nothing anywhere.\n//\n// WHAT THIS MEASURES, AND WHAT IT DOES NOT.\n//\n// It measures the outcome of every gateway request and the model tokens\n// consumed under each outcome. Tokens consumed by a request that returned\n// 5xx were paid for and thrown away; that is waste, and it is exact.\n//\n// It does NOT claim to have identified retries. Proving that request B is a\n// retry of request A needs client-side correlation that no APIM log\n// carries. What the log does support is the waste retries produce, and the\n// conditions that drive them: 429 throttling and backend 5xx. Those are\n// reported as what they are — pressure and loss — not as a retry count\n// inferred from timing.\n//\n// The distinction matters because a retry count is the kind of number a\n// reader would act on. An inferred one would be a guess wearing a figure's\n// clothing.\n//\n// Anchored on the GATEWAY side: every request belongs in the denominator,\n// including those that never reached a model. Both sides are collapsed to\n// one row per CorrelationId before joining, so the join cannot multiply.\n\nlet _startTime = todatetime('{TimeRange:start}');\nlet _endTime = todatetime('{TimeRange:end}');\nlet llm =\n ApiManagementGatewayLlmLog\n | where TimeGenerated between (_startTime .. _endTime)\n | summarize TotalTokensSet = make_set(TotalTokens, 2) by CorrelationId\n | extend TotalTokens = iff(array_length(TotalTokensSet) == 1, tolong(TotalTokensSet[0]), long(null))\n | project CorrelationId, TotalTokens;\nlet gateway =\n ApiManagementGatewayLogs\n | where TimeGenerated between (_startTime .. _endTime)\n | summarize\n ResponseCodeSet = make_set(ResponseCode, 2),\n ApiIdSet = make_set(ApiId, 2)\n by CorrelationId\n | extend\n // A request logged as both 200 and 500 is a fact about the log, not\n // something to resolve by choosing. It reports as indeterminate.\n ResponseCode = iff(array_length(ResponseCodeSet) == 1, toint(ResponseCodeSet[0]), int(null)),\n ApiId = iff(array_length(ApiIdSet) == 1, tostring(ApiIdSet[0]), '')\n | project CorrelationId, ResponseCode, ApiId;\nlet joined =\n gateway\n | join kind=leftouter (llm) on CorrelationId\n | extend Outcome = case(\n isnull(ResponseCode), 'INDETERMINATE — gateway records disagree on status',\n ResponseCode >= 500, 'SERVER ERROR (5xx) — consumption lost',\n ResponseCode == 429, 'THROTTLED (429) — drives client retry',\n ResponseCode >= 400, 'CLIENT ERROR (4xx)',\n 'SUCCEEDED');\nlet totalRequests = toscalar(joined | count);\nlet totalTokens = toscalar(joined | summarize sum(TotalTokens));\njoined\n| summarize\n Requests = count(),\n RequestsWithUsage = countif(isnotnull(TotalTokens)),\n TokensConsumed = sum(TotalTokens)\n by Outcome\n//\n// KUSTO'S sum() RETURNS 0 FOR A GROUP WHERE EVERY VALUE IS NULL, not null.\n// That turns \"no request in this group reported usage\" into the confident\n// claim \"this group consumed nothing\" — the exact substitution the rest of\n// the kit refuses to make. A bar at zero reads as measured absence of spend;\n// a gap reads as absence of evidence, which is what it is. So a token total\n// resting on zero reporting requests is returned absent.\n//\n// It matters most here. A 429 is throttled before it reaches a model, so its\n// bucket legitimately has no usage — but so does any outcome whose requests\n// predate the LLM diagnostic category being switched on. Reporting 0 for the\n// second case would read as \"these failures cost nothing\", which is the\n// opposite of the finding.\n| extend TokensConsumed = iff(RequestsWithUsage > 0, TokensConsumed, long(null))\n| extend\n ShareOfRequestsPct = round(100.0 * Requests / totalRequests, 1),\n // Share of all measured consumption sitting under this outcome. For\n // every row except SUCCEEDED, this is consumption with nothing to show\n // for it. Absent where the outcome reported no usage at all — a share of\n // an unknown is not zero.\n ShareOfTokensPct = iff(RequestsWithUsage > 0 and totalTokens > 0,\n round(100.0 * TokensConsumed / totalTokens, 1),\n real(null))\n| project Outcome, Requests, ShareOfRequestsPct, RequestsWithUsage, TokensConsumed, ShareOfTokensPct\n| order by TokensConsumed desc, Requests desc\n", "size": 0, "title": "Failure and retry waste", "queryType": 0, "resourceType": "microsoft.operationalinsights/workspaces", "crossComponentResources": [ "{Workspace}" ], "visualization": "table" }, "name": "query-04-retry-and-failure-waste" }, { "type": 1, "content": { "json": "### Validation candidates\n\nOperations ranked by input context paid for relative to output produced. A candidate worth testing — never a conclusion." }, "name": "heading-05-validation-candidates" }, { "type": 3, "content": { "version": "KqlItem/1.0", "query": "// Economic candidates worth testing before anything changes.\n// Read-only. Runs in your Log Analytics workspace; sends nothing anywhere.\n//\n// THIS QUERY RANKS. IT DOES NOT CONCLUDE.\n//\n// It reports, per operation and model, how much input context is being paid\n// for relative to the output produced. A high input-to-output ratio is the\n// signature of context that may not be earning its cost — retrieved\n// documents that go unused, prompts that accumulated, history replayed on\n// every turn.\n//\n// It is a CANDIDATE and nothing more. Whether that context is waste or\n// whether it is the reason the output is correct cannot be established from\n// a log, and no threshold is applied here: Metergrade does not know this\n// estate's quality, latency or reliability requirements, and a query that\n// invented one would be recommending a change it had not tested. Validation\n// against those requirements is what turns a candidate into a verdict, and\n// that step happens deliberately, not in a workbook tile.\n//\n// Ranked by total input tokens, because the largest consumer of context is\n// where testing is worth the effort first — not by ratio alone, which would\n// promote a trivial operation with an extreme ratio above a dominant one.\n\nlet _startTime = todatetime('{TimeRange:start}');\nlet _endTime = todatetime('{TimeRange:end}');\nlet llm =\n ApiManagementGatewayLlmLog\n | where TimeGenerated between (_startTime .. _endTime)\n | summarize\n ModelNameSet = make_set(ModelName, 2),\n PromptTokensSet = make_set(PromptTokens, 2),\n CompletionTokensSet = make_set(CompletionTokens, 2)\n by CorrelationId\n | extend\n ModelName = iff(array_length(ModelNameSet) == 1, tostring(ModelNameSet[0]), ''),\n PromptTokens = iff(array_length(PromptTokensSet) == 1, tolong(PromptTokensSet[0]), long(null)),\n CompletionTokens = iff(array_length(CompletionTokensSet) == 1, tolong(CompletionTokensSet[0]), long(null))\n // Only requests with usable economics can be ranked on economics.\n | where isnotnull(PromptTokens) and isnotnull(CompletionTokens)\n | project CorrelationId, ModelName, PromptTokens, CompletionTokens;\nlet gateway =\n ApiManagementGatewayLogs\n | where TimeGenerated between (_startTime .. _endTime)\n | summarize\n ApiIdSet = make_set(ApiId, 2),\n OperationIdSet = make_set(OperationId, 2)\n by CorrelationId\n | extend\n ApiId = iff(array_length(ApiIdSet) == 1, tostring(ApiIdSet[0]), ''),\n OperationId = iff(array_length(OperationIdSet) == 1, tostring(OperationIdSet[0]), '')\n | project CorrelationId, ApiId, OperationId;\nllm\n| join kind=leftouter (gateway) on CorrelationId\n| extend\n ApiId = iff(isempty(coalesce(ApiId, '')), '(unattributed)', ApiId),\n OperationId = iff(isempty(coalesce(OperationId, '')), '(unattributed)', OperationId),\n ModelName = iff(isempty(ModelName), '(not reported)', ModelName)\n| summarize\n Requests = count(),\n InputTokens = sum(PromptTokens),\n OutputTokens = sum(CompletionTokens),\n MedianInput = percentile(PromptTokens, 50),\n MedianOutput = percentile(CompletionTokens, 50)\n by ApiId, OperationId, ModelName\n// Guarded against divide-by-zero: an operation that produced no output at\n// all has no ratio, and reporting one as infinite would be a fabrication.\n| extend InputPerOutputToken = iff(OutputTokens > 0, round(1.0 * InputTokens / OutputTokens, 1), real(null))\n| project ApiId, OperationId, ModelName, Requests, InputTokens, OutputTokens, InputPerOutputToken, MedianInput, MedianOutput\n| order by InputTokens desc\n| take 50\n", "size": 0, "title": "Validation candidates", "queryType": 0, "resourceType": "microsoft.operationalinsights/workspaces", "crossComponentResources": [ "{Workspace}" ], "visualization": "table" }, "name": "query-05-validation-candidates" }, { "type": 1, "content": { "json": "### Findings summary\n\nOne forwardable page: what this check established, what it could not establish and why, and the ranked candidates." }, "name": "heading-06-findings-summary" }, { "type": 3, "content": { "version": "KqlItem/1.0", "query": "// THE LAST TILE, AND THE ONE THAT LEAVES THE ROOM.\n//\n// Every tile above answers one question well and is useless to anyone who\n// was not looking at the screen. This one is built to be forwarded: a\n// platform owner runs the check, exports this grid, and sends it to the\n// person who has to decide something. It is the whole output of the free\n// surface in a form that survives being read by someone who was not there.\n//\n// WHAT IT DELIBERATELY DOES NOT DO. It states no threshold, attaches no\n// currency to consumption, and calls nothing recoverable — see SCOPE.md.\n// Section 3 is the reason it can be trusted: a summary that reports only\n// what it found, and never what it missed, is a sales document.\n//\n// Read-only. Runs in your Log Analytics workspace; sends nothing anywhere.\n\nlet _startTime = todatetime('{TimeRange:start}');\nlet _endTime = todatetime('{TimeRange:end}');\nlet llm =\n ApiManagementGatewayLlmLog\n | where TimeGenerated between (_startTime .. _endTime)\n | summarize\n TimeGenerated = min(TimeGenerated),\n ModelNameSet = make_set(ModelName, 2),\n PromptTokensSet = make_set(PromptTokens, 2),\n CompletionTokensSet = make_set(CompletionTokens, 2),\n TotalTokensSet = make_set(TotalTokens, 2)\n by CorrelationId\n | extend\n ModelName = iff(array_length(ModelNameSet) == 1, tostring(ModelNameSet[0]), ''),\n PromptTokens = iff(array_length(PromptTokensSet) == 1, tolong(PromptTokensSet[0]), long(null)),\n CompletionTokens = iff(array_length(CompletionTokensSet) == 1, tolong(CompletionTokensSet[0]), long(null)),\n TotalTokens = iff(array_length(TotalTokensSet) == 1, tolong(TotalTokensSet[0]), long(null))\n | project CorrelationId, TimeGenerated, ModelName, PromptTokens, CompletionTokens, TotalTokens;\nlet gateway =\n ApiManagementGatewayLogs\n | where TimeGenerated between (_startTime .. _endTime)\n | summarize\n ApiIdSet = make_set(ApiId, 2),\n OperationIdSet = make_set(OperationId, 2),\n ResponseCodeSet = make_set(ResponseCode, 2)\n by CorrelationId\n | extend\n ApiId = iff(array_length(ApiIdSet) == 1, tostring(ApiIdSet[0]), ''),\n OperationId = iff(array_length(OperationIdSet) == 1, tostring(OperationIdSet[0]), ''),\n ResponseCode = iff(array_length(ResponseCodeSet) == 1, toint(ResponseCodeSet[0]), int(null)),\n HasGateway = true\n | project CorrelationId, ApiId, OperationId, ResponseCode, HasGateway;\nlet joined =\n llm\n | join kind=leftouter (gateway) on CorrelationId\n | extend\n HasGateway = coalesce(HasGateway, false),\n ApiId = coalesce(ApiId, ''),\n OperationId = coalesce(OperationId, '');\nlet requests = toscalar(joined | count);\nlet withUsage = toscalar(joined | where isnotnull(TotalTokens) | count);\nlet unattributed = toscalar(joined | where not(HasGateway) or (isempty(ApiId) and isempty(OperationId)) | count);\nlet partial = toscalar(joined | where HasGateway and isnotempty(ApiId) and isempty(OperationId) | count);\nlet inputTokens = toscalar(joined | summarize sum(PromptTokens));\nlet outputTokens = toscalar(joined | summarize sum(CompletionTokens));\nlet models = toscalar(joined | where isnotempty(ModelName) | summarize dcount(ModelName));\nlet firstSeen = toscalar(joined | summarize min(TimeGenerated));\nlet lastSeen = toscalar(joined | summarize max(TimeGenerated));\nlet failedRequests = toscalar(joined | where ResponseCode >= 500 or ResponseCode == 429 | count);\nlet failedTokens = toscalar(joined | where ResponseCode >= 500 or ResponseCode == 429 | summarize sum(TotalTokens));\nlet pct = (n:long, d:long) { iff(d > 0, strcat(tostring(round(100.0 * n / d, 1)), '%'), 'n/a') };\nlet num = (n:long) { iff(isnull(n), 'not established', tostring(n)) };\nlet candidates =\n joined\n | where isnotnull(PromptTokens) and isnotnull(CompletionTokens)\n | extend\n ApiLabel = iff(isempty(ApiId), '(unattributed)', ApiId),\n OpLabel = iff(isempty(OperationId), '(unattributed)', OperationId),\n Model = iff(isempty(ModelName), '(not reported)', ModelName)\n | summarize Requests = count(), InputTokens = sum(PromptTokens), OutputTokens = sum(CompletionTokens)\n by ApiLabel, OpLabel, Model\n | extend Ratio = iff(OutputTokens > 0, round(1.0 * InputTokens / OutputTokens, 1), real(null))\n | order by InputTokens desc\n | serialize Rank = row_number()\n | where Rank <= 5\n | project\n Order = 400 + Rank,\n Section = '4 · CANDIDATES WORTH TESTING',\n Item = strcat(tostring(Rank), '. ', ApiLabel, ' / ', OpLabel, ' / ', Model),\n Detail = strcat(\n tostring(InputTokens), ' input tokens over ', tostring(Requests),\n iff(Requests == 1, ' request · ', ' requests · '),\n iff(isnull(Ratio), 'no output produced', strcat(tostring(Ratio), ':1 input-to-output')));\nunion\n(print Order = 100, Section = '1 · SCOPE OF THIS CHECK',\n Item = 'Window examined',\n Detail = strcat(format_datetime(_startTime, 'yyyy-MM-dd'), ' to ', format_datetime(_endTime, 'yyyy-MM-dd'))),\n(print Order = 101, Section = '1 · SCOPE OF THIS CHECK',\n Item = 'Traffic actually seen',\n Detail = iff(isnull(firstSeen), 'none in this window',\n strcat(format_datetime(firstSeen, 'yyyy-MM-dd'), ' to ', format_datetime(lastSeen, 'yyyy-MM-dd')))),\n(print Order = 102, Section = '1 · SCOPE OF THIS CHECK',\n Item = 'Source',\n Detail = 'Azure API Management diagnostics in this Log Analytics workspace. Read-only. Nothing transmitted.'),\n(print Order = 200, Section = '2 · WHAT WAS OBSERVED',\n Item = 'Requests', Detail = num(requests)),\n(print Order = 201, Section = '2 · WHAT WAS OBSERVED',\n Item = 'Input tokens', Detail = num(inputTokens)),\n(print Order = 202, Section = '2 · WHAT WAS OBSERVED',\n Item = 'Output tokens', Detail = num(outputTokens)),\n(print Order = 203, Section = '2 · WHAT WAS OBSERVED',\n Item = 'Distinct models in use', Detail = num(models)),\n(print Order = 204, Section = '2 · WHAT WAS OBSERVED',\n Item = 'Requests that did not succeed',\n Detail = strcat(num(failedRequests), ' (', pct(failedRequests, requests), ') consuming ',\n num(failedTokens), ' tokens')),\n(print Order = 300, Section = '3 · WHAT COULD NOT BE ESTABLISHED',\n Item = 'Requests without usable token data',\n Detail = strcat(num(requests - withUsage), ' (', pct(requests - withUsage, requests),\n ') — no usage reported, or records disagreed. Token totals above are a floor, not an estimate.')),\n(print Order = 301, Section = '3 · WHAT COULD NOT BE ESTABLISHED',\n Item = 'Consumption with no accountable workload',\n Detail = strcat(num(unattributed), ' (', pct(unattributed, requests),\n ') — no API or operation identity, so this cannot be assigned to an owner.')),\n(print Order = 302, Section = '3 · WHAT COULD NOT BE ESTABLISHED',\n Item = 'Partially attributed',\n Detail = strcat(num(partial), ' (', pct(partial, requests), ') — API known, operation not.')),\n(print Order = 303, Section = '3 · WHAT COULD NOT BE ESTABLISHED',\n Item = 'What this check does not determine',\n Detail = 'What any of it costs, whether any figure is high or low, and whether a candidate is worth acting on. Those require validation against your own requirements.'),\ncandidates\n| order by Order asc\n| project Section, Item, Detail\n", "size": 0, "title": "Findings summary", "queryType": 0, "resourceType": "microsoft.operationalinsights/workspaces", "crossComponentResources": [ "{Workspace}" ], "visualization": "table" }, "name": "query-06-findings-summary" }, { "type": 1, "content": { "json": "## What happens next\n\nThis check established what your estate is doing. It did not establish what any of it costs, whether any figure is high or low for you, or whether a candidate is worth acting on. Those are the same question, and answering it is called validation.\n\n**Export the findings summary above** — it is written to be forwarded. It carries what was observed, what could not be established and why, and the candidates in order.\n\n### What validating one candidate involves\n\nTake the top candidate. Establish what it costs today at your own negotiated rates, which the check deliberately does not do. Define what \"still correct\" means for that operation — quality, latency, reliability — in terms your team would actually defend. Run the alternative against real traffic and compare it against those requirements, not against a cheaper bill. The result is one of three things: it holds, it does not, or the evidence is not sufficient to say. The third outcome is common and is a real answer.\n\nThat is a piece of work, not a button. It is worth being clear about the size of it before anyone commits to it.\n\n### If you want someone to read this with you\n\nMetergrade will go through your output with you, on your screen, in your own environment, with nothing transferred — and tell you honestly whether a full assessment is warranted at all. Ask at **hello@metergrade.com** and attach the summary above.\n\nThe **Assisted Assessment** is the assisted path once you know you want it: a fixed engagement in which Metergrade establishes the baseline formally with your team and returns an executive evidence package. Details and price: metergrade.com/pricing\n\n---\n\nNothing in this workbook was transmitted to Metergrade. Verify that this kit is what Metergrade published: metergrade.com/docs/verifying-the-kit" }, "name": "what-happens-next" } ], "styleSettings": {}, "fallbackResourceIds": [] }