Live Oversight of AI Agents in Office 365

What 'live' would actually have to mean

Live oversight means seeing an action as it happens, or within a few seconds of it, not finding out about it later from a report. That bar matters because Office 365's own admin tools are built around the second thing, not the first.

Keep that distinction in mind as the next two sections go through what Microsoft actually ships. Neither one clears the live bar, and that is by design, not an oversight.

• Live: you see the action within seconds, while it is happening.
• Not live: you find out from a report or a log entry, after the fact.

What the Copilot usage reports actually show

The Microsoft Copilot Chat usage report and the separate Microsoft Copilot usage report, both in the Microsoft 365 admin center, give you adoption metrics: total and daily active users, total prompts submitted, and per-user activity across 7, 30, 90 or 180 day views. That is useful for tracking rollout, not for watching one agent work.

The admin center's Copilot controls govern policy, such as whether users can create agents or whether web search grounding is allowed. They are a settings panel, not a monitor, and neither the usage reports nor the controls update in anything close to real time.

Microsoft 365 Copilot Premium also ships what Microsoft calls Microsoft Agents, scoped or focused versions of Copilot that act as assistants and automate business processes using web and organizational data through Microsoft Graph and a layer called Work IQ. Those agents are still covered by the same usage reports and the same audit log described below. There is no separate live feed for them either.

• Usage reports are aggregated over day-long or week-long windows, never a live stream.
• Copilot controls set policy; they do not show you an agent acting right now.

The audit log is not live either

The Microsoft Purview unified audit log is the closest thing to a record of a specific action, and Microsoft is explicit that it is not instant. Its own documentation states that Microsoft doesn't guarantee a specific time after an event occurs for the corresponding audit record to appear in search results, and that for core services such as Exchange, SharePoint, OneDrive and Teams, audit record availability is typically 60 to 90 minutes after the event.

Broad searches are slower still. In a large tenant, a search job that is not narrowly scoped can take up to 48 hours to complete. Retention is capped too: 180 days by default, extended to one year for Microsoft 365 E5 or a Purview Audit add-on, so this is a record for compliance review, not a dashboard.

• Typical latency for core services: 60 to 90 minutes, with no guaranteed upper bound.
• Broad search jobs in large tenants: up to 48 hours to complete.
• Retention: 180 days by default, up to one year with the right license.

The case most of these queries are actually about

A developer or admin often hands a coding agent a Microsoft Graph credential, through an MCP server or a custom integration, so the agent can read or write Outlook, SharePoint or Teams data as part of its job. That agent is not running inside Microsoft's cloud. It is running on the developer's own Mac, and Microsoft's reports and audit log were never built to show you what a process on someone else's machine is doing moment to moment.

For that setup, the only place live oversight can come from is the machine the agent is actually running on.

• A Graph-connected coding agent runs locally, outside anything Office 365's own tools observe in real time.
• Watching it live means watching the process, not waiting on Microsoft's logs.

What live oversight of the agent itself looks like

Forkbench is a desktop app for the Mac built around this gap. Each terminal running an agent carries a live pulse that quickens as the agent works harder, driven by CPU use and output rate, never by a token count. A red needs-you flag appears with a count the moment an agent is blocked and waiting on you, and the task it claimed sits beside that terminal so you know what it is doing without reading the whole transcript.

This is oversight of the process on your Mac. It is not a window into Office 365 itself. Forkbench cannot see what a Graph credential actually touched inside Microsoft's services; that is still only visible, with the delay described above, in Purview's own audit log.

Know the other limits too. An unpinned Vault key can still be read by the program it was handed to, so scoping the Graph credential narrowly still matters. If you relay a blocked agent's question to your phone through Talk, that text passes through Forkbench's own servers and is not end to end encrypted.

• The pulse is driven by CPU and output rate, not a token meter.
• The needs-you flag surfaces a blocked agent immediately, with a count.
• Forkbench supervises the Mac-side process, not anything inside Office 365's cloud.
• Talk passes through Forkbench's servers and is not end to end encrypted.

Putting both halves together

Use Microsoft's audit log for what it is good at: proving after the fact what an account or an agent touched inside Office 365, for compliance or an incident review. Do not expect it to tell you about a problem while it is happening.

For that, you supervise the agent process where it actually runs. And because neither tool stops a credential from acting once it has been handed out, keep the Graph permission scoped to only what that agent's job needs, on top of whatever live view you have of the process itself.

This is not a reason to skip Purview. An audit log you only check after an incident is still far better than none, and for a regulated organization it may be a requirement, not a choice. It is a reason to stop expecting it to replace watching the agent while it works.

• Audit log: for proof after the fact, not for catching a problem live.
• Process supervision: for catching a problem while it is happening.
• Scope the Graph credential narrowly either way; neither tool limits what it can do once granted.