Microsoft is enhancing tools that users and developers utilize to record on-screen work sessions and parlay those recorded actions into skills that can be used to teach Microsoft Scout, Copilot Cowork, or Copilot Studio how to perform a task.
MVPs and AI experts say the Skill Recorder technology points to a future when process documentation can be built automatically, and therefore far more efficiently, by observing and capturing work in real time, rather than documenting it after the fact.
“By capturing interactions as people complete real work, organisations can build a living library of how processes are actually executed. That creates the foundation for AI to understand workflows, identify repetition, recommend improvements and ultimately automate them,” Microsoft expert and AI Transformation Consultant Mark Stephens wrote on LinkedIn.
Common tasks a user or developer may automate with Skill Recorder include teaching an agent a repetitive task by recording once and reusing infinitely; scheduling a workflow that runs after a process recording is turned into a timed automation; and running skills locally on a system with zero GPU footprint required, according to Sudhanshu Sharma, AI lead at E.ON Germany, in another LinkedIn post.
As documented by Microsoft in its initial release, Skill Recorder captures an on-screen work session and uses the GitHub Copilot Command Line Interface, or CLI, to reconstruct that session as an intent and a sequential set of steps. It then builds a reusable Skill (for on-demand use) or Automation (for use on a schedule or in response to a trigger) for Scout, Copilot Cowork, or Copilot Studio.
Skill Recorder captures apps in use, click activity, window switches, pages visited, and (when chosen by the user) spoken narration. It then uses the GitHub Copilot CLI to reconstruct the user’s steps.
In using Recorder, a user or developer moves through the following steps:
- Record. Skill Recorder captures a screen and activity locally, in the background.
- Control. In the recording process, a top bar shows capture and microphone state. Users can mute, unmute, or switch mics on the fly, then finish the recording if successful, or discard the recording if things don’t play out as planned.
- Analyze. GitHub Copilot reconstructs the overall intent and an ordered list of steps. Users review and edit those steps to ensure accuracy.
- Create. From an approved analysis, Recorder generates a reusable skill and/or a scheduled automation.
Recording, storage, frame extraction, and optional narration transcription all happen on an individual computer; nothing leaves that system during the recording process. Only when the user chooses the Analyze function does Skill Recorder send the event timeline (window/document titles, URLs, and clipboard previews), extracted screen images, and narration text to GitHub for Copilot to process.
Microsoft urges users to not record, type, or narrate passwords, access tokens, API keys, credentials, or other confidential information.
The latest release of Skill Recorder, v0.4.2 issued last week, provides a hotfix for installation failures on machines whose Node Package Manager, or npm registry (npm is the default package manager) is configured globally, such as with corporate-managed devices behind a proxy. The provided installers point to a new portable runtime so that a configured mirror is discovered automatically.
The installers discover the registry configured on the machine. No data migration is required for the latest release and functionality.
More Insights on AI Agent Skills:
- With Agent 365 Skills, Microsoft Accelerates Rollout of Governed Enterprise Agents
- Skills in Dataverse Capture Business Process Details to Enable AI Automation
Community Summit North America is the largest independent innovation, education, and training event for Microsoft business applications delivered by Expert Users, Microsoft Leaders, MVPs, and Partners. Register now to attend Community Summit in Nashville, TN from October 11-15.
Tom Smith
Analyst
Cloud Wars, Agent and Copilot
Related Posts
