Scans a Windows screen into a numbered element table so an agent clicks and types by id instead of guessing pixels.
Screenshots are a poor interface for an agent that has to click something. It guesses coordinates, misses by a few pixels, and has no idea whether the button it hit was the real one. The XuLan ScreenPilot MCP server takes a different route on Windows: it scans the screen once, numbers every element it can find, and lets the agent act by element id.
Each scan runs three engines side by side and merges their results, with accessibility data taking priority over text recognition, and text recognition over shape detection.
| Engine | What it reads | Best for |
|---|---|---|
| UIA | The Windows accessibility tree | Native apps and browsers |
| OCR | RapidOCR on onnxruntime, CPU only | Games and custom-drawn interfaces |
| Visual | OpenCV shape detection | Icons, checkboxes, radio buttons, sliders, close buttons |
Overlapping hits are de-duplicated and numbered top-to-bottom. A rescan keeps the same element on the same id, which matters when an agent is several steps into a task. An annotated screenshot is written alongside the table, colour-coded by engine.
The ScreenPilot MCP server exposes 20 tools over stdio. The everyday loop is short:
scan_screen to get the table, optionally scoped to one windowclick_element and type_text to act on an idclick_safe to act only when expected text appears or disappears afterwardswatch_burst and verify_typing to catch toasts and confirm typed textcleanup_windows to close windows the agent itself openedThe cleanup step is cautious by design. It sends a polite close request only, never kills processes, skips windows with unsaved work and leaves Explorer and Task Manager alone. Clicks also report which top-level window they actually landed on, and can refuse a target that is covered by something else.
Windows 10 or 11 and Python 3.11 or newer are required; everything runs on the CPU, with no cloud model and no API key. You clone the project, install its Python dependencies with pip (uiautomation, rapidocr-onnxruntime, opencv-python, the mcp package and a few others), then point your client at mcp_server.py with python. Setting PYTHONIOENCODING to utf-8 in the server environment is part of the suggested config.
The optional element book, which finds familiar UI by name or image without a full rescan, needs SigLIP ONNX weights in a folder named by SCREENPILOT_MODELS_DIR.
Dark professional apps with small text hurt OCR recall, and dense 4K toolbars are hard for the matcher. Fuller internals are documented in Chinese only. Clicks are deliberately jittered to look human unless precise=True is passed. For other desktop and system bridges, browse our other integrations directory; if you are comparing Windows automation options, the same integrations category lists the rest of what we track.
Check the linked publisher documentation and your connected client's tool list. Names and availability can vary with the server version and account permissions.
| Tool | What it does |
|---|---|
| scan_screen | Scan the screen or a named window and return the numbered element table. |
| scan_image | Run the same detection engines offline on a static image file. |
| click_element | Click an element by id, at control level through UIA when possible. |
| click_safe | Click an element and treat it as successful only when expected evidence appears or disappears. |
| click_xy | Click a coordinate and report which top-level window was actually hit. |
| type_text | Type text into an element by id. |
| press_key | Send a key press. |
| scroll_screen | Scroll the screen. |
| snap_screen | Capture a screenshot of the screen. |
| crop_screen | Crop an element at native resolution to read fine text or icons; supports force_refresh. |
| crop_xy | Crop a coordinate region at native resolution; supports force_refresh. |
| book_find | Look up recurring UI in the persistent element book by text or vector search without a full rescan. |
| watch_burst | Watch a window for a few seconds to catch toasts and other transient feedback. |
| verify_typing | Confirm that typed text actually shows on screen. |
| probe_hang | Probe whether a window has stopped responding. |
| split_static_dynamic | Separate the static UI layer from moving content across frames. |
| window_stack | Report window stacking, including windows flagged as off screen. |
| activate_window | Bring a window to the foreground. |
| register_window | Record a window the agent launched so it can be cleaned up later. |
| cleanup_windows | Gently close windows recorded in this session's ledger, with safety checks and no process killing. |
{
"mcpServers": {
"screenpilot": {
"command": "python",
"args": [
"<path>/screenpilot/mcp_server.py"
],
"env": {
"PYTHONIOENCODING": "utf-8"
}
}
}
}Replace <path> with the folder you cloned the project into. Install the Python dependencies with pip first. Windows 10/11 only.
| Variable | Description | Required |
|---|---|---|
| PYTHONIOENCODING | Set to utf-8 in the suggested MCP config so non-ASCII window text passes through cleanly. | Optional |
| SCREENPILOT_MODELS_DIR | Folder holding the SigLIP int8 ONNX models used by the optional element-book vector search and semantic matcher. | Optional |
| SCREENPILOT_DATASETS_DIR | Location of datasets for the public benchmark scripts. | Optional |
Run an Actual Budget ledger from your AI agent, with phone approvals for every change on paired machines.
An Odoo module that serves a read-only MCP endpoint at /mcp, scoped to each user's own access rights.
Agent memory that keeps every correction with its history, plus saved working state so a crashed run can resume.
Drives DaVinci Resolve 21.1, free edition included, from Claude Code through a Lua bridge running inside the app.
Lets an AI assistant upload files, images and text to Pixeldrain and download them back, with optional client-side encryption.
Open-source screen recorder and editor whose MCP server lets an agent record, edit and export a demo video.