SSyncDev

XuLan ScreenPilot MCP Server

Scans a Windows screen into a numbered element table so an agent clicks and types by id instead of guessing pixels.

Local server

What is the XuLan ScreenPilot MCP Server?

Screenshots are a poor interface for an agent that has to click something. It guesses coordinates, misses by a few pixels, and has no idea whether the button it hit was the real one. The XuLan ScreenPilot MCP server takes a different route on Windows: it scans the screen once, numbers every element it can find, and lets the agent act by element id.

Three detectors, one table

Each scan runs three engines side by side and merges their results, with accessibility data taking priority over text recognition, and text recognition over shape detection.

Engine What it reads Best for
UIA The Windows accessibility tree Native apps and browsers
OCR RapidOCR on onnxruntime, CPU only Games and custom-drawn interfaces
Visual OpenCV shape detection Icons, checkboxes, radio buttons, sliders, close buttons

Overlapping hits are de-duplicated and numbered top-to-bottom. A rescan keeps the same element on the same id, which matters when an agent is several steps into a task. An annotated screenshot is written alongside the table, colour-coded by engine.

What an agent can do with it

The ScreenPilot MCP server exposes 20 tools over stdio. The everyday loop is short:

  • scan_screen to get the table, optionally scoped to one window
  • click_element and type_text to act on an id
  • click_safe to act only when expected text appears or disappears afterwards
  • watch_burst and verify_typing to catch toasts and confirm typed text
  • cleanup_windows to close windows the agent itself opened

The cleanup step is cautious by design. It sends a polite close request only, never kills processes, skips windows with unsaved work and leaves Explorer and Task Manager alone. Clicks also report which top-level window they actually landed on, and can refuse a target that is covered by something else.

Setup and requirements

Windows 10 or 11 and Python 3.11 or newer are required; everything runs on the CPU, with no cloud model and no API key. You clone the project, install its Python dependencies with pip (uiautomation, rapidocr-onnxruntime, opencv-python, the mcp package and a few others), then point your client at mcp_server.py with python. Setting PYTHONIOENCODING to utf-8 in the server environment is part of the suggested config.

The optional element book, which finds familiar UI by name or image without a full rescan, needs SigLIP ONNX weights in a folder named by SCREENPILOT_MODELS_DIR.

Honest limits

Dark professional apps with small text hurt OCR recall, and dense 4K toolbars are hard for the matcher. Fuller internals are documented in Chinese only. Clicks are deliberately jittered to look human unless precise=True is passed. For other desktop and system bridges, browse our other integrations directory; if you are comparing Windows automation options, the same integrations category lists the rest of what we track.

Recorded tools

Check the linked publisher documentation and your connected client's tool list. Names and availability can vary with the server version and account permissions.

ToolWhat it does
scan_screenScan the screen or a named window and return the numbered element table.
scan_imageRun the same detection engines offline on a static image file.
click_elementClick an element by id, at control level through UIA when possible.
click_safeClick an element and treat it as successful only when expected evidence appears or disappears.
click_xyClick a coordinate and report which top-level window was actually hit.
type_textType text into an element by id.
press_keySend a key press.
scroll_screenScroll the screen.
snap_screenCapture a screenshot of the screen.
crop_screenCrop an element at native resolution to read fine text or icons; supports force_refresh.
crop_xyCrop a coordinate region at native resolution; supports force_refresh.
book_findLook up recurring UI in the persistent element book by text or vector search without a full rescan.
watch_burstWatch a window for a few seconds to catch toasts and other transient feedback.
verify_typingConfirm that typed text actually shows on screen.
probe_hangProbe whether a window has stopped responding.
split_static_dynamicSeparate the static UI layer from moving content across frames.
window_stackReport window stacking, including windows flagged as off screen.
activate_windowBring a window to the foreground.
register_windowRecord a window the agent launched so it can be cleaned up later.
cleanup_windowsGently close windows recorded in this session's ledger, with safety checks and no process killing.

How to install the XuLan ScreenPilot MCP Server

{
  "mcpServers": {
    "screenpilot": {
      "command": "python",
      "args": [
        "<path>/screenpilot/mcp_server.py"
      ],
      "env": {
        "PYTHONIOENCODING": "utf-8"
      }
    }
  }
}

Replace <path> with the folder you cloned the project into. Install the Python dependencies with pip first. Windows 10/11 only.

Configuration

VariableDescriptionRequired
PYTHONIOENCODINGSet to utf-8 in the suggested MCP config so non-ASCII window text passes through cleanly.Optional
SCREENPILOT_MODELS_DIRFolder holding the SigLIP int8 ONNX models used by the optional element-book vector search and semantic matcher.Optional
SCREENPILOT_DATASETS_DIRLocation of datasets for the public benchmark scripts.Optional

Frequently asked questions

No. All three detection engines run locally on the CPU, and no API key is involved. Only the optional element-book vector search needs extra model files on disk.