Live Device Mirror — RoboClaw AI Studio

1What the Mirror Does

Live Device Mirror gives you a continuous JPEG stream of your target device or browser, rendered inside a VS Code webview. Unlike a static screenshot, it lets you:

Interact — tap, swipe, type — and your actions are forwarded straight to Appium / Selenium.
Inspect — hover an element and the inspector overlays the bounding box, XPath, predicate, class-chain, and accessibility ID.
Capture — grab the current frame into your test workspace without disrupting the stream.
Generate locators — feed the current page source into the locator engine in one click.

⚙️

How it works. A long-lived Python helper (live_capture.py) holds the Appium / Selenium session, fetches frames at the configured FPS, and pushes them over an NDJSON stdio channel to the extension. No HTTP layer, no per-frame Python startup tax.

2Prerequisites

RoboClaw installed and Python environment provisioned.
A configured session — run RoboClaw: Configure Session first.
For mobile: Appium 2.x running and a device/simulator connected.
For web: ChromeDriver/Geckodriver available on PATH (auto-detected).

3Start the Mirror

Open the RoboClaw sidebar.
In Step 1: Setup & Settings, confirm the connection pill says ● Connected.
Click ▶ Start Live Device Mirror (or run RoboClaw: Start Live Device Mirror).

The stream appears inline in the sidebar. To stop:

⌘⇧PRoboClaw: Stop Live Device Mirror

🔌

Reconnect dropped sessions. If your device cable hiccups, run RoboClaw: Reconnect Last Session — RoboClaw restores the exact same capabilities.

4Controls Cheat-Sheet

Action	How
Tap	Single click on the live frame
Long-press	Click and hold for > 700 ms
Swipe	Click + drag (start → end)
Type text	Click target field, then start typing in the focused webview
Hardware key	Buttons row above the stream (Home / Back / Volume / Enter)
Element inspect	Hover with `Alt` held
Capture frame	Click 📸 in the stream toolbar
Pause stream	Click ⏸ in the stream toolbar

5Tap, Swipe & Type

Coordinates are auto-scaled to the device's native resolution — what you click in the webview maps 1-to-1 to where Appium / Selenium taps.

Multi-step interaction

For complex flows, queue up actions and replay them as a sequence:

Click Record in the stream toolbar.
Perform your gestures (each is logged in the Interaction Log panel).
Click Stop → RoboClaw emits a Robot Framework snippet you can paste into your test.

# Recorded interaction (paste into your .robot)
Tap Coordinates      412  680
Wait Until Element Is Visible  ${SHIFT_HEADER}  5s
Swipe                412  680  412  240  800
Input Text           ${USERNAME_FIELD}  supervisor.lead

6Element Inspector

Hold Alt and move the cursor over the stream. RoboClaw overlays a bounding box on the nearest element and pops a detail panel:

XPath (deterministic, generated by the locator engine)
iOS Predicate / Class Chain (iOS only)
Accessibility ID / Resource ID
CSS Selector (web only)
All raw attributes with confidence scores per locator strategy
Copy buttons for every locator strategy

🎯

Confidence-scored alternatives. Even a single element exposes 4–6 locator strategies ranked by stability score — pick the one your team prefers without ever leaving the webview.

7Capture from the Live Stream

Capture in Live mode is non-disruptive — the stream keeps running.

Click 📸 in the stream toolbar (or run RoboClaw: Capture Current Screen).
The current frame is saved as a PNG, plus the matching XML page source is fetched at the same instant.
Files land under Robogent-Input/<TC>/Screens/.

Need a burst or a full-page scroll capture?

RoboClaw: Burst Capture (2s interval) — auto-capture every N seconds.
RoboClaw: Scroll & Capture (Full Page) — stitches a long-page screenshot via incremental scroll.

8Pop-out Panel

For more screen real-estate, pop the mirror out beside your editor:

Click the ⛶ icon in the stream toolbar,
or run RoboClaw: Open Live Device Panel.
A new editor-area webview opens — drag it to a split, snap it to a side, dock it however you like.
The Interaction Log and Element Inspector get full-height columns in the pop-out.

🖥️

Both views stay in sync. The sidebar mini-view and the pop-out panel share the same underlying stream — close one, the other keeps streaming.

9FPS / Quality / Width Tuning

The stream toolbar exposes three sliders. Settings persist across reloads (per-workspace).

Setting	Range	When to bump it
FPS	1 – 30	30 for video-heavy flows; 5–8 for typical UI; 1–2 for slow networks
Quality	30 – 95 (JPEG)	50 default; bump to 85 only when reading small text
Max width	320 – 1200 px	Lower = less bandwidth; keep ≥ 640 to stay touch-precise

You can also pre-seed the sliders via RoboClaw: Live Device: Set FPS from the Command Palette.

📶

Remote / VPN devices? Drop FPS to 3–5 and quality to 40 — RoboClaw will gracefully degrade and you'll get smooth interaction with minimal lag.

10Troubleshooting

"Stream blank / black"

Confirm the session pill shows ● Connected.
For iOS Real Device: WebDriverAgent must be installed and trusted. xcrun simctl list devices for simulator.
Some Android apps block screen capture (banking apps). Use a debug build.

"Taps register in the wrong place"

Most often this is iOS with mismatched logical/physical resolution. Make sure your platformVersion matches the actual device — RoboClaw applies the correct scale factor based on it.

"Stream lags but device is responsive"

Drop FPS to 5 and width to 720. Confirm your Mac/PC isn't on a heavy CPU load (parallel emulator + simulator is the most common culprit).

Resetting the mirror state

⌘⇧PRoboClaw: Reset Live Device Mirror

Restarts the Python helper, clears cached frames, and reconnects with current capabilities.