How a RAG-enhanced, 7-agent AI system built inside VS Code eliminated test script hallucinations, automated locator generation from live devices, and scaled Frito-Lay's MFG Pro+ quality assurance to 1,886+ Robot Framework test scripts across 3 manufacturing plants and 2 continents.
A strategic overview of how AI-driven test automation transformed quality assurance for a mission-critical manufacturing platform.
RoboClaw AI Studio is an enterprise-grade, AI-powered test automation platform built as a VS Code extension that generates production-ready Robot Framework test scripts from test case specifications, live device screenshots, and Appium page source XML — all grounded by Retrieval-Augmented Generation (RAG) to eliminate AI hallucinations.
Deployed for PepsiCo/Frito-Lay's MFG Pro+ — a dual-platform (iOS Mobile + Web) manufacturing execution system used by frontline operators across plants in Topeka, Beloit, and Jonesboro — the system scaled from zero automation to 1,886 test scripts covering 34 functional modules, with a 7-agent AI architecture that orchestrates code generation, code review, self-healing, screenshot annotation, and pre-generation analysis in a unified workflow.
Why traditional test automation approaches couldn't keep pace with a rapidly evolving manufacturing platform.
MFG Pro+ evolved across 34 functional modules simultaneously — Confirm Shift, Digital Quality, Production Schedule, Forms Execution, Bag Certification, KPI, Reports, and more — with bi-weekly releases that constantly changed UI elements, XPaths, and workflows.
Three North American plants (Topeka, Beloit, Jonesboro) and European sites (SKELM, SRODA) each with unique line configurations (BCP1, FCC1, FCP1, TC1-TC3, PC1-PC2), product codes, and shift structures requiring site-specific test data.
Mobile app (iPad/iPhone via Appium XCUITest) for frontline operators + Web app (Chrome/Selenium) for management. Different locator strategies, navigation patterns, and interaction models requiring distinct automation approaches per platform.
10+ personas — Processing Operator (PO), Packaging Management Operator (PMO), General User (GU), Supply Chain Leader (SCL), Quality Technician (QT), Department Manager (DM) — each with unique permissions, workflows, and form sections.
Lab testing, PAE process, Weak Link, and Food Safety modules with color-coded metrics (Red/Yellow/Green specification ranges), real-time calculations, and compliance tracking requiring pixel-precise validation across 11 production lines × 5 categories.
Generic AI code generators invented non-existent keywords, fabricated XPath locators, assumed incorrect library imports, and produced test scripts that looked correct but failed at runtime — creating a false sense of coverage.
"The fundamental problem wasn't generating some test code — it was generating correct test code that actually runs against a real manufacturing app without inventing elements that don't exist." — Project Engineering Team
A Zero-Rewrites VS Code extension orchestrating 20+ Python scripts through a 7-agent AI system with RAG-enhanced generation and self-healing locators.
The VS Code extension acts purely as an orchestration layer — it spawns existing Python scripts (fetch_page_source.py, locator_engine_advanced.py, screenshot_annotator.py, ado_robogent_bridge.py) as subprocesses via a TypeScript Python Bridge. Zero logic was rewritten in TypeScript. This meant the entire AI/ML pipeline could evolve independently while the extension provided the UX wrapper.
Every test script generation passes through five sequential validation loops, each designed to catch a different class of errors before code reaches the developer.
file_search,
variable values read from actual Config.robot — not assumed. A strict decision matrix prevents common hallucination
patterns: inventing Click On Element With Text instead of using existing Click Mobile Element With Wait.
${UPPER_SNAKE_CASE} with feature prefixes like
CS_, MS_, AL_).
recommended_xpath as the primary locator, with accessibility_id,
ios_predicate, and ios_class_chain as fallback strategies. Variable naming follows
strict conventions: ${REPORT_BUG_MENU_ICON}, ${REPORT_AN_ISSUE_PAGE}.
Every XPath comes from machine-generated JSON — never manually parsed from raw XML.
Seven specialized AI agents, each with a defined role, collaborate through a dispatch model where the master orchestrator delegates tasks to purpose-built sub-agents.
A full-featured VS Code extension serving as the developer-facing interface for the entire AI automation platform.
8 keyboard shortcuts mapped to Cmd+Shift+R combos (Mac) / Ctrl+Shift+R (Windows):
Configure, ADO Import, Screen Capture, Burst,
Locators, ANnotate, Generate Script, Pipeline.
From ADO test case to running Robot Framework script in 6 guided steps.
The pipeline begins with Appium session configuration (UDID, bundle ID, platform) and test case import.
The ADO bridge (ado_robogent_bridge.py) fetches test cases directly from Azure DevOps,
parsing reproduction steps into structured step objects. Alternative inputs: CSV files, Figma API/upload.
fetch_page_source.py connects to the running Appium session and simultaneously captures
a PNG screenshot and the Appium XML page source from the connected iPad/iPhone.
Burst mode captures at 2-second intervals for dynamic screens; scroll-capture handles long scrollable views.
locator_engine_advanced.py analyzes the XML DOM to generate all XPath variants (static, dynamic,
qualified, indexed) plus iOS-specific strategies (accessibility_id, ios_predicate, ios_class_chain) with confidence scoring.
screenshot_annotator.py then produces annotated images and unified TOON reports.
The @RoboClaw agent executes the 5-loop Processing Engine: retrieves workspace patterns (RAG), validates references (anti-hallucination), maximizes keyword reuse, maps locators to variables, and applies self-learning. Outputs: Main .robot (DataDriver + Template), PageRobots .robot (keywords + variables), DataDriver .xlsx, and optional Python libraries.
Five breakthrough capabilities that differentiate RoboClaw from conventional test automation tools.
A single-file innovation that combines visual annotation data (element types, bounding boxes, color codes)
with complete locator data (all XPath variants, ios_predicate, ios_class_chain) in one .toon file.
Previously, generating locators required running locator_engine_advanced.py separately, producing
a second JSON file. The TOON format runs both tools simultaneously via screenshot_annotator.py --report,
reducing pipeline steps, token consumption, and eliminating file synchronization issues.
5-section structure: metadata → color_legend → category_summary → elements (with inline locators) → full locators section.
self_healing_locator.py (1,999 lines) implements 10 distinct healing strategies that activate
when a locator fails at runtime:
Each healed locator receives a 0.0–1.0 confidence score. Results feed back to the Self-Learning loop, continuously improving future locator selection.
Zero hallucinations by design. A lightweight JSON RAG system with vector similarity matching ensures every generated keyword, variable, import path, and XPath locator is grounded in existing workspace code.
Anti-hallucination decision matrix:
| Scenario | Hallucination ✗ | RAG-Grounded ✓ |
|---|---|---|
| Need click keyword | Click On Element With Text | Click Mobile Element With Wait |
| Need XPath | Guess from memory | Read _locators.json → recommended_xpath |
| Need timeout | ${MY_TIMEOUT} 10s | Read Config.robot → ${normalWait} |
| Need setup | Write inline Open Application | Use Test Setup Activities for Mobile App |
Standard Appium XML exports contain 155+ visible elements per screen. The step_xml_mapper.py
maps each test step to the 3–10 most relevant elements, achieving an 86–92% element reduction.
This dramatically improves variable mapping accuracy by eliminating noise from container elements, hidden views,
and unrelated UI components. The step-filtered JSON (_step_filtered.json) is the preferred input format
for the @RoboClaw generation pipeline.
The extension's Builder mode creates a workflow mapping JSON that links each test step to its
corresponding screenshot, locator set, and expected screen state. This mapping (_workflow_mapping.json)
drives multi-screen test flows where different steps operate on different screens, ensuring the correct locators
are used for each step. Combined with Figma design import (API + upload), the system supports
test generation from design files before the UI is even implemented.
A comprehensive automation framework spanning two continents, five manufacturing sites, and every major functional module of the MFG Pro+ platform.
| Category | Modules |
|---|---|
| Shift Management | ConfirmShift, ShiftPlanner, ShiftAndActivities, WF_SelfAssignment, LeaderSelfAssignment |
| Forms & Execution | AllForms, FormsExecutionAllTypes (Regular, Collaborative, Comparative, Conditional, Preventive Control, CrossHour, CrossShift), FormsOrSectionBuilder |
| Task & Workflow | TSA (AddActivity, AddLines, AddRolesWorkflow), CreateTask, TaskLibrary, WorkFlowBuilder, WorkFlowLibrary |
| Reports & KPI | EndOfShiftReports (FMD, PFMS, PSM, SMS), ReportDownload, KPI, PCQI |
| Production | ProductionSchedule (LineDetailsTab), ProductOverride, BagCertification, Breaker |
| Communication | Announcements, Notifications, Review, TakeOver, ReportBug |
| Libraries | FormLibrary, SectionLibrary, OtherAppsLinks, CompletedForms |
| Auth & Common | Login (MfgProApp, MfgProWeb), Common_Non_DQ, Approval |
Process Analytical Engine — production metrics and specification compliance
Laboratory testing — sample collection, result entry, specification ranges
Production bottleneck tracking — line performance analysis
Food safety compliance — HACCP, temperature, allergen controls
Potato-specific quality metrics — moisture, defects, color grading
| Region | Site | Production Lines |
|---|---|---|
| North America (PFNA) | Topeka, KS | TC1, TC2, TC3, TCS1, PC1, PC2, SUN1 |
| Beloit, WI | BCP1 | |
| Jonesboro, AR | FCC1, FCP1, FUN1 | |
| Europe (EU) | SKELM | Packaging, Processing, CIL, Centerline |
| SRODA | Packaging, Processing, Defect Handling |
Measurable outcomes across engineering productivity, test reliability, and operational efficiency.
The complete technology ecosystem powering the RoboClaw AI Studio platform.
| Layer | Technology | Purpose |
|---|---|---|
| IDE Platform | VS Code Extension API (v1.85+) | Extension hosting, webview panels, command palette, settings, keybindings |
| Extension Core | JavaScript (ES2022) | Command handlers, Python Bridge, sidebar provider, service layer |
| AI Agents | GitHub Copilot Chat (Claude) + Custom Agent Definitions | 7 specialized agents with RAG-enhanced prompts, tool access, sub-agent dispatch |
| Test Framework | Robot Framework 7.x + DataDriver | Test execution engine with data-driven pattern, Test Template architecture |
| Mobile Automation | Appium 2.0+ (XCUITest / UiAutomator2) | iOS + Android device automation, page source capture, element interaction |
| Web Automation | Selenium (SeleniumLibrary) | Chrome/Edge browser automation for MFG Pro+ Web App |
| API Testing | RequestsLibrary + GetBearerToken.py | REST API validation with OAuth2 authentication |
| AI/ML Pipeline | Python 3.11 (20+ custom scripts) | Locator engine, screenshot annotation, self-healing, RAG indexing, ADO/Figma integration |
| Healing Database | SQLite (locator_healing_system.py) | Locator history, healing fingerprints, confidence scores, healing events |
| RAG System | JSON Vector DB (common_keywords_vector_db.py) | Keyword/variable similarity search for anti-hallucination grounding |
| Visual Analysis | Pillow (PIL) + OpenCV | Screenshot annotation, bounding box rendering, image comparison |
| DevOps | Azure DevOps (ADO) API | Test case management, work item tracking, result publishing |
| Design Integration | Figma REST API | Design-to-test pipeline, wireframe parsing, skeleton generation |
| Data Management | openpyxl + DataDriver | Excel-based test data generation and parameterized execution |
| Execution | Pabot (parallel Robot Framework) | Multi-device parallel test execution for regression suites |
| Packaging | vsce (VS Code Extension Manager) | VSIX packaging and distribution |
Key insights from building an enterprise AI test automation platform, and where the technology is heading.
The single most impactful architectural decision was mandatory workspace grounding. Generic AI code generators produce plausible-looking but wrong code. By forcing every reference through RAG retrieval + anti-hallucination validation, we achieved zero hallucinated keywords or locators in production scripts. The 10-step RAG checklist (executed before every generation) is non-negotiable.
Building the extension as a pure orchestration layer over existing Python scripts meant the AI/ML pipeline could evolve independently. When the locator engine gained new strategies, the extension needed zero changes. When the screenshot annotator added TOON reports, only a new command handler was added. This decoupling was critical for maintaining development velocity across both frontend and backend.
Splitting responsibilities across 7 specialized agents produced better results than a single monolithic generator. Each agent is an expert in its domain: @CodyView knows the 18-point review checklist cold, @SelfHealingAgent knows 10 recovery strategies, @StepAnalyzer knows how to classify reuse opportunities. The master orchestrator (@RoboClaw) delegates effectively because each sub-agent has a focused scope.
Capturing real device screenshots + XML page source from a running Appium session produces more accurate locators than working from static design files. The live DOM contains actual element hierarchies, accessibility labels, and dynamic content that design files cannot represent. Combining both approaches (Figma for early test case generation + live capture for locator extraction) provides the best coverage.
With 1,886 test files, even minor UI changes can break dozens of scripts. The 10-strategy self-healing system transforms locator breakage from a blocking event into an auto-recoverable incident. The SQLite healing database builds institutional memory — when a locator heals successfully, that knowledge persists and improves future healing confidence for similar elements.