web-agent-protocol

<!-- markdownlint-disable first-line-h1 --> <!-- markdownlint-disable html --> <!-- markdownlint-disable no-duplicate-header --> <div align="center"> <img src="chrome-extension/assets/beholder-tool-kit-long.png" width="100%" alt="OTA-tool-kits" style="border-radius: 10px;" /> </div> <br> <div align="center" style="line-height: 1;"> <a href="https://www.otatech.ai/"><img alt="Homepage" src="https://img.shields.io/badge/Visit-otatech.ai-blue"/></a> <a href="https://huggingface.co/OTA-AI/OTA-v1"><img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-OTA%20AI-ffc107?color=ffc107&logoColor=white"/></a> <a href="https://github.com/OTA-Tech-AI/webagentprotocol/blob/main/LICENSE"><img alt="Code License" src="https://img.shields.io/badge/Code_License-MIT-f5de53?&color=f5deff"/></a> <br><br><br> </div> # Web Agent Protocol ## Overview The Web Agent Protocol (WAP) is a standardized framework designed to enable seamless interaction between users, web agents, and browsers by recording and replaying browser actions. It separates the concerns of action recording and execution, allowing for efficient automation and reusability. The Python SDK for WAP implements the full specification, making it easy to: 1. **Collect** user‑interaction data with the [OTA‑WAP Chrome extension](https://github.com/OTA-Tech-AI/webagentprotocol/tree/main/chrome-extension). 2. **Convert** the raw event stream into either **_exact‑replay_** or **_smart‑replay_** action lists. 3. **Convert** recorded actions into **_MCP_** servers for reuse by any agent or user 4. **Replay** those lists using the **_WAP-Replay_** protocol to ensure accurate browser operations. ### WAP FULL DEMO [](https://www.youtube.com/watch?v=joh9FXJfnwk) ### Without WAP  ### WAP Record  ### WAP Replay  ## Example using WAP  ## Setup Install the dependencies with the following command: Create a conda env ```bash conda create -n WAP python=3.11 ``` Activate the conda env ```bash conda activate WAP ``` Install the dependencies ```bash pip install -r requirements.txt ``` Setup your repo source path: ``` set PYTHONPATH=C:/path/to/webagentprotocol # for Windows export PYTHONPATH=/path/to/webagentprotocol # for Linux ``` Create **.env** file under the repo root directory with your own API keys: ``` OPENAI_API_KEY=sk-proj-... DEEPSEEK_API_KEY=sk-... ``` ## Record ### WAP record extension Please refer to [OTA‑WAP Chrome Extension](https://github.com/OTA-Tech-AI/webagentprotocol/tree/main/chrome-extension) to setup action capturer in your Chrome browser. ### Start data‑collection server Run the following command to start the server to collect data from the extension: ```bash python action_collect_server.py ``` **Once the server is up, you can start to record from the page using WAP Chrome extension.** The server listens on http://localhost:4934/action-data by default, please make sure the Host and Port in the extension settings match this server config. Each session will be saved to: ```bash data/YYYYMMDD/taskid/summary_event_<timestamp>.json ``` An example of the formatted data which you will received in the WAP backend server is like: ```json { "taskId": "MkCAhQsHgXn7YgaK", "type": "click", "actionTimestamp": 1746325231479, "eventTarget": { "type": "click", "target": "<a ota-use-interactive-target=\"1\" data-ordinal=\"3\" href=\"https://www.allrecipes.com/recipe/68925/cheesy-baked-salmon/\" data-tax-levels=\"\" data-doc-id=\"6592066\" class=\"comp mntl-card-list-card--extendable mntl-universal-card mntl-document-card mntl-card card card--no-image\" id=\"mntl-card-list-card--extendable_3-0\">\n<div class=\"loc card__top\"><div class=\"card__media mntl-image card__media universal-image__container\">...", "targetId": "mntl-card-list-card--extendable_3-0", "targetClass": "comp mntl-card-list-card--extendable mntl-universal-card mntl-document-card mntl-card card card--no-image" }, "allEvents": {}, "pageHTMLContent": "<header data-tracking-container=\"true\" data-collapsible=\"true\" class=\"comp header mntl-header mntl-header--magazine mntl-header--open-search-bar mntl-header--myr\" id=\"header_1-0\"><a data-tracking-container=\"true\" id=\"mntl-skip-to-content_1-0\" class=\"mntl-skip-to-content mntl-text-link\" rel=\"nocaes\" href=\"#main\"></a><div class=\"mntl-header__menu-top\">..." } ``` ## Generate replay lists | Mode | Command