Implement WebVoyager with Java and OpenAI vision model
Dựa vào thuật toán ML WebVoyager như browser-use mình thử implement bằng Java dựa theo repo https://github.com/MinorJerry/WebVoyager/tree/main
Dưới đây là các bước:
Step 1: Tạo các hình chữ nhật (rectangle) để labeling các đối tượng trên 1 trang để giúp AI model có thể thêm context.

Step 2: đọc html page rồi generate các đối tượng với tagName, Text, attributes, aria-label, việc này giúp bổ sung context cho LLM model
[
[0]: <img> "";,
[1]: <label> "Username";,
[2]: <input> "";,
[3]: <label> "Password";,
[4]: <input> "";,
[5]: <button> "Login";,
[6]: <a> "Elemental Selenium";
]
Code snippet:
public static Object[] getWebWementRect(WebDriver driver) {
String jsScript = """
let labels = [];
function markPage() {
var bodyRect = document.body.getBoundingClientRect();
var items = Array.prototype.slice.call(
document.querySelectorAll('*')
).map(function(element) {
var vw = Math.max(document.documentElement.clientWidth || 0, window.innerWidth || 0);
var vh = Math.max(document.documentElement.clientHeight || 0, window.innerHeight || 0);
var rects = [...element.getClientRects()].filter(bb => {
var center_x = bb.left + bb.width / 2;
var center_y = bb.top + bb.height / 2;
var elAtCenter = document.elementFromPoint(center_x, center_y);
return elAtCenter === element || element.contains(elAtCenter)\s
}).map(bb => {
const rect = {
left: Math.max(0, bb.left),
top: Math.max(0, bb.top),
right: Math.min(vw, bb.right),
bottom: Math.min(vh, bb.bottom)
};
return {
...rect,
width: rect.right - rect.left,
height: rect.bottom - rect.top
}
});
var area = rects.reduce((acc, rect) => acc + rect.width * rect.height, 0);
return {
element: element,
include:\s
(element.tagName === "INPUT" || element.tagName === "TEXTAREA" || element.tagName === "SELECT") ||
(element.tagName === "BUTTON" || element.tagName === "A" || (element.onclick != null) || window.getComputedStyle(element).cursor == "pointer") ||
(element.tagName === "IFRAME" || element.tagName === "VIDEO" || element.tagName === "LI" || element.tagName === "TD" || element.tagName === "OPTION")
,
area,
rects,
text: element.textContent.trim().replace(/\\s{2,}/g, ' ')
};
}).filter(item =>
item.include && (item.area >= 20)
);
// Only keep inner clickable items
// first delete button inner clickable items
const buttons = Array.from(document.querySelectorAll('button, a, input[type="button"], div[role="button"]'));
//items = items.filter(x => !buttons.some(y => y.contains(x.element) && !(x.element === y) ));
items = items.filter(x => !buttons.some(y => items.some(z => z.element === y) && y.contains(x.element) && !(x.element === y) ));
items = items.filter(x =>\s
!(x.element.parentNode &&\s
x.element.parentNode.tagName === 'SPAN' &&\s
x.element.parentNode.children.length === 1 &&\s
x.element.parentNode.getAttribute('role') &&
items.some(y => y.element === x.element.parentNode)));
items = items.filter(x => !items.some(y => x.element.contains(y.element) && !(x == y)))
// Function to generate random colors
function getRandomColor(index) {
var letters = '0123456789ABCDEF';
var color = '#';
for (var i = 0; i < 6; i++) {
color += letters[Math.floor(Math.random() * 16)];
}
return color;
}
function getFixedColor(index) {
var color = '#000000'
return color
}
//function getFixedColor(index){
// var colors = ['#FF0000', '#00FF00', '#0000FF', '#000000']; // Red, Green, Blue, Black
// return colors[index % 4];
//}
// Lets create a floating border on top of these elements that will always be visible
items.forEach(function(item, index) {
item.rects.forEach((bbox) => {
newElement = document.createElement("div");
var borderColor = COLOR_FUNCTION(index);
newElement.style.outline = `2px dashed ${borderColor}`;
newElement.style.position = "fixed";
newElement.style.left = bbox.left + "px";
newElement.style.top = bbox.top + "px";
newElement.style.width = bbox.width + "px";
newElement.style.height = bbox.height + "px";
newElement.style.pointerEvents = "none";
newElement.style.boxSizing = "border-box";
newElement.style.zIndex = 2147483647;
// newElement.style.background = `${borderColor}80`;
// Add floating label at the corner
var label = document.createElement("span");
label.textContent = index;
label.style.position = "absolute";
//label.style.top = "-19px";
label.style.top = Math.max(-19, -bbox.top) + "px";
//label.style.left = "0px";
label.style.left = Math.min(Math.floor(bbox.width / 5), 2) + "px";
label.style.background = borderColor;
label.style.color = "white";
label.style.padding = "2px 4px";
label.style.fontSize = "12px";
label.style.borderRadius = "2px";
newElement.appendChild(label);
document.body.appendChild(newElement);
labels.push(newElement);
// item.element.setAttribute("-ai-label", label.textContent);
});
})
// For the first way
// return [labels, items.map(item => ({
// rect: item.rects[0] // assuming there's at least one rect
// }))];
// For the second way
return [labels, items]
}
return markPage();""".replace("COLOR_FUNCTION", "getFixedColor");
ArrayList<?> raw = (ArrayList<?>) ((JavascriptExecutor) driver).executeScript(jsScript);
//capture screenshot
File screenshot = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);
String screenshotPath = "screenshot-login.png";
File destFile = new File(screenshotPath);
screenshot.renameTo(destFile);
ArrayList<WebElement> labels = (ArrayList<WebElement>) raw.get(0);
ArrayList<Map<String, Object>> items = (ArrayList<Map<String, Object>>) raw.get(1);
// System.out.println("Labels: " + labels.size());
// System.out.println("Items: " + items.size());
List<String> formatElements = new ArrayList<>();
List<WebElement> elements = new ArrayList<>();
items.forEach(item -> {
String labelText = ((WebElement) item.get("element")).getText();
elements.add(((WebElement) item.get("element")));
String tagName = ((WebElement) item.get("element")).getTagName();
String typeAttribute = ((WebElement) item.get("element")).getAttribute("type");
String ariaLabel = ((WebElement) item.get("element")).getDomProperty("aria-label");
int index = items.indexOf(item);
System.out.println("==========================");
System.out.println("Element Index: " + index);
System.out.println("text: " + labelText);
System.out.println("tagName: "+tagName);
System.out.println("type: "+ typeAttribute);
System.out.println("aria-label: "+ariaLabel);
List<String> inputTypes = List.of("text", "search", "password", "email", "tel");
List<String> buttonTypes = List.of("button", "submit");
List<String> tags = List.of("input", "textarea", "button");
if (
(labelText == null && (inputTypes.contains(typeAttribute))) || (tagName.equalsIgnoreCase("textarea")) || (tagName.equalsIgnoreCase("button") && buttonTypes.contains(typeAttribute))
) {
if (ariaLabel == null) {
formatElements.add(String.format("[%d]: <%s> \"%s\";\n", index, tagName, labelText));
} else {
formatElements.add(String.format("[%d]: <%s> \"%s\";\n", index, tagName, ariaLabel));
}
} else if (labelText != null && labelText.length() < 200) {
if (ariaLabel == null) {
formatElements.add(String.format("[%d]: <%s> \"%s\";", index, tagName, labelText));
// System.out.printf("[%d]: <%s> \"%s\";\n",index,tagName, labelText);
} else {
formatElements.add(String.format("[%d]: <%s> \"%s\";\n", index, tagName, ariaLabel));
// System.out.printf("[%d]: <%s> \"%s\";\n", index, tagName, ariaLabel);
}
} else {
if (ariaLabel == null) {
formatElements.add(String.format("[%d]: <%s> \"%s\";", index, tagName, labelText));
// System.out.printf("<%s> \"%s\";\n", tagName, labelText);
} else {
formatElements.add(String.format("[%d]: <%s> \"%s\";", index, tagName, ariaLabel));
// System.out.printf("<%s> \"%s\";\n", tagName, ariaLabel);
}
}
});
// System.out.println(formatElements);
return new Object[]{
labels,
elements,
formatElements
};
}
Step 3: tạo một method call LLM với image
public static String callGPT(String prompt,String base64Image) {
ChatModel chatModel = OpenAiChatModel.builder()
.apiKey(ApiKeys.OPENAI_API_KEY) // Please use your own OpenAI API key
.modelName(GPT_4_O_MINI)
.maxTokens(50)
.build();
SystemMessage systemMessage = SystemMessage.from(Prompts.SYSTEM_PROMPT_TEXT_ONLY);
UserMessage userMessage = UserMessage.from(
TextContent.from(prompt),
ImageContent.from(base64Image, "image/png")
);
String response = chatModel.chat(systemMessage,userMessage).aiMessage().text();
System.out.println("AI Response: " + response);
String[] responseParts = response.split("Thought:|Action:|Observation:");
//extract action and thought pattern = r'Thought:|Action:|Observation:'
String thought = responseParts[1].trim();
String action = responseParts[2].trim();
// System.out.println("Thought: " + thought);
return action;
}
Step 4: code demo
public static void main(String[] args) {
WebDriver driver = new ChromeDriver();
driver.get("https://the-internet.herokuapp.com/login");
try{
driver.findElement(By.tagName("body")).click();
}catch (Exception e){
System.out.println(e.toString());
}
((JavascriptExecutor) driver).executeScript("window.onkeydown = function(e) {if(e.keyCode == 32 && e.target.type != 'text' && e.target.type != 'textarea') {e.preventDefault();}};");
Object[] raw = Utils.getWebWementRect(driver);
List<WebElement> rects = (List<WebElement>) raw[0];
List<WebElement> elements = (List<WebElement>) raw[1];
List<String> webText = (List<String>) raw[2];
String base64Image = Utils.base64EncodeImage(driver);
String promptTask = formatPrompt("Type tomsmith into username textbox", String.valueOf(webText));
String action = Utils.callGPT(promptTask, base64Image);
System.out.println(action);
Pattern typePattern = Pattern.compile("Type \\[?(\\d+)]?[; ]+\\[?(.[^]]*)]?");
Matcher matcher = typePattern.matcher(action);
Map<String, String> actionMap = new HashMap<>();
while (matcher.find()) {
System.out.println("Found action: " + matcher.group());
String key = matcher.group(1);
String value = matcher.group(2);
actionMap.put("action", "type");
actionMap.put("elementIndex", key);
actionMap.put("content", value);
}
int elementIndex = Integer.parseInt(actionMap.get("elementIndex"));
System.out.println(webText.get(elementIndex));
WebElement targetElement = elements.get(elementIndex);
for (WebElement rect : rects) {
((JavascriptExecutor) driver).executeScript("arguments[0].remove()",rect);
}
targetElement.sendKeys(actionMap.get("content"));
Utils.takeScreenshot(driver, "screenshot-after-action.png");
driver.quit();
}
public static String formatPrompt(String task,String webText) {
return """
## tasks
{task}
## context
Observation: please analyze the attached screenshot and give the Thought and Action.
```
{web_text}
```
""".replace("{task}", task).replace("{web_text}", webText);
}
System Prompt
// Some codecode
package voyager;
public class Prompts {
public static String SYSTEM_PROMPT_TEXT_ONLY = """
Imagine you are a robot browsing the web, just like humans. Now you need to complete a task. In each iteration, you will receive an Accessibility Tree with numerical label representing information about the page, then follow the guidelines and choose one of the following actions:
1. Click a Web Element.
2. Delete existing content in a textbox and then type content.
3. Scroll up or down. Multiple scrolls are allowed to browse the webpage. Pay attention!! The default scroll is the whole window. If the scroll widget is located in a certain area of the webpage, then you have to specify a Web Element in that area. I would hover the mouse there and then scroll.
4. Wait. Typically used to wait for unfinished webpage processes, with a duration of 5 seconds.
5. Go back, returning to the previous webpage.
6. Google, directly jump to the Google search page. When you can't find information in some websites, try starting over with Google.
7. Answer. This action should only be chosen when all questions in the task have been solved.
Correspondingly, Action should STRICTLY follow the format:
- Click [Numerical_Label]
- Type [Numerical_Label]; [Content]
- Scroll [Numerical_Label or WINDOW]; [up or down]
- Wait
- GoBack
- Google
- ANSWER; [content]
Key Guidelines You MUST follow:
* Action guidelines *
1) To input text, NO need to click textbox first, directly type content. After typing, the system automatically hits `ENTER` key. Sometimes you should click the search button to apply search filters. Try to use simple language when searching.
2) You must Distinguish between textbox and search button, don't type content into the button! If no textbox is found, you may need to click the search button first before the textbox is displayed.
3) Execute only one action per iteration.
4) STRICTLY Avoid repeating the same action if the webpage remains unchanged. You may have selected the wrong web element or numerical label. Continuous use of the Wait is also NOT allowed.
5) When a complex Task involves multiple questions or steps, select "ANSWER" only at the very end, after addressing all of these questions (steps). Flexibly combine your own abilities with the information in the web page. Double check the formatting requirements in the task when ANSWER.
* Web Browsing Guidelines *
1) Don't interact with useless web elements like Login, Sign-in, donation that appear in Webpages. Pay attention to Key Web Elements like search textbox and menu.
2) Vsit video websites like YouTube is allowed BUT you can't play videos. Clicking to download PDF is allowed and will be analyzed by the Assistant API.
3) Focus on the date in task, you must look for results that match the date. It may be necessary to find the correct year, month and day at calendar.
4) Pay attention to the filter and sort functions on the page, which, combined with scroll, can help you solve conditions like 'highest', 'cheapest', 'lowest', 'earliest', etc. Try your best to find the answer that best fits the task.
Your reply should strictly follow the format:
Thought: {Your brief thoughts (briefly summarize the info that will help ANSWER)}
Action: {One Action format you choose}
Then the User will provide:
Observation: {Accessibility Tree of a web page}
""";
}
Và đây là kết quả

Last updated