OpenVLHarness:给多模态模型接感知/搜索/编程工具,基准提升约 9.7–12.6 分
UCLA 与 Apple 的 OpenVLHarness 为多模态大模型提供分割、深度、OCR 等工具与多模态记忆,八模型在 22 项基准上平均提升约 9.7–12.6 分。
研究者账号介绍 OpenVLHarness(UCLA 与 Apple):与其让视觉语言模型只「多想一会儿」,不如给它可调用的感知、搜索与编程工具。帖文称,在八个模型、二十二项基准上,接入该框架可带来约 9.7 到 12.6 分的提升;对比之下,把 GPT-6 Luna 的推理力度从 xhigh 拉到 max,token 开销约增 2.7 倍,分数只多约 0.7。
设计上有两点被强调。一是能力层:分割、深度、OCR、相机位姿、搜索等工具的结果会渲染成带标注的图像,并附简短说明与紧凑数据,便于模型消费。二是多模态记忆:掩码、深度图与图像可在多步之间被引用,编码智能体还能基于它们计算例如两物体间距。消融显示,只保留先前图像的文字描述几乎无增益(约 +0.3 到 +0.7),保留图像本身可再加约 2.2–3.4 分;若给工具却不做渲染层,8B 模型的通用 VQA 甚至从 58.0 掉到 52.4。
对研发与评测同学,这说明「工具返回值如何呈现」可能比单纯加长思维链更划算;复现时仍须核对论文/代码仓中的模型名单、基准集合与许可证,帖中型号命名以原文为准。
OpenVLHarness tool-using VLMs beat longer thinking
Should a vision-language model think longer, or go and measure? OpenVLHarness from UCLA and Apple gives a multimodal LLM perception, search and coding tools, and adds 9.7–12.6 points across eight models on 22 benchmarks. Extra reasoning alone does far less: raising GPT-6 Luna from xhigh to max effort costs 2.7× more tokens for +0.7 points.
Two design choices make the tools useful:
• Capability layer: each tool (segmentation, depth, OCR, camera pose, search) returns its result rendered for the model, as an annotated image plus a short description and compact data
• Multimodal memory: masks, depth maps and images stay addressable across steps, so a coding agent can compute over them, e.g. the distance between two objects
The ablation is telling: keeping only text descriptions of earlier images adds almost nothing (+0.3 to +0.7, within run-to-run noise), while keeping the images themselves adds 2.2–3.4 points. And raw tool access without the rendering layer even hurt general VQA on the 8B model (58.0 → 52.4).