Anonymous manuscript under review at EMNLP 2026

Personalized tool use without exposing the user profile.

UserToolBench tests whether an LLM can infer stable user preferences from interaction history, decide when clarification is needed, and produce executable user-aligned tool-call trajectories under incomplete information.

View repository Read paper Explore code

10user profiles

36tool sets

1,065dialogue turns

170unique tools

Reference trajectories use a sanitized persistent profile. Evaluated models see only history, the current request, and tool schemas.

From personalized wording to personalized decisions

A helpful assistant must do more than imitate a user's tone. It must choose tools, fill arguments, ask the right clarification questions, and coordinate multi-step actions in a way that remains consistent with the user's established behavior.

Preference inference

Hidden profile, visible history

Models recover decision-relevant preferences from prior interactions rather than receiving a structured persona directly.

Executable alignment

Tool calls are the output

Evaluation focuses on tool choice, argument values, clarification behavior, and trajectory structure.

Long horizon

Persistent behavioral consistency

Relevant signals can be distributed across earlier topics, requiring selective retrieval rather than indiscriminate memory use.

A benchmark built around incomplete requests

UserToolBench combines privacy-sanitized profiles, public API-style tool ecosystems, and multi-turn trajectories across lack-of-information, single-tool, and multi-tool settings.

Dataset composition and tool-domain distribution

Two levels of correctness

Generic task completion can look successful while still making the wrong decision for the user.

Exact trajectory accuracy: matches the profile-conditioned tool sequence and required decision-relevant arguments.
Relaxed task completion accuracy: credits executable task-complete alternatives when ordering or planning flexibility is acceptable.
Clarification judgment: asks when a constraint is unrecoverable and infers it only when history clearly supports the value.

Current models remain far from reliable personalized delegation

The best tested model reaches 49.36% average exact trajectory accuracy. Performance falls sharply on multi-tool coordination, and relaxed task completion substantially overestimates personalized alignment.

Exact trajectory accuracy versus relaxed task completion accuracy

Core generation and evaluation code

The repository includes multi-role trajectory synthesis, tool-call prediction, graph-aware execution matching, argument checking, and score aggregation.

Core code and tool schemas are available now. Sanitized benchmark data and model outputs are planned for a separate release.

Quick start

git clone https://github.com/xxy212/UserToolBench.git
cd UserToolBench/utb
python -m venv .venv
source .venv/bin/activate
pip install openai python-dotenv tqdm

# See utb/README.md for generation and evaluation commands.

Code documentation Open an issue

Release status and citation

Artifact	Status
Generation code	Available
Evaluation code	Available
Tool schemas	Available
Anonymous paper	Included
Sanitized benchmark data	Planned release

Temporary anonymous citation

@inproceedings{anonymous2026usertoolbench,
  title     = {UserToolBench: A User-Profile-Hidden Benchmark for
               Personalized Decision Making in Tool-Use LLMs},
  author    = {Anonymous Authors},
  booktitle = {Under review at EMNLP},
  year      = {2026},
  url       = {https://github.com/xxy212/UserToolBench}
}