Enterprise agent tool-use datasets: workflows, state and verified outcomes

What belongs in an enterprise workflow record

A useful record begins before the first tool invocation. Preserve the user request, the actor or role, the time at which the task is posed, and the data snapshot available to the agent. Then identify the tool interface, response objects, document revisions and expected result. Without this starting context, an evaluator may reward an answer that happens to match a text reference while relying on information the agent could never legitimately see.

Keep observation, reasoning annotation and execution separate. An event log can show that an order lookup occurred; an annotator can explain why the lookup was appropriate; a state diff can demonstrate that a refund was actually recorded. These are related forms of evidence, not interchangeable labels. Decide which are recorded from an actual workflow and which are authored later. This distinction is particularly important when comparing human demonstrations, generated traces and replayable environments.

Public research supplies useful design comparisons

IBM VAKRA enterprise-agent benchmark describes an evaluation setting that joins APIs and document retrieval. Its relevance here is the combination of tools, business state and evidence, rather than an assumed ready-to-license corporate archive. Inspect the current benchmark release and permitted uses before adopting any component.

OpenAPI Specification 3.2.1 specifies HTTP API descriptions. A tool-use package can retain an interface specification alongside the task, but an interface description does not supply the underlying records, authentication policy or expected business outcome. JSON Schema object reference provides a separate way to express constraints on object fields; passing such validation establishes shape, not factual correctness or authorization.

Freeze a coherent state instead of collecting disconnected examples

Consider a return request for an order delivered twelve days ago. The task is not reproducible if the order database changes the delivery date, the customer record loses its region, or a newly edited return policy changes the allowed window between evaluation runs. Tie all inputs to a snapshot identifier. Preserve the effective date of policies and a mapping from identifiers in documents to identifiers in API responses.

For a multi-step workflow, record the state after each permitted mutation or retain a replayable transaction history. Do not silently reset only some systems. An inventory reservation written in one service can affect a later shipping check even if the user never sees that intermediate action. An evaluator must know whether repeated runs start from the same conditions and whether irreversible external systems have been replaced with isolated test fixtures.

Make interface meaning explicit

An argument named amount is insufficient. State currency, major versus minor units, allowed sign, rounding convention and whether tax is included. Define whether a status field represents a request, approval or completed transaction. Record timezone and timestamp precision where deadlines matter. Tool descriptions should distinguish optional fields, absent fields and explicit null values rather than forcing a model to infer their meanings.

A versioned schema also needs representative responses: success, empty result, permission denial, conflicting records and recoverable failure. A tool-use dataset made only of successful paths can conceal whether the agent knows when to stop. Preserve error semantics rather than replacing every failure with the same generic message. If the task requires a write operation, describe idempotency behavior so repeated calls cannot be mistaken for separate legitimate transactions.

Evaluate authorization separately from answer relevance

Access rules should be part of the task conditions, not a note added after the examples are written. A support reader may inspect an order but lack permission to issue a refund. A correct final explanation can still accompany an unauthorized action. Score that action explicitly. Similarly, an agent should not use a document merely because it is present somewhere in the archive if that document is unavailable to its assigned role.

Include negative cases in which the correct behavior is to request clarification, decline an unsupported action or return insufficient evidence. A benchmark can remain useful when several valid execution paths exist; define acceptable results and prohibited side effects rather than requiring an exact preferred sequence. Keep secrets, private credentials and personal source identifiers out of a public example bundle.

A worked return-policy example

The downloadable example asks a reader-role agent to assess eligibility for a return without issuing money. The order lookup reports an unopened item delivered twelve days earlier. A mock policy permits returns within thirty days. The expected response cites both the policy revision and the lookup result. The expected state change is null. This combination makes the intended boundary inspectable: deciding eligibility and executing a refund are different tasks.

Change one condition at a time when testing the specification. If the order is opened, the policy no longer answers eligibility without another rule. If the actor is not allowed to view the order, the lookup itself should fail. If the return policy belongs to a different region, the answer should not borrow it. These variations explain what a representative pilot must demonstrate; they are not claimed results from running a model.

Trace fields and provenance that survive preparation

Use stable task, source and event identifiers. Preserve ordering and parent-child relationships for concurrent operations, retries and tool responses. If an annotation team normalizes a response or removes sensitive fields, retain the transformation version and an appropriate source reference. A human-readable explanation is useful, but the machine-readable lineage should make it possible to reconstruct which inputs support each reviewed label.

W3C PROV overview separates entities, activities and responsible agents as a provenance model. A project need not implement every PROV serialization to benefit from that distinction. For example, the source policy is an entity, redaction is an activity, and the organization performing review is a responsible party. A single permission_review=true flag does not replace the underlying review scope or documentation.

Splits, leakage and acceptance

Split at the business process or source family that matches the deployment question. Randomly separating paraphrases of the same order task may leave the same policy, customer pattern and state layout in both training and evaluation. Hold out relevant organizations, workflow variants or time periods when those differences matter. Record intentional shared reference documents so a downstream team can distinguish permitted common context from accidental contamination.

Pilot acceptance should include schema checks, referential integrity, role enforcement, replay stability and independent outcome review. Report failures by layer rather than collapsing them into one pass rate. An unreadable document, malformed argument, wrong decision and unauthorized write require different corrections. Agree how ambiguous or non-replayable records are handled before counting them as accepted tasks.

Cost and provider comparisons

Quote the accepted task, state snapshot, environment integration, demonstration, annotation and review separately. A thousand traces against one static toy service do not equal a thousand independently grounded workflows across several systems. Setup and replay maintenance can dominate a small pilot, whereas additional tasks may have lower preparation costs once a coherent environment is established. Public benchmark size is not a commercial price benchmark.

Ask a proposed provider to identify what is retained: original source records, interface descriptions, executable fixtures, generated dialogue, reviewed outcomes and usage documentation. A planning budget can multiply explicitly assumed review time by a chosen labor rate, but that models only that component. Fiund does not publish a category-wide transaction average here. Actual source access, confidentiality, preparation, permitted uses and delivery commitments require a project-specific proposal.

Download the worked specification

The trace file describes an invented return-eligibility task, frozen order state, reader permission and prohibited refund action. Its companion checklist is a proposed acceptance rubric with checks not run. Neither file contains a real enterprise integration, customer order or successful execution result. Use the relationships to specify a pilot; actual tool access and source permissions require separate confirmation.

Illustrative task, trace and result JSON

Pilot acceptance worksheet CSV

Questions buyers ask