Terminal and enterprise agent task datasets

A prompt is only one part of a task

An instruction can describe realistic work without providing a reproducible problem. A useful executable task needs an initial state, the files and tools available, constraints, expected outcomes and a way to verify completion. If the environment is missing or unstable, a model’s apparent failure may reflect infrastructure rather than reasoning. Preserve the task definition and the environment together.

The task objective should be legible to a competent person. Avoid hidden requirements that exist only in a verifier. At the same time, the verifier should not reward an obvious shortcut that produces a superficial artifact while failing the intended work. A reference solution helps inspect feasibility, but it is not the only valid solution. Define what success means independently from the exact sequence of commands used by the author.

Public benchmarks are comparisons, not automatic training inventory

Terminal-Bench’s primary paper describes the historical 2.0 release as 89 terminal tasks with environments, solutions and verification. The maintainers’ 4.0 update explains a later continuous-benchmark approach with changed resources and task fixes. These references show why a benchmark name without a version is insufficient.

Keep public evaluation artifacts distinct from custom training tasks. Reusing a published benchmark task or an answer-revealing derivative for training can compromise the meaning of later evaluation. A source should identify task lineage, intended role and known overlap with public tests. A private collection can use similar general skills without copying held-out questions. The commercial offer should describe what was authored or licensed and how its relationship to evaluation material was checked.

Environments need versioned, reproducible boundaries

Pin the operating environment, dependency versions, initial files and tool interfaces needed to solve the task. Distinguish a deterministic local artifact from an external service that can change. If network access is required, specify the allowed destination and expected behavior. A task that depends on live content can drift even when its prompt remains unchanged.

Record compute, memory, storage and time assumptions. A valid solution can fail under an unreasonable resource limit; a generous limit can hide an unintended performance shortcut. The right budget follows the task rather than a blanket choice. Preserve logs and error categories so infrastructure failures can be separated from task failures. A reproducible package should let a reviewer understand which conditions were fixed and which remain external dependencies.

A verifier should test the intended outcome

Tests can be too strict, too permissive or simply wrong. Exact string matching may reject a correct alternative, while checking only whether a file exists can reward an empty or irrelevant result. Define the substantive outcome and include negative controls. A verifier should fail intentionally incorrect outputs and accept reasonable valid solutions without depending on the author’s preferred implementation.

Where a model judge is used, preserve its instructions, version, criteria and uncertainty. A judge’s score is not automatically ground truth. Some tasks benefit from a combination of executable tests and human review. The chosen method should be documented so buyers can reproduce or audit it. A task file with a plausible expected answer should not be advertised as validated merely because its JSON structure is correct.

Reference solutions and trajectories answer different questions

A reference solution demonstrates one feasible path and helps inspect the environment and verifier. A trajectory records the steps a person or agent actually took, potentially including exploration, errors and recovery. A concise ideal solution is not equivalent to a genuine workflow trace. Label synthetic, edited, human-authored and agent-generated material explicitly.

For trajectories, preserve commands, observations, timestamps where relevant and the relation to task success. Redact credentials or private system details appropriately without making the sequence impossible to interpret. A transcript that omits the output needed to choose the next action may be unsuitable for training a decision process. Likewise, a successful final artifact does not prove every step in a purported reasoning trace is genuine or necessary.

Difficulty should follow the work, not artificial obstruction

A difficult task may require several dependent decisions, domain knowledge, debugging or careful evidence gathering. It should not be hard only because instructions are ambiguous, dependencies are broken or the verifier has a hidden trick. Describe the skill family and why the sequence is challenging. A long prompt or many files does not by itself establish deep reasoning.

Sample both ordinary and difficult cases from the target workflow. Software repair, data analysis, system configuration and enterprise record reconciliation require different environments and verification. A buyer should know the distribution rather than receive a task count dominated by one easy template. Independent reviewers can identify whether variation is substantive or merely cosmetic, such as renaming files while preserving the same solution structure.

Task families and contamination need explicit lineage

Several tasks can be derived from a common template, source problem or environment. Keep that lineage so related variants are grouped appropriately. Otherwise a training/evaluation split can test recognition of a familiar pattern rather than the intended new skill. Record known public ancestors and any edits that change the objective, data or verifier.

Do not treat a formatting conversion as newly authored intellectual content. Dependencies and source files can also carry conditions separate from the task instructions. The commercial scope should identify what can be redistributed and what the buyer must obtain separately. A dataset manifest should preserve those distinctions after packaging. The goal is a useful task collection with a traceable role, not a renamed copy of a public leaderboard’s test set.

Pricing and commercial evidence

Turing’s public dataset offering provides a named supplier comparison for terminal and enterprise task data. The checked page does not establish a comparable numeric license rate per accepted task. A contractor’s compensation, benchmark runtime cost and a commercial task license are different quantities. No reliable category-wide average is asserted here.

A written quote can separate task design, environment preparation, reference solution, verifier development, independent validation, trajectory production and ongoing maintenance. Acceptance should define which of those components is included. A cheap prompt-only batch cannot be assumed equivalent to tasks whose environments and tests were independently exercised. No directly comparable disclosed purchase value was verified; company funding or a platform acquisition would not supply that missing data-license price.

Example schema: a reproducible task package

A commercial task package can be inspected component by component. The example fields below separate the problem statement, environment, evidence of feasibility and method of scoring. A buyer may request some components without others, but the quote and acceptance record should say which are included.

The distinction matters when a task changes. Fixing instructions, replacing a dependency or adjusting a verifier can alter what a result means. Retain version identifiers and change rationale so a historical score can be connected to its actual conditions. A collection that cannot reproduce its environment may still offer useful scenario ideas, but should not be presented as the same product as independently executable evaluation tasks.

Case study design: validate the grader with contrasts

For a data-cleaning task, include one correct output, one output with a subtle row error and one superficially well-formatted but empty result. Ask whether the verifier distinguishes them for the intended reason. Then try a valid alternative implementation that reaches the same substantive outcome. An overly narrow grader may reject it even though the task is solved.

This design tests the meaning of acceptance rather than only whether a reference script runs. Record infrastructure errors separately from task errors and preserve the environment used. A buyer can then compare a prompt-only source, a partially prepared package and a validated task collection without assuming they are commercially equivalent. The procedure is a proposed sample review, not a statement that any offered tasks have already passed it.

Build a sample that can fail for the right reason

A task sample should include an ordinary valid solution, a different valid approach and deliberately incorrect outputs. Inspect whether the verifier distinguishes them. Recreate the environment from the supplied definition, identify external dependencies and review what happens when a prerequisite is missing. These are acceptance activities to agree, not a claim that a proposed collection has already passed them.

Send Fiund the intended skill families, agent tools, environment constraints, desired task role and commercial use. Explain whether you need newly authored tasks, licensed environments, reference solutions, workflow traces or a complete validated package. Keep evaluation independence explicit. Source rights, preparation and acceptance are reviewed for the actual project. A category page does not promise access to private benchmark answers or permit repackaging public held-out tests as uncontaminated training data.

Questions buyers ask