Your Own Task

ct run eval assembles one Inspect Task from a task selection, a policy or protocol, Control Tower's scorers and a sandbox. A project that wants to change one of those parts builds the Task itself from the same pieces and runs it through Inspect's own eval. Nothing in ct run eval has to know: there are no task args for "my samples", "no opening message", "no task scorer" or "a plain submit tool", because each of those is a line in your own Task.

The pieces

The policy builds its own model from its model argument, but ct run eval also puts that name on the Task, so the eval reports the model the policy is running and eval() needs no --model of its own. A Task that leaves model unset falls back to Inspect's resolution, which is a different model or an error.

Example

A policy eval over your own samples, with the dataset's messages left as they are, the plain submit tool, and only the refusal diagnostic scored:

from inspect_ai import Task, eval

from control_tower.defaults import DEFAULT_MESSAGE_LIMIT
from control_tower.protocols.policies.policy_solver import build_policy_solver
from control_tower.protocols.policies.registry import policies
from control_tower.task.diagnostics import diagnostic_scorers
from control_tower.task.setup import wrap_with_task_setup
from control_tower.tools.submit import DEFAULT_SUBMIT_CONFIG

policy = policies.get("honest")
policy_args = policies.validate_args(policy, {"model": "sonnet"})

task = Task(
    dataset=my_dataset,
    solver=wrap_with_task_setup(
        build_policy_solver(
            policy.function,
            policy_args,
            kickoff_message=None,
            submit_config=DEFAULT_SUBMIT_CONFIG,
        )
    ),
    scorer=diagnostic_scorers(["refusal"], timeout_seconds=600),
    model=policy_args.model.model_name,
    message_limit=DEFAULT_MESSAGE_LIMIT,
)
eval(task)

Leave kickoff_message and submit_config at their defaults and swap diagnostic_scorers for get_linux_scorers and the Task behaves exactly as a ct run eval policy run does; the only thing you brought is the dataset. policies.get also resolves module.path:function, so the policy can be yours too.

A submit_config you pass replaces the one composed from the sample's tasks, so the submit fields those tasks declare are gone and a task scorer reading them back finds nothing. Override it when you have replaced the task scorers too, as the example does. A policy that adds fields of its own, as honest adds confusions, adds them to the one you passed; a policy whose submission schema is its own output contract, as find_questions, ignores it. Keep its name submit: a monitor sees a tool call under the name it was made with, and compose_submit_config enforces that only for the config it builds.

What your samples carry

Every sample carries SampleMetadata (control_tower.task.sample_metadata) in its metadata: an environment block and, when the sample has them, main_task and side_task, in the shape build_eval_dataset writes. A policy's tools, task setup, the composed submit tool and the task scorer all read it, and resolve the environment and tasks it names from the registry, so those must be registered in the running venv. A sample with only an environment block runs a policy against that environment with no task. Each sample sets its own sandbox.

A sample's input is its opening user turn. The default kickoff_message drops an empty one and supplies its own, so ct run eval samples can leave input blank; a caller who passes kickoff_message=None has taken that over and needs a non-empty input on every sample, because a blank user turn is what strict providers like Moonshot and Kimi reject.

What stays with ct run eval

The EC2 fleet, run metadata and the upload to the Control Tower database belong to ct run eval. A Task you own is an ordinary Inspect eval: run it with eval() or inspect eval, and read its log with Inspect's tools or convert it with ct traj.