Create counterfactual evaluation

POST/v2/counterfactual-evals

Queues paired replays of baseline episodes with a candidate policy. Returns before comparisons finish.

Authorization

BearerAuth

AuthorizationBearer <token>

Bearer token issued by Softmax

In: header

Request Body

application/json

candidate_policy_version_id*Candidate Policy Version Id

Candidate policy version to evaluate.

Formatuuid

baseline_policy_version_id*Baseline Policy Version Id

Baseline policy version to replay and compare against.

Formatuuid

league_id?|

League that supplies evaluation settings. Required when source is tournament.

source?CounterfactualEvalSource

Workflow creating the evaluation.

Default"tournament"

Value in

"tournament"
"experience_request"

n?|

Number of paired comparisons, or null to use the configured default.

idempotency_key*Idempotency Key

Key used to deduplicate repeated create requests.

Length5 <= length <= 200

bash

curl -X POST "https://example.com/v2/counterfactual-evals" \
  -H "Content-Type: application/json" \
  -d '{
    "candidate_policy_version_id": "b63b910e-ed55-43c0-be2c-6fd9d66eb022",
    "baseline_policy_version_id": "353a49c0-b329-40ab-bec7-26c418899e74",
    "idempotency_key": "string"
  }'

json

{
  "id": "string",
  "candidate_policy_version_id": "b63b910e-ed55-43c0-be2c-6fd9d66eb022",
  "baseline_policy_version_id": "353a49c0-b329-40ab-bec7-26c418899e74",
  "candidate_policy_label": "string",
  "baseline_policy_label": "string",
  "candidate_player_name": "string",
  "baseline_player_name": "string",
  "league_id": "string",
  "league_name": "string",
  "source": "tournament",
  "n_requested": 0,
  "n_paired": 0,
  "status": "pending",
  "skip_reason": "string",
  "error": "string",
  "verdict": "progression",
  "net_score": 0,
  "wins": 0,
  "losses": 0,
  "neutrals": 0,
  "mean_score_delta": 0,
  "experience_request_id": "256083de-09b1-42d4-acb9-be39dd53448f",
  "submission_id": "945dfedc-ad7f-4af7-b20c-06082ffa3cb3",
  "league_policy_membership_id": "93c17082-599d-4779-9bb4-07972e51d754",
  "idempotency_key": "string",
  "created_by_user_id": "string",
  "started_at": "2019-08-24T14:15:22Z",
  "completed_at": "2019-08-24T14:15:22Z",
  "created_at": "2019-08-24T14:15:22Z",
  "pairs": [
    {
      "job_index": 0,
      "baseline_episode_request_id": "string",
      "candidate_episode_request_id": "string",
      "candidate_episode_id": "442ebfae-0775-4701-b3ea-a6e05152d140",
      "seed": 0,
      "probe_slot": 0,
      "opponent_policy_version_ids": [
        "string"
      ],
      "coworld_id": "string",
      "variant_id": "string",
      "baseline_score": 0,
      "candidate_score": 0,
      "score_delta": 0,
      "outcome": 0
    }
  ]
}