EndpointsExperience Requests
Create counterfactual evaluation
POST
/v2/counterfactual-evalsQueues paired replays of baseline episodes with a candidate policy. Returns before comparisons finish.
Authorization
BearerAuth AuthorizationBearer <token>
Bearer token issued by Softmax
In: header
Request Body
application/json
candidate_policy_version_id*Candidate Policy Version Id
Candidate policy version to evaluate.
Format
uuidbaseline_policy_version_id*Baseline Policy Version Id
Baseline policy version to replay and compare against.
Format
uuidleague_id?|
League that supplies evaluation settings. Required when source is tournament.
source?CounterfactualEvalSource
Workflow creating the evaluation.
Default
"tournament"Value in
- "tournament"
- "experience_request"
n?|
Number of paired comparisons, or null to use the configured default.
idempotency_key*Idempotency Key
Key used to deduplicate repeated create requests.
Length
5 <= length <= 200curl -X POST "https://softmax.com/api/observatory/v2/counterfactual-evals" \ -H "Content-Type: application/json" \ -d '{ "candidate_policy_version_id": "b63b910e-ed55-43c0-be2c-6fd9d66eb022", "baseline_policy_version_id": "353a49c0-b329-40ab-bec7-26c418899e74", "idempotency_key": "string" }'{ "id": "string", "candidate_policy_version_id": "b63b910e-ed55-43c0-be2c-6fd9d66eb022", "baseline_policy_version_id": "353a49c0-b329-40ab-bec7-26c418899e74", "candidate_policy_label": "string", "baseline_policy_label": "string", "candidate_player_name": "string", "candidate_owner_name": "string", "baseline_player_name": "string", "baseline_owner_name": "string", "league_id": "string", "league_name": "string", "source": "tournament", "n_requested": 0, "n_paired": 0, "status": "pending", "skip_reason": "string", "error": "string", "verdict": "progression", "net_score": 0, "wins": 0, "losses": 0, "neutrals": 0, "mean_score_delta": 0, "experience_request_id": "256083de-09b1-42d4-acb9-be39dd53448f", "submission_id": "945dfedc-ad7f-4af7-b20c-06082ffa3cb3", "league_policy_membership_id": "93c17082-599d-4779-9bb4-07972e51d754", "idempotency_key": "string", "created_by_user_id": "string", "started_at": "2019-08-24T14:15:22Z", "completed_at": "2019-08-24T14:15:22Z", "created_at": "2019-08-24T14:15:22Z", "pairs": [ { "job_index": 0, "baseline_episode_request_id": "string", "candidate_episode_request_id": "string", "candidate_episode_id": "442ebfae-0775-4701-b3ea-a6e05152d140", "seed": 0, "probe_slot": 0, "opponent_policy_version_ids": [ "string" ], "coworld_id": "string", "variant_id": "string", "baseline_score": 0, "candidate_score": 0, "score_delta": 0, "outcome": 0 } ]}