> ## Documentation Index > Fetch the complete documentation index at: https://wb-21fd5541-remote-scorers-ag.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Score agent turns with a remote scorer > Score W&B Weave agent turns with your own HTTP endpoint and see the results as tags or ratings on the Signals tab. export const GitHubLink = ({url, compact = false}) => {compact ? "View source" : <> GitHub source } ; A remote scorer signal scores each completed agent turn with an HTTP endpoint that you host, instead of with an LLM judge. When a turn ends, the W\&B Weave agent scoring worker sends the turn to your endpoint in an HTTP `POST` and records the response as feedback on the turn. The result appears as a tag or rating on the **Signals** tab of the [Agents view](/weave/guides/tracking/view-agent-signals). This page covers remote scorers for agent turns. To score Calls traced with `@weave.op`, see [Score Calls with remote scorers](/weave/guides/evaluation/remote-scorers). You configure remote scorers with the Python SDK or the Weave UI. The TypeScript SDK doesn't include `RemoteScorer`. ## How agent turn scoring works An agent turn is scored in the following sequence: 1. A turn ends. When a root span (a span with no parent) ends, Weave treats it as a completed turn and emits a `weave.genai.turn_ended` event. 2. The agent scoring worker loads the project's active signals that target `weave.genai.turn_ended`, then applies each signal's filters and sample rate. 3. For each `RemoteScorer` on a matching signal, the worker builds a `schema_version: 2` request from the turn's span, including its messages, resolves the scorer's credentials, checks the endpoint URL against the allowed hosts, and sends the `POST`. 4. The worker validates the response and writes the result as feedback on the turn. Tags and ratings appear on the **Signals** tab. Weave scores only completed turns. Individual LLM and tool spans, and whole conversations, aren't remote scoring targets. A remote scorer signal is a `Monitor` whose `op_names` is `["weave.genai.turn_ended"]`, whether you create it in the UI or with the SDK. The worker retries a failed attempt with the same `Idempotency-Key`, up to three attempts within 30 seconds of the first attempt. A `5xx`, `408`, or `429` response is retried. A timeout uses the whole 30 seconds, so a timed-out request isn't retried. Any other `4xx` response isn't retried. If your endpoint can't score a turn in time, return `503` quickly rather than letting the request time out, so that Weave retries it. Agent turn scoring requires the structured result format, because Weave stores the tags and ratings as typed feedback columns. ## Enable remote scoring Remote scoring is off until it's enabled for your organization or deployment, and the scoring worker calls a scorer endpoint only if its host is on an allowlist. How you enable it depends on your deployment type. **Multi-tenant Cloud** An organization admin or billing admin enables remote scoring for the organization: 1. Open `https://wandb.ai/account-settings/[ORG]/settings`, replacing `[ORG]` with the organization that owns your project. 2. Select the **Remote scoring** tab. 3. Turn on **Enable remote scoring**. 4. Under **Allowed hosts**, click **Add host** and enter each host that remote scorers may call. Saving with remote scoring enabled requires at least one host. Leave the port blank to allow any port on that host. 5. Click **Save settings**. **Dedicated Cloud** Ask W\&B to enable remote scoring for your deployment and configure its allowed hosts. **Self-Managed** If you run W\&B Weave in a [W\&B Self-Managed](/platform/hosting/hosting-options/self-managed) deployment, set these environment variables through `extraEnv` on each scoring worker: the online evaluation worker, the call scoring worker, and the agent scoring worker. | Environment variable | Default | Effect | | ------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------- | | `WF_SCORING_WORKER_REMOTE_SCORING_ENABLED` | `false` | Turns on outbound requests to remote scorers. When `false`, no remote scorer runs, regardless of other settings. | | `WF_SCORING_WORKER_REMOTE_HTTP_TIMEOUT_SECONDS` | `30` | Timeout for each request to a scorer endpoint. | | `WF_SCORING_WORKER_REMOTE_SCORER_ALLOWED_HOSTS` | empty | Comma-separated operator allowlist of `host` or `host:port` entries. | | `WF_SCORING_WORKER_REMOTE_SCORER_VALIDATE_HOSTS` | `true` | Enforces the allowed hosts lists. Private, loopback, and cloud metadata addresses are rejected regardless of this setting. | | `WF_SCORING_WORKER_REMOTE_SCORER_ALLOW_INSECURE_HTTP` | `false` | Permits `http://` endpoint URLs. | | `WF_SCORING_WORKER_REMOTE_SCORER_ALLOWED_PRIVATE_CIDRS` | empty | Comma-separated CIDR networks, such as `10.0.0.0/8`, whose private addresses remote scorers may call. | To show the remote scoring settings and scorer options in the Weave UI, also set `GORILLA_GATE_WEAVE_REMOTE_SCORING=true` on the W\&B server. **Allowed host rules**. | \| `allowInsecureHttp` | `WF_SCORING_WORKER_REMOTE_SCORER_ALLOW_INSECURE_HTTP` | `false` | Permits `http://` endpoint URLs. | \| `requireStructuredResultSchema` | `WF_SCORING_WORKER_REMOTE_SCORER_REQUIRE_STRUCTURED_RESULT_SCHEMA` | `true` | Rejects responses whose `result` isn't in the structured score format. | **Allowed host rules** The scoring worker checks every scorer endpoint URL, and separately the OAuth token endpoint URL when a scorer uses OAuth, against these rules: * An entry matches an exact host, with an optional port. An entry without a port allows any port on that host. * An entry that starts with `*.` matches subdomains at any depth, but not the domain itself. `*.corp.example.com` matches `a.corp.example.com` and `a.b.corp.example.com`, not `corp.example.com`. The suffix after `*.` must contain at least two labels, so `*.com` is rejected. A wildcard can't be combined with an IP address. * When both an operator allowlist and an organization allowlist exist, the URL must satisfy both. An empty operator allowlist adds no restriction. When no allowlist exists at all, the worker rejects every host. * Loopback, private, internal, and cloud metadata addresses are rejected. On Self-Managed, private addresses in the networks listed in `WF_SCORING_WORKER_REMOTE_SCORER_ALLOWED_PRIVATE_CIDRS` are allowed. * HTTPS is required unless the deployment permits insecure HTTP. * Redirects aren't followed. ## Build the scorer endpoint Your endpoint accepts a JSON `POST` from Weave and returns a JSON score. For a reference implementation, see [Sample code](#sample-code). ### Request Weave sends one HTTP `POST` per scored target to the scorer's endpoint URL, with these headers: | Header | Value | | ------------------------ | ---------------------------------------------------------------------------------- | | `Content-Type` | `application/json` | | `Authorization` | `Bearer [TOKEN]` | | `Idempotency-Key` | A key derived from the scored target, the monitor version, and the scorer version. | | `X-Correlation-ID` | A correlation ID for this request. | | `X-Weave-Schema-Version` | `1` or `2`, equal to `schema_version` in the body. | Weave might deliver the same scoring attempt more than once. Use `Idempotency-Key` to deduplicate if needed for your endpoint. The key is stable for one request version, so a V1 and a V2 request for the same Call carry different keys. Every request body has these top-level fields: | Field | Description | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | `schema_version` | Integer, `1` or `2`. | | `scoring_call_id`, `scoring_trace_id` | Identifiers for this scoring attempt. | | `monitor` | `name` and `version_digest` of the monitor that selected the target. | | `scorer` | `name`, `ref`, and optional `config`. `config` is the mapping set on the `RemoteScorer`, passed through unchanged. | | `triggered_at` | Optional ISO 8601 timestamp. | Weave omits an optional field with no value rather than sending it as `null`. Weave might add optional fields to a version without changing its number, so ignore fields you don't recognize. Request and response bodies are limited to 1 MiB each and contain JSON text only, never images, audio, or video. A target that exceeds these limits isn't sent, so it isn't scored. Each request carries one target. An agent turn request carries two version numbers. The top-level `schema_version` is the envelope version, which is `2` for agent turns. The scored data is under `scoring_target`, a tagged union with three fields: * `type`: The kind of target. `agent_turn` for a turn. The contract also defines `call`, which agent turn scoring never sends. * `schema_version`: The payload version for that type. It counts independently of the envelope version. The `agent_turn` payload is at payload version `1`. * `payload`: The data for that type. Check the envelope version first, then check `scoring_target.type` together with `scoring_target.schema_version` to choose how to score the payload. Return `4xx` for a combination your endpoint doesn't handle, such as `call` if you score only agent turns. The `agent_turn` payload at payload version `1` has these fields: | Field | Description | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | | `event_type` | `weave.genai.turn_ended`. | | `project_id` | The project, as `[YOUR-TEAM]/[YOUR-PROJECT]`. | | `trace_id`, `span_id` | Identifiers of the turn's root span. | | `span_name` | The span name, for example `invoke_agent sample-support-agent`. | | `operation_name` | The GenAI operation name, for example `invoke_agent`. `null` if the trace didn't record it. | | `started_at`, `ended_at` | ISO 8601 timestamps. | | `conversation` | `id` and `name`. Each is `null` if the trace didn't record it. | | `agent` | `name`, `version`, and `description`. Each is `null` if the trace didn't record it. | | `status` | `code`, `message`, and `error_type`. `code` is `UNSET`, `OK`, or `ERROR`. `message` and `error_type` are `null` if the trace didn't record them. | | `messages` | `system_instructions`, `input`, and `output`. Each is a list, `[]` when the turn has no messages of that kind. | A turn that ends without an explicit status arrives with `status.code` set to `UNSET`. Treat `UNSET` as a normal completed turn and `ERROR` as the failure signal. Each message in `input` and `output` has `role`, `content`, and `finish_reason`. `content` is plain text, or a JSON-encoded array of parts when the message carries structured content such as tool calls. ```json lines theme={null} { "schema_version": 2, "scoring_target": { "type": "agent_turn", "schema_version": 1, "payload": { "event_type": "weave.genai.turn_ended", "project_id": "[YOUR-TEAM]/[YOUR-PROJECT]", "trace_id": "0af7651916cd43dd8448eb211c80319c", "span_id": "b7ad6b7169203331", "span_name": "invoke_agent sample-support-agent", "operation_name": "invoke_agent", "started_at": "2026-06-08T12:00:00+00:00", "ended_at": "2026-06-08T12:00:01+00:00", "conversation": { "id": "conversation-0001", "name": "Support session" }, "agent": { "name": "sample-support-agent", "version": "2026-06-08", "description": "Answers support questions" }, "status": { "code": "UNSET", "message": null, "error_type": null }, "messages": { "system_instructions": [ "Answer the user accurately and concisely." ], "input": [ { "role": "user", "content": "What are your support hours?", "finish_reason": "" } ], "output": [ { "role": "assistant", "content": "Our support team is available Monday through Friday, 9am to 5pm Eastern.", "finish_reason": "stop" } ] } } }, "scoring_call_id": "018f8d6c-8d5f-7000-8000-000000000005", "scoring_trace_id": "018f8d6c-8d5f-7000-8000-000000000006", "monitor": { "name": "example_remote_scorer_agent_monitor", "version_digest": "monitor-version-digest" }, "scorer": { "name": "example_remote_scorer", "ref": "weave:///[YOUR-TEAM]/[YOUR-PROJECT]/object/example_remote_scorer:scorer-version-digest", "config": { "example_threshold": 0.8 } }, "triggered_at": "2026-06-08T12:00:02+00:00" } ``` Weave adds optional fields to a payload without changing its payload version. Weave removes, renames, or changes the meaning of a field only with a new payload version for that type, or a new envelope version for changes to the envelope. A new target type joins the V2 union at payload version `1`. ### Response Return HTTP `200` with a JSON object that has two fields: * `schema_version`: Integer equal to the request's `schema_version`. * `result`: One score object, a list of score objects, or an object of the form `{"scores": [...]}`. A score object has these fields: | Field | Required | Description | | ------------ | -------- | ------------------------------------------------------------------------------------------ | | `value` | Yes | A tag, as a string of at most 36 characters, or a rating, as a number from `0.0` to `1.0`. | | `reason` | No | A string that explains the score. | | `confidence` | No | A number from `0.0` to `1.0`. | Weave treats any non-`200` response as a scorer failure and records no feedback for that attempt. Weave doesn't follow redirects, so a redirect is also a failure. Weave doesn't parse the body of an error response. Return `4xx` for requests your endpoint never accepts and `5xx` for temporary problems. For an agent turn request, the response's `schema_version` is `2`. The following example returns one score object: ```json lines theme={null} { "schema_version": 2, "result": { "value": "concise", "reason": "72 characters.", "confidence": 0.9 } } ``` Weave normalizes the tags and reasons in the result before storing them. See [How Weave normalizes scores](/weave/guides/tracking/view-agent-signals#how-weave-normalizes-scores). ## Authenticate requests from Weave Weave authenticates to your endpoint with a bearer token. The request carries no W\&B credential. The token proves to your endpoint that the request came from Weave, not the reverse. Each `RemoteScorer` uses one of two modes: | Mode | What Weave does | What you configure | | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | `oauth_client_credentials` | Requests a token from your OAuth token endpoint with the client credentials grant, sending the client ID and secret with HTTP Basic authentication (`client_secret_basic`), then sends that token to your scorer endpoint. | The token endpoint URL, the client ID, the name of the secret that holds the client secret, and an optional scope. | | `static_bearer` | Sends a fixed token to your scorer endpoint. | The name of the secret that holds the token. | Store the client secret or bearer token in the secret store of the team that owns the project before you register the scorer. The `RemoteScorer` configuration holds only the secret name. The scoring worker resolves the value at scoring time. ## Create a remote scorer signal Create the signal in the Weave UI or with the Python SDK. Both create a `RemoteScorer` attached to a monitor that targets `weave.genai.turn_ended`. ### Weave UI Create the signal from the Agents view: 1. In the Weave project sidebar, click **Agents**. 2. In the tab bar, click **Signals**. 3. Click **New signal**, then click **Remote scorer**. 4. In the **Remote scorer** drawer, **Scored by** is set to **Remote scorer**. Configure the following fields: * **Scorer name**: The name shown in the **Scorer** column of the Signals table. Up to 128 characters. * **Scoring endpoint URL**: The URL Weave sends the `POST` to. * **Authentication**: **Static bearer** or **OAuth client credentials**. For static bearer, select or enter the **Bearer token secret name**. For OAuth, enter the **Token endpoint URL**, **Client ID**, **Client secret name**, and optional **Scope**. Secret fields take team secret names, not values. * **Config (JSON, optional)**: A JSON object passed to your endpoint as `scorer.config`. * **Only score turns matching** (Optional): Expand **Advanced**, then add filters to restrict which turns the signal scores, for example by agent name, agent version, operation name, tool name, or status code. To score every turn, leave it empty. Weave combines multiple filters with `AND` logic. * **Sample rate** (Optional): Expand **Advanced**, then set the fraction of matching turns the signal scores. 5. Click **Create signal**. The remote scorer form has no tag or rating fields. Your endpoint determines what it returns, and the Signals table shows the tags and ratings it receives. In the **Scorer** column, a remote scorer signal shows a webhook icon. ### Python SDK Publish a `RemoteScorer`, then activate a `Monitor` that lists it in `scorers` and targets `weave.genai.turn_ended` in `op_names`. ```python lines theme={null} import weave from weave.flow.monitor import Monitor from weave.scorers.remote_scorer import RemoteScorer, StaticBearerAuthConfig weave.init("[YOUR-TEAM]/[YOUR-PROJECT]") scorer = RemoteScorer( name="policy_remote_scorer", endpoint_url="https://scoring.example.com/weave/score", config={"threshold": 0.9}, # Sent to your endpoint as scorer.config auth_config=StaticBearerAuthConfig( mode="static_bearer", bearer_secret_name="WEAVE_REMOTE_SCORER_BEARER_TOKEN", ), ) weave.publish(scorer, name="policy_remote_scorer") monitor = Monitor( name="policy_remote_signal", scorers=[scorer], op_names=["weave.genai.turn_ended"], # Score completed agent turns sampling_rate=1.0, ) monitor.activate() ``` For OAuth client credentials, pass an `OAuthClientCredentialsConfig` as `auth_config` instead. ## Sample code The [`examples/remote_scorer`](https://github.com/wandb/weave/tree/master/examples/remote_scorer) directory in the `weave` repository is the reference implementation of this contract and the source of truth for the sample code. One endpoint in that sample accepts V1 Call requests, V2 Call requests, and V2 agent turn requests. For agent turns, these files apply: * `remote_scorer_app.py`: A FastAPI app with `GET /health` and `POST /score`. * `auth.py`: A development-only bearer token check against the `REMOTE_SCORER_DEV_BEARER_TOKEN` environment variable. * `scoring_logic.py`: Unwraps either envelope with `extract_scoring_target`, then scores the turn's last output message. * `sample_request_v2_agent_turn.json`: A complete V2 agent turn request. * `register_remote_scorer.py --agent-turn`: Publishes a `RemoteScorer` and activates a monitor for completed agent turns. * `trigger_test_agent_turn.py`: Logs one turn with `weave.conversation.log_turn`. To run the endpoint locally and send it a V2 agent turn request without Weave, start the app, then send the sample request: ```bash lines theme={null} python3 -m venv .venv && source .venv/bin/activate python -m pip install -r requirements.txt export REMOTE_SCORER_DEV_BEARER_TOKEN="dev-token" uvicorn remote_scorer_app:app --host 127.0.0.1 --port 8000 ``` ```bash lines theme={null} curl -sS http://127.0.0.1:8000/score \ -H "Authorization: Bearer dev-token" \ -H "Content-Type: application/json" \ -H "Idempotency-Key: local-contract-check" \ -H "X-Correlation-ID: local-contract-check" \ -H "X-Weave-Schema-Version: 2" \ --data @sample_request_v2_agent_turn.json ``` The sample requires Weave `0.53.0` or later. A local run verifies only the contract. ## Test the signal Before you test, deploy the endpoint at an HTTPS URL that's on the allowed hosts, and register it. Log one completed turn, then check the **Signals** tab. Scoring is asynchronous, so the result appears after a delay. ```python lines theme={null} import uuid from opentelemetry import trace import weave from weave.conversation import Message, log_turn weave.init("[YOUR-TEAM]/[YOUR-PROJECT]") result = log_turn( conversation_id=f"sample-{uuid.uuid4().hex}", agent_name="sample-support-agent", system_instructions=["Answer the user accurately and concisely."], messages=[Message.user("What are your support hours?")], output_messages=[ Message.assistant( "Our support team is available Monday through Friday, 9am to 5pm Eastern." ) ], ) # weave.init exports spans from a background thread. Flush before a short script exits. trace.get_tracer_provider().force_flush(30_000) print(result.conversation_id, result.trace_ids) ``` Your endpoint receives a V2 request with `scoring_target.type` set to `agent_turn`, and Weave records the result as feedback on that turn. ## Troubleshooting | Symptom | What to check | | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Remote scorer** doesn't appear in the **New signal** drawer | Remote scoring is enabled in the owning organization's **Remote scoring** settings, or by your deployment administrator. | | The endpoint URL is rejected | The scorer host, and the token endpoint host for OAuth, are in the allowed hosts. Check for a port mismatch, a non-HTTPS URL, or a private or internal address. | | The endpoint returns `401` or `403` | The secret name, the OAuth client credentials, audience, and scope, and your endpoint's token validation. | | The endpoint returns `400` for agent turns | The endpoint checks the envelope version and accepts `scoring_target.type` `agent_turn` at payload version `1`. | | No result appears on the **Signals** tab | Scoring is asynchronous. Confirm the turn matched the signal's filters and sample rate, and that the turn's root span ended. | | The turn matched the signal but no tag or rating appears | Weave records nothing for a failed scoring attempt. Check your endpoint's request logs for the `X-Correlation-ID` header. If the endpoint received no request, the URL failed the allowed host rules or the secret didn't resolve. If it received a request, the response failed validation. | | No result after a `200` response | The response's `schema_version` is `2`, and `result` uses one of the three structured shapes. |