> ## Documentation Index > Fetch the complete documentation index at: https://wb-21fd5541-remote-scorers-ag.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Score calls with remote scorers > Score W&B Weave calls with a scorer that runs on your own infrastructure and records the result as call feedback. export const GitHubLink = ({url, compact = false}) => {compact ? "View source" : <> GitHub source } ; A remote scorer is a scorer that runs on your infrastructure instead of inside W\&B Weave. When a monitor selects a call, the Weave scoring worker sends the call to your HTTPS endpoint in an HTTP `POST` and records the response as feedback on that call. Use a remote scorer when your scoring logic can't run in Weave, such as a policy check against internal data or a model you host yourself. This page covers remote scorers for calls traced with `@weave.op`. To score agent turns in the Agents view, see [Score agent turns with a remote scorer](/weave/guides/tracking/remote-scorer-signals). You configure remote scorers with the Python SDK or the Weave UI. The TypeScript SDK doesn't include `RemoteScorer`. ## How remote scoring works A call is scored in the following sequence: 1. A call to a monitored op ends. 2. The scoring worker finds the active monitors whose operations include that op, then applies each monitor's filter and sampling rate. 3. For each `RemoteScorer` on a matching monitor, the worker builds a `schema_version: 1` request that contains the call, resolves the scorer's credentials, checks the endpoint URL against the allowed hosts, and sends the `POST`. 4. The worker validates the response and writes the result as feedback on the call. It also records the scoring attempt as a call, whether or not the attempt succeeded. A remote scorer runs only through a monitor. `weave.Evaluation` and `call.apply_scorer()` don't invoke it, and calling `RemoteScorer.score()` in your code raises `NotImplementedError`. Each selected call produces one request, sent once. If the request times out or gets no response, Weave doesn't retry it. The default timeout is 30 seconds. ## Enable remote scoring Remote scoring is off until it's enabled for your organization or deployment, and the scoring worker calls a scorer endpoint only if its host is on an allowlist. How you enable it depends on your deployment type. **Multi-tenant Cloud** An organization admin or billing admin enables remote scoring for the organization: 1. Open `https://wandb.ai/account-settings/[ORG]/settings`, replacing `[ORG]` with the organization that owns your project. 2. Select the **Remote scoring** tab. 3. Turn on **Enable remote scoring**. 4. Under **Allowed hosts**, click **Add host** and enter each host that remote scorers may call. Saving with remote scoring enabled requires at least one host. Leave the port blank to allow any port on that host. 5. Click **Save settings**. **Dedicated Cloud** Ask W\&B to enable remote scoring for your deployment and configure its allowed hosts. **Self-Managed** If you run W\&B Weave in a [W\&B Self-Managed](/platform/hosting/hosting-options/self-managed) deployment, set these values on the Weave trace server Helm chart under `weave_workers.remoteScoring`. The chart renders them as environment variables on each worker that can call a remote scorer: the online evaluation worker, the call scoring worker, and the agent scoring worker. | Helm value | Environment variable | Default | Effect | | ------------------------------- | ------------------------------------------------------------------ | ------- | ---------------------------------------------------------------------------------------------------------------- | | `enabled` | `WF_SCORING_WORKER_REMOTE_SCORING_ENABLED` | `false` | Turns on outbound requests to remote scorers. When `false`, no remote scorer runs, regardless of other settings. | | `httpTimeoutSeconds` | `WF_SCORING_WORKER_REMOTE_HTTP_TIMEOUT_SECONDS` | `30` | Timeout for each request to a scorer endpoint. | | `allowedHosts` | `WF_SCORING_WORKER_REMOTE_SCORER_ALLOWED_HOSTS` | empty | Operator allowlist of `host` or `host:port` entries. Comma-separated in the environment variable. | | `validateHosts` | `WF_SCORING_WORKER_REMOTE_SCORER_VALIDATE_HOSTS` | `true` | Enforces the allowlist and the address checks under **Allowed host rules**. | | `allowInsecureHttp` | `WF_SCORING_WORKER_REMOTE_SCORER_ALLOW_INSECURE_HTTP` | `false` | Permits `http://` endpoint URLs. | | `requireStructuredResultSchema` | `WF_SCORING_WORKER_REMOTE_SCORER_REQUIRE_STRUCTURED_RESULT_SCHEMA` | `true` | Rejects responses whose `result` isn't in the structured score format. | **Allowed host rules** The scoring worker checks every scorer endpoint URL, and separately the OAuth token endpoint URL when a scorer uses OAuth, against these rules: * An entry matches an exact host, with an optional port. An entry without a port allows any port on that host. * An entry that starts with `*.` matches subdomains at any depth, but not the domain itself. `*.corp.example.com` matches `a.corp.example.com` and `a.b.corp.example.com`, not `corp.example.com`. The suffix after `*.` must contain at least two labels, so `*.com` is rejected. A wildcard can't be combined with an IP address. * When both an operator allowlist and an organization allowlist exist, the URL must satisfy both. An empty operator allowlist adds no restriction. When no allowlist exists at all, the worker rejects every host. * Loopback, private, internal, and cloud metadata addresses are rejected. * HTTPS is required unless the deployment permits insecure HTTP. * Redirects aren't followed. ## Build the scorer endpoint Your endpoint accepts a JSON `POST` from Weave and returns a JSON score. ### Request Weave sends one HTTP `POST` per scored target to the scorer's endpoint URL, with these headers: | Header | Value | | ------------------------ | ---------------------------------------------------------------------------------- | | `Content-Type` | `application/json` | | `Authorization` | `Bearer [TOKEN]` | | `Idempotency-Key` | A key derived from the scored target, the monitor version, and the scorer version. | | `X-Correlation-ID` | A correlation ID for this request. | | `X-Weave-Schema-Version` | `1` or `2`, equal to `schema_version` in the body. | Weave might deliver the same scoring attempt more than once. If your endpoint has side effects, use `Idempotency-Key` to deduplicate. The key is stable for one request version, so a V1 and a V2 request for the same call carry different keys. Every request body has these top-level fields: | Field | Description | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | `schema_version` | Integer, `1` or `2`. | | `scoring_call_id`, `scoring_trace_id` | Identifiers for this scoring attempt. | | `monitor` | `name` and `version_digest` of the monitor that selected the target. | | `scorer` | `name`, `ref`, and optional `config`. `config` is the mapping set on the `RemoteScorer`, passed through unchanged. | | `triggered_at` | Optional ISO 8601 timestamp. | Weave omits an optional field with no value rather than sending it as `null`. Weave might add optional fields to a version without changing its number, so ignore fields you don't recognize. Request and response bodies are limited to 1 MiB each and contain JSON text only, never images, audio, or video. A target that exceeds these limits isn't sent, so it isn't scored. Each request carries one target. For a call, `schema_version` is `1` and the scored call is at the top level under `original_call`: | Field | Description | | ------------------------ | --------------------------------- | | `project_id` | The project, as `entity/project`. | | `call_id`, `trace_id` | Identifiers of the scored call. | | `op_name` | The op ref of the scored call. | | `inputs` | The call's inputs. | | `output` | The call's output. | | `started_at`, `ended_at` | ISO 8601 timestamps. | ```json lines theme={null} { "schema_version": 1, "original_call": { "project_id": "entity/project", "call_id": "018f8d6c-8d5f-7000-8000-000000000001", "trace_id": "018f8d6c-8d5f-7000-8000-000000000002", "op_name": "weave:///entity/project/op/sample_remote_scorer_target:*", "inputs": { "message": "test message for scoring" }, "started_at": "2026-06-08T12:00:00+00:00", "ended_at": "2026-06-08T12:00:01+00:00", "output": { "reply": "received: test message for scoring", "status": "ok" } }, "scoring_call_id": "018f8d6c-8d5f-7000-8000-000000000003", "scoring_trace_id": "018f8d6c-8d5f-7000-8000-000000000004", "monitor": { "name": "example_remote_scorer_monitor", "version_digest": "monitor-version-digest" }, "scorer": { "name": "example_remote_scorer", "ref": "weave:///entity/project/object/example_remote_scorer:scorer-version-digest", "config": { "example_threshold": 0.8 } }, "triggered_at": "2026-06-08T12:00:02+00:00" } ``` ### Response Return HTTP `200` with a JSON object that has two fields: * `schema_version`: Integer equal to the request's `schema_version`. * `result`: One score object, a list of score objects, or an object of the form `{"scores": [...]}`. A score object has these fields: | Field | Required | Description | | ------------ | -------- | ------------------------------------------------------------------------------------------ | | `value` | Yes | A tag, as a string of at most 36 characters, or a rating, as a number from `0.0` to `1.0`. | | `reason` | No | A string that explains the score. | | `confidence` | No | A number from `0.0` to `1.0`. | Weave treats any non-`200` response as a scorer failure and records no feedback for that attempt. Weave doesn't follow redirects, so a redirect is also a failure. Weave doesn't parse the body of an error response. Return `4xx` for requests your endpoint never accepts and `5xx` for temporary problems. For a call request, the response's `schema_version` is `1`. This response returns one rating and one tag: ```json lines theme={null} { "schema_version": 1, "result": [ { "value": 1.0, "reason": "Message is 32 characters; concise messages score best.", "confidence": 1.0 }, { "value": "concise", "reason": "Message length category is concise.", "confidence": 0.9 } ] } ``` ## Authenticate requests from Weave Weave authenticates to your endpoint with a bearer token. The request carries no W\&B credential. The token proves to your endpoint that the request came from Weave, not the reverse. Each `RemoteScorer` uses one of two modes: | Mode | What Weave does | What you configure | | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | `oauth_client_credentials` | Requests a token from your OAuth token endpoint with the client credentials grant, sending the client ID and secret with HTTP Basic authentication (`client_secret_basic`), then sends that token to your scorer endpoint. | The token endpoint URL, the client ID, the name of the secret that holds the client secret, and an optional scope. | | `static_bearer` | Sends a fixed token to your scorer endpoint. | The name of the secret that holds the token. | Store the client secret or bearer token in the secret store of the team that owns the project before you register the scorer. The `RemoteScorer` configuration holds only the secret name. The scoring worker resolves the value at scoring time. The Weave UI also offers **Deployment default**, which stores no authentication configuration for the scorer and relies on a fallback credential set by the deployment operator. Multi-tenant Cloud sets no fallback credential. Use one of the two explicit modes. ## Register a remote scorer A remote scorer is a `RemoteScorer` object attached to a monitor. Create it with the Python SDK or in the Weave UI. ### Python SDK Publish a `RemoteScorer`, then activate a `Monitor` that lists it in `scorers` and names the ops to score in `op_names`. `endpoint_url` is required. `config` and `auth_config` are optional. ```python lines theme={null} import weave from weave.flow.monitor import Monitor from weave.scorers.remote_scorer import ( OAuthClientCredentialsConfig, RemoteScorer, StaticBearerAuthConfig, ) weave.init("my-team/my-weave-project") # Static bearer token, read from the team secret named WEAVE_REMOTE_SCORER_BEARER_TOKEN auth = StaticBearerAuthConfig( mode="static_bearer", bearer_secret_name="WEAVE_REMOTE_SCORER_BEARER_TOKEN", ) # OAuth client credentials instead: # auth = OAuthClientCredentialsConfig( # mode="oauth_client_credentials", # token_endpoint_url="https://idp.example.com/oauth2/token", # client_id="weave-remote-scorer", # client_secret_name="WEAVE_REMOTE_SCORER_CLIENT_SECRET", # scope="score:remote", # ) scorer = RemoteScorer( name="policy_remote_scorer", endpoint_url="https://scoring.example.com/weave/score", config={"threshold": 0.9}, # Sent to your endpoint as scorer.config auth_config=auth, ) weave.publish(scorer, name="policy_remote_scorer") monitor = Monitor( name="policy_remote_monitor", scorers=[scorer], op_names=["generate_response"], # Op names in this project, or full weave:/// op refs sampling_rate=1.0, ) monitor.activate() ``` `monitor.activate()` publishes the monitor as active and expands bare op names into full op refs for the current project. ### Weave UI Create the remote scorer as part of a new monitor: 1. In the Weave project sidebar, click **Monitors**, then **+ New Monitor**. 2. Configure the monitor's name, operations, filter, and sampling rate. For the fields, see [Set up custom monitors](/weave/guides/evaluation/custom-monitors#create-a-monitor-in-weave). 3. Under **Scorer**, select **New remote scorer**. 4. Under **Remote scorer configuration**, configure the following fields: * **Scorer Name**: Must start with a letter or number. Can contain letters, numbers, hyphens, and underscores. * **Scoring endpoint URL**: The URL Weave sends the `POST` to. * **Authentication**: **Static bearer** or **OAuth client credentials**. Then enter the secret name and, for OAuth, the **Token endpoint URL**, **Client ID**, and optional **Scope**. Secret fields take team secret names, not values. * **Config (JSON, optional)**: A JSON object passed to your endpoint as `scorer.config`. 5. Click **Create monitor**. ## Test the scorer Trigger a scored call and confirm the result: 1. Call the monitored op at least once. 2. Confirm that your endpoint received a request. Scoring is asynchronous, so the request arrives after the call ends. 3. In the **Traces** tab, open the call and check its feedback. Your endpoint receives a V1 request with the call under `original_call`, and Weave records the result as feedback on that call. ## Troubleshooting | Symptom | What to check | | ------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Remote scorer options don't appear in the UI | Remote scoring is enabled in the owning organization's **Remote scoring** settings, or by your deployment administrator. | | The endpoint URL is rejected | The scorer host, and the token endpoint host for OAuth, are in the allowed hosts. Check for a port mismatch, a non-HTTPS URL, or a private or internal address. | | The endpoint returns `401` or `403` | The secret name, the OAuth client credentials, audience, and scope, and your endpoint's token validation. | | The token request succeeds but scoring fails, or the reverse | Each URL separately. The token endpoint and scorer endpoint are validated independently and can use different hosts. | | No feedback appears | Scoring is asynchronous. Confirm the call matched the monitor's operation, filter, and sampling rate. | | No feedback after a `200` response | The response's `schema_version` equals the request's, and `result` uses one of the three structured shapes. | ## Sample code The [`examples/remote_scorer`](https://github.com/wandb/weave/tree/master/examples/remote_scorer) directory in the `weave` repository is the reference implementation of this contract and the source of truth for the sample code. It's deployment independent: the Python service shows the contract, not a hosting recommendation. The sample includes the following files: * `remote_scorer_app.py`: A FastAPI app with `GET /health` and `POST /score` that accepts V1 and V2 requests. * `scoring_logic.py`: Framework-independent request parsing and scoring, written to be copied into your own service. * `auth.py`: A development-only bearer token check against the `REMOTE_SCORER_DEV_BEARER_TOKEN` environment variable. * `register_remote_scorer.py`: Publishes a `RemoteScorer` and activates a `Monitor` for an op, or for agent turns with `--agent-turn`. * `trigger_test_trace.py` and `trigger_test_agent_turn.py`: Create a traced call or an agent turn that a monitor can select. * `sample_request.json`, `sample_request_v2_call.json`, and `sample_request_v2_agent_turn.json`: Example request bodies. To run the endpoint locally and check the contract without Weave, start the app, then send it the sample request: ```bash lines theme={null} python3 -m venv .venv && source .venv/bin/activate python -m pip install -r requirements.txt export REMOTE_SCORER_DEV_BEARER_TOKEN="dev-token" uvicorn remote_scorer_app:app --host 127.0.0.1 --port 8000 ``` ```bash lines theme={null} curl -sS http://127.0.0.1:8000/score \ -H "Authorization: Bearer dev-token" \ -H "Content-Type: application/json" \ -H "Idempotency-Key: local-contract-check" \ -H "X-Correlation-ID: local-contract-check" \ -H "X-Weave-Schema-Version: 1" \ --data @sample_request.json ``` The sample requires Weave `0.53.0` or later. A local run verifies only the contract. The Weave scoring worker rejects loopback addresses, and hosted deployments don't permit insecure HTTP. To test with Weave, deploy the endpoint at an HTTPS URL on the allowed hosts and register it as described in [Register a remote scorer](#register-a-remote-scorer).