Score calls with remote scorers

A remote scorer is a scorer that runs on your infrastructure instead of inside W&B Weave. When a monitor selects a call, the Weave scoring worker sends the call to your HTTPS endpoint in an HTTP POST and records the response as feedback on that call. Use a remote scorer when your scoring logic can’t run in Weave, such as a policy check against internal data or a model you host yourself. This page covers remote scorers for calls traced with @weave.op. To score agent turns in the Agents view, see Score agent turns with a remote scorer. You configure remote scorers with the Python SDK or the Weave UI. The TypeScript SDK doesn’t include RemoteScorer.

How remote scoring works

A call is scored in the following sequence:

A call to a monitored op ends.
The scoring worker finds the active monitors whose operations include that op, then applies each monitor’s filter and sampling rate.
For each RemoteScorer on a matching monitor, the worker builds a schema_version: 1 request that contains the call, resolves the scorer’s credentials, checks the endpoint URL against the allowed hosts, and sends the POST.
The worker validates the response and writes the result as feedback on the call. It also records the scoring attempt as a call, whether or not the attempt succeeded.

A remote scorer runs only through a monitor. weave.Evaluation and call.apply_scorer() don’t invoke it, and calling RemoteScorer.score() in your code raises NotImplementedError. Each selected call produces one request, sent once. If the request times out or gets no response, Weave doesn’t retry it. The default timeout is 30 seconds.

Enable remote scoring

Remote scoring is off until it’s enabled for your organization or deployment, and the scoring worker calls a scorer endpoint only if its host is on an allowlist. How you enable it depends on your deployment type. Multi-tenant Cloud An organization admin or billing admin enables remote scoring for the organization:

Open https://wandb.ai/account-settings/[ORG]/settings, replacing [ORG] with the organization that owns your project.
Select the Remote scoring tab.
Turn on Enable remote scoring.
Under Allowed hosts, click Add host and enter each host that remote scorers may call. Leave the port blank to allow any port on that host. Saving with remote scoring enabled requires at least one host.
Click Save settings.

Dedicated Cloud Ask W&B to enable remote scoring for your deployment and configure its allowed hosts. Self-Managed Set these values on the Weave trace server Helm chart under weave_workers.remoteScoring. The chart renders them as environment variables on each worker that can call a remote scorer: the online evaluation worker, the call scoring worker, and the agent scoring worker.

Helm value	Environment variable	Default	Effect
`enabled`	`WF_SCORING_WORKER_REMOTE_SCORING_ENABLED`	`false`	Turns on outbound requests to remote scorers. When `false`, no remote scorer runs, regardless of other settings.
`httpTimeoutSeconds`	`WF_SCORING_WORKER_REMOTE_HTTP_TIMEOUT_SECONDS`	`30`	Timeout for each request to a scorer endpoint.
`allowedHosts`	`WF_SCORING_WORKER_REMOTE_SCORER_ALLOWED_HOSTS`	empty	Operator allowlist of `host` or `host:port` entries. Comma-separated in the environment variable.
`validateHosts`	`WF_SCORING_WORKER_REMOTE_SCORER_VALIDATE_HOSTS`	`true`	Enforces the allowlist and the address checks under Allowed host rules.
`allowInsecureHttp`	`WF_SCORING_WORKER_REMOTE_SCORER_ALLOW_INSECURE_HTTP`	`false`	Permits `http://` endpoint URLs.
`requireStructuredResultSchema`	`WF_SCORING_WORKER_REMOTE_SCORER_REQUIRE_STRUCTURED_RESULT_SCHEMA`	`true`	Rejects responses whose `result` isn’t in the structured score format.

Allowed host rules The scoring worker checks every scorer endpoint URL, and separately the OAuth token endpoint URL when a scorer uses OAuth, against these rules:

An entry matches an exact host, with an optional port. An entry without a port allows any port on that host.
An entry that starts with *. matches subdomains at any depth, but not the apex domain. *.corp.example.com matches a.corp.example.com and a.b.corp.example.com, not corp.example.com. The suffix after *. must contain at least two labels, so *.com is rejected. A wildcard can’t be combined with an IP address.
When both an operator allowlist and an organization allowlist exist, the URL must satisfy both. An empty operator allowlist adds no restriction. When no allowlist exists at all, the worker rejects every host.
Loopback, private, internal, and cloud metadata addresses are rejected.
HTTPS is required unless the deployment permits insecure HTTP.
Redirects aren’t followed.

Build the scorer endpoint

Your endpoint accepts a JSON POST from Weave and returns a JSON score.

Request

Weave sends one HTTP POST per scored target to the scorer’s endpoint URL, with these headers:

Header	Value
`Content-Type`	`application/json`
`Authorization`	`Bearer [TOKEN]`
`Idempotency-Key`	A key derived from the scored target, the monitor version, and the scorer version.
`X-Correlation-ID`	A correlation ID for this request.
`X-Weave-Schema-Version`	`1` or `2`, equal to `schema_version` in the body.

Weave might deliver the same scoring attempt more than once. If your endpoint has side effects, use Idempotency-Key to deduplicate. The key is stable for one request version, so a V1 and a V2 request for the same call carry different keys. Every request body has these top-level fields:

Field	Description
`schema_version`	Integer, `1` or `2`.
`scoring_call_id`, `scoring_trace_id`	Identifiers for this scoring attempt.
`monitor`	`name` and `version_digest` of the monitor that selected the target.
`scorer`	`name`, `ref`, and optional `config`. `config` is the mapping set on the `RemoteScorer`, passed through unchanged.
`triggered_at`	Optional ISO 8601 timestamp.

Weave omits an optional field with no value rather than sending it as null. Weave might add optional fields to a version without changing its number, so ignore fields you don’t recognize. Request and response bodies are limited to 1 MiB each and contain JSON text only, never images, audio, or video. A target that exceeds these limits isn’t sent, so it isn’t scored. Each request carries one target. For a call, schema_version is 1 and the scored call is at the top level under original_call:

Field	Description
`project_id`	The project, as `entity/project`.
`call_id`, `trace_id`	Identifiers of the scored call.
`op_name`	The op ref of the scored call.
`inputs`	The call’s inputs.
`output`	The call’s output.
`started_at`, `ended_at`	ISO 8601 timestamps.

{
  "schema_version": 1,
  "original_call": {
    "project_id": "entity/project",
    "call_id": "018f8d6c-8d5f-7000-8000-000000000001",
    "trace_id": "018f8d6c-8d5f-7000-8000-000000000002",
    "op_name": "weave:///entity/project/op/sample_remote_scorer_target:*",
    "inputs": {
      "message": "test message for scoring"
    },
    "started_at": "2026-06-08T12:00:00+00:00",
    "ended_at": "2026-06-08T12:00:01+00:00",
    "output": {
      "reply": "received: test message for scoring",
      "status": "ok"
    }
  },
  "scoring_call_id": "018f8d6c-8d5f-7000-8000-000000000003",
  "scoring_trace_id": "018f8d6c-8d5f-7000-8000-000000000004",
  "monitor": {
    "name": "example_remote_scorer_monitor",
    "version_digest": "monitor-version-digest"
  },
  "scorer": {
    "name": "example_remote_scorer",
    "ref": "weave:///entity/project/object/example_remote_scorer:scorer-version-digest",
    "config": {
      "example_threshold": 0.8
    }
  },
  "triggered_at": "2026-06-08T12:00:02+00:00"
}

Response

Return HTTP 200 with a JSON object that has two fields:

schema_version: Integer equal to the request’s schema_version.
result: One score object, a list of score objects, or an object of the form {"scores": [...]}.

A score object has these fields:

Field	Required	Description
`value`	Yes	A tag, as a string of at most 36 characters, or a rating, as a number from `0.0` to `1.0`.
`reason`	No	A string that explains the score.
`confidence`	No	A number from `0.0` to `1.0`.

Weave treats any non-200 response as a scorer failure and records no feedback for that attempt. Weave doesn’t follow redirects, so a redirect is also a failure. Weave doesn’t parse the body of an error response. Return 4xx for requests your endpoint never accepts and 5xx for temporary problems. For a call request, the response’s schema_version is 1. This response returns one rating and one tag:

{
  "schema_version": 1,
  "result": [
    {
      "value": 1.0,
      "reason": "Message is 32 characters; concise messages score best.",
      "confidence": 1.0
    },
    {
      "value": "concise",
      "reason": "Message length category is concise.",
      "confidence": 0.9
    }
  ]
}

Authenticate requests from Weave

Weave authenticates to your endpoint with a bearer token. The request carries no W&B credential. The token proves to your endpoint that the request came from Weave, not the reverse. Each RemoteScorer uses one of two modes:

Mode	What Weave does	What you configure
`oauth_client_credentials`	Requests a token from your OAuth token endpoint with the client credentials grant, sending the client ID and secret with HTTP Basic authentication (`client_secret_basic`), then sends that token to your scorer endpoint.	The token endpoint URL, the client ID, the name of the secret that holds the client secret, and an optional scope.
`static_bearer`	Sends a fixed token to your scorer endpoint.	The name of the secret that holds the token.

Store the client secret or bearer token in the secret store of the team that owns the project before you register the scorer. The RemoteScorer configuration holds only the secret name. The scoring worker resolves the value at scoring time. The Weave UI also offers Deployment default, which stores no authentication configuration for the scorer and relies on a fallback credential set by the deployment operator. Multi-tenant Cloud sets no fallback credential. Use one of the two explicit modes.

Register a remote scorer

A remote scorer is a RemoteScorer object attached to a monitor. Create it with the Python SDK or in the Weave UI.

Python SDK

Publish a RemoteScorer, then activate a Monitor that lists it in scorers and names the ops to score in op_names. endpoint_url is required. config and auth_config are optional.

import weave
from weave.flow.monitor import Monitor
from weave.scorers.remote_scorer import (
    OAuthClientCredentialsConfig,
    RemoteScorer,
    StaticBearerAuthConfig,
)

weave.init("my-team/my-weave-project")

# Static bearer token, read from the team secret named WEAVE_REMOTE_SCORER_BEARER_TOKEN
auth = StaticBearerAuthConfig(
    mode="static_bearer",
    bearer_secret_name="WEAVE_REMOTE_SCORER_BEARER_TOKEN",
)

# OAuth client credentials instead:
# auth = OAuthClientCredentialsConfig(
#     mode="oauth_client_credentials",
#     token_endpoint_url="https://idp.example.com/oauth2/token",
#     client_id="weave-remote-scorer",
#     client_secret_name="WEAVE_REMOTE_SCORER_CLIENT_SECRET",
#     scope="score:remote",
# )

scorer = RemoteScorer(
    name="policy_remote_scorer",
    endpoint_url="https://scoring.example.com/weave/score",
    config={"threshold": 0.9},  # Sent to your endpoint as scorer.config
    auth_config=auth,
)
weave.publish(scorer, name="policy_remote_scorer")

monitor = Monitor(
    name="policy_remote_monitor",
    scorers=[scorer],
    op_names=["generate_response"],  # Op names in this project, or full weave:/// op refs
    sampling_rate=1.0,
)
monitor.activate()

monitor.activate() publishes the monitor as active and expands bare op names into full op refs for the current project.

Weave UI

Create the remote scorer as part of a new monitor:

In the Weave project sidebar, click Monitors, then + New Monitor.
Configure the monitor’s name, operations, filter, and sampling rate. For the fields, see Set up custom monitors.
Under Scorer, select New remote scorer.
Under Remote scorer configuration, configure the following fields:
- Scorer Name: Must start with a letter or number. Can contain letters, numbers, hyphens, and underscores.
- Scoring endpoint URL: The URL Weave sends the POST to.
- Authentication: Static bearer or OAuth client credentials. Then enter the secret name and, for OAuth, the Token endpoint URL, Client ID, and optional Scope. Secret fields take team secret names, not values.
- Config (JSON, optional): A JSON object passed to your endpoint as scorer.config.
Click Create monitor.

Test the scorer

Trigger a scored call and confirm the result:

Call the monitored op at least once.
Confirm that your endpoint received a request. Scoring is asynchronous, so the request arrives after the call ends.
In the Traces tab, open the call and check its feedback.

Your endpoint receives a V1 request with the call under original_call, and Weave records the result as feedback on that call.

Troubleshooting

Symptom	What to check
Remote scorer options don’t appear in the UI	Remote scoring is enabled in the owning organization’s Remote scoring settings, or by your deployment administrator.
The endpoint URL is rejected	The scorer host, and the token endpoint host for OAuth, are in the allowed hosts. Check for a port mismatch, a non-HTTPS URL, or a private or internal address.
The endpoint returns `401` or `403`	The secret name, the OAuth client credentials, audience, and scope, and your endpoint’s token validation.
The token request succeeds but scoring fails, or the reverse	Each URL separately. The token endpoint and scorer endpoint are validated independently and can use different hosts.
No feedback appears	Scoring is asynchronous. Confirm the call matched the monitor’s operation, filter, and sampling rate.
No feedback after a `200` response	The response’s `schema_version` equals the request’s, and `result` uses one of the three structured shapes.

Sample code

The examples/remote_scorer directory in the weave repository is the reference implementation of this contract and the source of truth for the sample code. It’s deployment independent: the Python service shows the contract, not a hosting recommendation. The sample includes the following files:

remote_scorer_app.py: A FastAPI app with GET /health and POST /score that accepts V1 and V2 requests.
scoring_logic.py: Framework-independent request parsing and scoring, written to be copied into your own service.
auth.py: A development-only bearer token check against the REMOTE_SCORER_DEV_BEARER_TOKEN environment variable.
register_remote_scorer.py: Publishes a RemoteScorer and activates a Monitor for an op, or for agent turns with --agent-turn.
trigger_test_trace.py and trigger_test_agent_turn.py: Create a traced call or an agent turn that a monitor can select.
sample_request.json, sample_request_v2_call.json, and sample_request_v2_agent_turn.json: Example request bodies.

To run the endpoint locally and check the contract without Weave, start the app, then send it the sample request:

python3 -m venv .venv && source .venv/bin/activate
python -m pip install -r requirements.txt
export REMOTE_SCORER_DEV_BEARER_TOKEN="dev-token"
uvicorn remote_scorer_app:app --host 127.0.0.1 --port 8000

curl -sS http://127.0.0.1:8000/score \
  -H "Authorization: Bearer dev-token" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: local-contract-check" \
  -H "X-Correlation-ID: local-contract-check" \
  -H "X-Weave-Schema-Version: 1" \
  --data @sample_request.json

The sample requires Weave 0.53.0 or later. A local run verifies only the contract. The Weave scoring worker rejects loopback addresses, and hosted deployments don’t permit insecure HTTP. To test with Weave, deploy the endpoint at an HTTPS URL on the allowed hosts and register it as described in Register a remote scorer.

Get Started

Guides

Cookbooks

Reference

Details & Support

How remote scoring works

Enable remote scoring

Build the scorer endpoint

Request

Response

Authenticate requests from Weave

Register a remote scorer

Python SDK

Weave UI

Test the scorer

Troubleshooting

Sample code

​How remote scoring works

​Enable remote scoring

​Build the scorer endpoint

​Request

​Response

​Authenticate requests from Weave

​Register a remote scorer

​Python SDK

​Weave UI

​Test the scorer

​Troubleshooting

​Sample code

How remote scoring works

Enable remote scoring

Build the scorer endpoint

Request

Response

Authenticate requests from Weave

Register a remote scorer

Python SDK

Weave UI

Test the scorer

Troubleshooting

Sample code