Limits

The evaluations API enforces limits to keep the service stable under large or high-fan-out evaluations. They are set well above typical usage, so most evaluations never reach them.

This page lists every limit, what happens when you exceed it, and how to restructure an evaluation that does.

A request over the body limit is rejected with 413. An oversized field or record is rejected with 422 at validation. To store large blobs (documents, images, long transcripts), reference them by URL or ID instead of inlining them into a record.

How to fix each error:

• 413: the whole request is over the body limit. Lower upload_batch_size to send fewer records per request.
• 422 on a JSON field: the 1 MB limit applies to each field of each record, not to a batch. A smaller batch doesn't help: shrink the oversized field itself.

evaluation.run() and evaluation.optimize() upload results while they execute. The SDK handles upload failures in two ways:

• Invalid payload (400, 413, or 422): the run stops at the first rejected upload instead of processing the remaining records, and is marked failed. The error is raised as is, whatever max_concurrency.
• Any other failure (network errors, 5xx, and other 4xx such as 401, 403, 404, 408, 409, or 429): the SDK retries the upload. A batch that still fails after 5 attempts fails the run. To change that number, set the MISTRAL_EVAL_UPLOAD_MAX_FAILED_ATTEMPTS environment variable (an integer of at least 1).

When an upload gets no answer (a timeout or a 504, for example), the server may already have stored the batch. The SDK re-sends the same batch unchanged and treats a 409 on that re-send as "already uploaded", so the run continues. Such a batch is never failed fast or counted against the retry limit.

Calls to the evaluations API use a timeout of at least 60 seconds. A longer Mistral(timeout_ms=...) still applies.

A failed API call raises ObservabilityAPIError with status_code, error_code, and details. For validation failures (422), the message names the offending fields, and details lists them as {"loc": [...], "msg": "..."} entries:

API request failed with status 422 (VALIDATION_ERROR): records.0.input: JSON field exceeds the maximum allowed size of 1000000 bytes

API request failed with status 422 (VALIDATION_ERROR): records.0.input: JSON field exceeds the maximum allowed size of 1000000 bytes

The submitted values are never included in the message, in details, or in the SDK debug logs.

from mistralai.evaluations import ObservabilityAPIError

try:
    ...
except ObservabilityAPIError as error:
    print(error.status_code, error.error_code, error.details)

from mistralai.evaluations import ObservabilityAPIError

try:
    ...
except ObservabilityAPIError as error:
    print(error.status_code, error.error_code, error.details)

A 5xx on an upload raises ObservabilityUnknownOutcomeError: the server may or may not have stored the data. It's a subclass of ObservabilityAPIError, so an except ObservabilityAPIError block also catches it.

An evaluation run expands into records × generations × evaluators score rows, so each factor is capped:

Exceeding any of these returns a 422: either a "list too long" error or, for the per-run totals, a validation error naming the current count and the maximum.

The SDK batches uploads for you, so you normally don't manage this. If you call the API directly, keep each request within these bounds.

The evaluators-per-run cap (250) is the one most likely to affect an evaluation that scores many nodes or branches of a workflow graph.

A common anti-pattern is one evaluator per node. For example, an accuracy metric split into accuracy/MyWorkflow::node_1, accuracy/MyWorkflow::node_2, and so on. This attaches hundreds of evaluators to every record, even though each record only exercises a handful of nodes. Most of the resulting score rows are empty placeholders.

Instead, use one evaluator per metric, with the node as a dimension in the score's metadata (or in the score name). Emit a score only for the nodes a record actually touches. You keep the same per-node breakdown, stay well under the cap, and avoid creating thousands of empty scores.

These values are server-side defaults that protect shared infrastructure. If your use case needs a higher limit, contact your Mistral representative instead of working around it.

• API reference: evaluation.run() parameters, including upload_batch_size.
• Workflow evaluation plugin limits: the extra limits that apply when an evaluation runs inside a workflow.