Async environment
curl --request POST \
  --url https://model-{model_id}.api.baseten.co/environments/{env_name}/async_predict \
  --header 'Authorization: <authorization>' \
  --header 'Content-Type: application/json' \
  --data '{
  "model_input": "<any>",
  "webhook_endpoint": "<string>",
  "priority": 123,
  "max_time_in_queue_seconds": 123,
  "inference_retry_config": "<any>"
}'
{
  "request_id": "<string>"
}

​
Parameters

​
model_id
string
required
The ID of the model you want to call.
​
env_name
string
required
The name of the model’s environment you want to call.

​
Headers

​
Authorization
string
required
Your Baseten API key, formatted with prefix Api-Key (e.g. {"Authorization": "Api-Key abcd1234.abcd1234"}).

​
Body

There is a 256 KiB size limit to /async_predict request payloads.
​
model_input
json
required
JSON-serializable model input.
​
webhook_endpoint
string
default:"null"
Baseten does not store model outputs. If webhook_endpoint is empty, your model must save prediction outputs so they can be accessed later.
URL of the webhook endpoint. We require that webhook endpoints use HTTPS. Both HTTP/2 and HTTP/1.1 protocols are supported.
​
priority
integer
default:0
Priority of the request. A lower value corresponds to a higher priority (e.g. requests with priority 0 are scheduled before requests of priority 1).priority is between 0 and 2, inclusive.
​
max_time_in_queue_seconds
integer
default:600
Maximum time a request will spend in the queue before expiring.max_time_in_queue_seconds must be between 10 seconds and 72 hours, inclusive.
​
inference_retry_config
json
Exponential backoff parameters used to retry the model predict request.

​
Response

​
request_id
string
required
The ID of the async request.
{
  "request_id": "<string>"
}

​
Rate limits

Two types of rate limits apply when making async requests:
  • Calls to the /async_predict endpoint are limited to 200 requests per second.
  • Each organization is limited to 50,000 QUEUED or IN_PROGRESS async requests, summed across all deployments.
If either limit is exceeded, subsequent /async_predict requests will receive a 429 status code. To avoid hitting these rate limits, we advise:
  • Implementing a backpressure mechanism, such as calling /async_predict with exponential backoff in response to 429 errors.
  • Monitoring the async queue size metric. If your model is accumulating a backlog of requests, consider increasing the number of requests your model can process at once by increasing the number of max replicas or the concurrency target in your autoscaling settings.