Default Timeout for Amazon Bedrock invoke_model

CK 3788
Created at Oct 22, 2025 16:24:10

737 0 0

The concept of "default timeout" for Amazon Bedrock invoke_model primarily refers to the client-side timeout of the AWS SDKs (like Boto3) or the AWS CLI, rather than a configurable service-side timeout on Bedrock itself. Bedrock will process your request until completion or until it hits internal service limits, but your client might stop waiting for the response.

Let's break down how to manage these timeouts.

Default Timeout for `invoke_model`

TThe default timeout is generally determined by the underlying HTTP client used by the AWS SDKs and CLI. For Boto3 and the AWS CLI, the default connection and read timeouts are typically set to 60 seconds (for connect_timeout) and a longer duration (often 60-120 seconds or more, sometimes infinite for read_timeout to wait for the initial response or the entire response to complete), depending on the specific SDK version and environment.

It's crucial to understand:

CConnect Timeout: This refers to the duration the client waits to establish a connection to the Bedrock endpoint.
RRead Timeout: This refers to the duration the client waits for the first byte of the response after establishing a connection, as well as how long it waits for subsequent chunks of a streaming response.

IIf the time taken for a model's generation exceeds your client's configured timeout, your client will raise a timeout error, even though Bedrock may still be processing the request in the background.

How to Set it Up (General)

To set up timeouts, you configure the HTTP client used by the specific tool (AWS CLI or Boto3 API). This involves passing configuration parameters when making the request or initializing the client.

AWS CLI

For the AWS CLI, you can set connections and read timeouts using global parameters.

--cli-connect-timeoutThe maximum socket connection time is in seconds.
--cli-read-timeoutThe maximum socket read time is measured in seconds.

Example:
To invoke a Bedrock model and allow up to 120 seconds for connection and 300 seconds (5 minutes) for the response:

aws bedrock-runtime invoke-model \
    --model-id anthropic.claude-v2 \
    --body '{ "prompt": "\n\nHuman: What is the capital of France?\n\nAssistant:", "max_tokens_to_sample": 100 }' \
    --content-type application/json \
    --accept application/json \
    --cli-connect-timeout 120 \
    --cli-read-timeout 300 \
    output.json

Note: If you are using the streaming API (invoke-model-with-response-stream), the read-timeout will apply to the initial connection and then to the time between receiving successive chunks of data.

Boto3 API (Python SDK)

With Boto3, you configure timeouts when creating the Bedrock client using a Config object from botocore.

connect_timeoutThe maximum socket connection time is in seconds.
read_timeoutThe maximum socket read time is measured in seconds.

Example for Non-Streaming invoke_model:

import boto3
import json
from botocore.config # import ConfigDefine your custom timeout configuration
# Allow 120 seconds to establish a connection
# Allow 300 seconds (5 minutes) to receive the full response
config = Config(
    connect_timeout=120,
    read_timeout=300
)

# Create a Bedrock Runtime client with the custom configuration
client = boto3.client(
    service_name='bedrock-runtime',
    region_name='us-east-1', # Replace with your desired region
    config=config
)

model_id = 'anthropic.claude-v2' # Example model

body = json.dumps({
    "prompt": "\n\nHuman: What is the capital of France?\n\nAssistant:",
    "max_tokens_to_sample": 100
})

try:
    response = client.invoke_model(
        body=body,
        modelId=model_id,
        accept='application/json',
        contentType='application/json'
    )
    response_body = json.loads(response.get('body').read())
    print(response_body['completion'])except Exception as e:
    print(f"Error invoking model: {e}")
    
    # Handle specific timeout errors if needed
    if "Read timed out" in str(e) or "Connect timed out" in str(e):
        print("The request timed out.")

Example for Streaming invoke_model_with_response_stream:

For streaming, read_timeout is particularly important as it dictates how long the client will wait for each chunk of data from the stream. If a model pauses generation for longer than read_timeout, your client will time out. It's often set to a higher value or even None (infinite wait for a chunk, but the connection itself might still have limits) for streaming use cases if you expect very long pauses.

import boto3
import json
from botocore.config import Configconfig = Config(
    connect_timeout=120,
    read_timeout=600 # Allow up to 10 minutes between stream chunks
)

client = boto3.client(
    service_name='bedrock-runtime',
    region_name='us-east-1',
    config=config
)
model_id = 'anthropic.claude-3-sonnet-20240229-v1:0' # Streaming model example
body = json.dumps({
    "messages": [
        {
            "role": "user",
            "content": "Tell me a very long story about a space-faring cat exploring the galaxy."
        }
    ],
    "max_tokens": 2000
})

try:
    response = client.invoke_model_with_response_stream(
        body=body,
        modelId=model_id,
        accept='application/json',
        contentType='application/json'
    )    stream = response.get('body')
    for event in stream:
        if 'chunk' in event:
            chunk = json.loads(event['chunk']['bytes'].decode('utf-8'))
            if chunk.get('type') == 'content_block_delta':
                text = chunk['delta']['text']
                print(text, end='') # Print without newline to simulate continuous output
            elif chunk.get('type') == 'message_stop':
                print("\n[Story End]")
    print("\nStreaming completed.")
except Exception as e:
    print(f"\nError invoking streaming model: {e}")
    if "Read timed out" in str(e) or "Connect timed out" in str(e):
        print("The request timed out.")

Important Considerations:

Client vs. Service Timeout: Always remember you are configuring the client's patience, not Bedrock's processing time. If If your client times out, the Bedrock invocation may still be running and will continue to consume resources.
Retry Logic: AWS SDKs often include built-in retry logic. Sometimes timeouts trigger these retries, but if long generation times consistently hit the timeout, the primary solution is to increase the timeout.
NeNetwork Latency: When setting timeouts, consider the network latency between your client and the AWS region.
Model Performance: Some models or very complex prompts can lead to longer generation times. Set Your timeouts are appropriately based on the expected model behavior.

Comments 0