Chat API token counting

max_tokens parameter

The max_tokens parameter in the ConversationTokenBufferMemory class is used to limit the number of tokens.

Output token limit

The models documentation mentions 4096 output tokens for many models.

Counting tokens

Counting tokens for chat API calls To see how many tokens are in a text string without making an API call use.

Rate limit

Your rate limit is calculated as the maximum of max_tokens and the estimated number of tokens based on.

Context length

The token count of your prompt plus max_tokens cannot exceed the models context length.

Questions about max_tokens

The max_tokens parameter in the chat completion endpoint raises questions about its..