Understand tokens - Writer AI Studio

Tokenization

Tokenization is the process of breaking input text down into smaller units. Depending on the tokenization method the model uses, a single word can be represented by one token or may be broken down into multiple sub-word tokens. Here’s how different text types typically tokenize:

"hello world"           → ["hello", " world"]           # 2 tokens
"API"                   → ["API"]                       # 1 token
"tokenization"          → ["token", "ization"]          # 2 tokens
"www.writer.com"        → ["www", "writer", "com"]      # 3 tokens

Input and output token usage

Every API call consumes tokens for both the prompt you send and the response generated:

# Example API call
response = client.chat.completions.create(
  model="palmyra-x5",
  messages=[
    {"role": "user", "content": "Write a product description for a cozy sweater"},  # ~8 tokens
  ],
  max_tokens=150,  # Maximum tokens to include in the response
)

# Total token usage = prompt tokens + response tokens

Inspect token usage

You can inspect the usage information from the API response to see the token usage for each response.

The prompt tokens reported in the response include the tokens used for the model’s system prompt.

Python

from writerai import Writer

# Initialize the client. If you don't pass the `api_key` parameter,
# the client looks for the `WRITER_API_KEY` environment variable.
client = Writer()

response = client.chat.chat(
  model="palmyra-x5",
  messages=[{"role": "user", "content": "Write a product description for a cozy sweater"}],
  max_tokens=150,
)

# Log token usage to understand patterns
print(f"Prompt tokens: {response.usage.prompt_tokens}")
print(f"Response tokens: {response.usage.completion_tokens}")
print(f"Total tokens: {response.usage.total_tokens}")

JavaScript

import { Writer } from "writer-sdk";

// Initialize the Writer client. If you don't pass the `apiKey` parameter,
// the client looks for the `WRITER_API_KEY` environment variable.
const client = new Writer();

const response = await client.chat.chat({
  model: "palmyra-x5",
  messages: [
    { role: "user", content: "Write a product description for a cozy sweater" },
  ],
  maxTokens: 150,
});

// Log token usage to understand patterns
console.log(`Prompt tokens: ${response.usage.prompt_tokens}`);
console.log(`Response tokens: ${response.usage.completion_tokens}`);
console.log(`Total tokens: ${response.usage.total_tokens}`);

You can view your organization’s overall token usage and spend in the Consumption dashboard.