## Model Delegation With model delegation, you can use another model as a tool in a [chat completion](https://dev.writer.com/home/chat-completion). The predefined LLM tool allows you to delegate specific tasks to another model available to your organization, including [external models](https://dev.writer.com/home/external-models) from providers like AWS Bedrock, Microsoft Azure, and NVIDIA NIM. For example, in a chat application using Palmyra X5, you can delegate focused analysis tasks to a different model. This guide shows you how to set up model delegation with the Writer API. After completing these steps, you can route specific tasks within a chat completion to the model best suited to handle them. You need an API key to access the Writer API. Get an API key by following the steps in the [API quickstart](https://dev.writer.com/home/quickstart). We recommend setting the API key as an environment variable in a `.env` file with the name `WRITER_API_KEY`. ## Tool structure Use the LLM tool to delegate specific tasks to another model when using the [chat endpoint](https://dev.writer.com/api-reference/completion-api/chat-completion). Using [tool calling](https://dev.writer.com/home/tool-calling), you can specify the model you want to use for a given task. When the primary chat model calls the LLM tool based on the user’s input, it signals it in the chat API response. To use the LLM tool, add it to the `tools` array in your `chat-completion` endpoint request. The LLM tool object has the following structure: | Parameter | Type | Description | | --- | --- | --- | | `type` | `string` | The type of tool, which is `llm` for LLM tool | | `function` | `object` | An object containing the tool’s description and model | | `function.description` | `string` | A description of what the model will be used for. | | `function.model` | `string` | The ID of the model to be used for this tool. This can be a [Palmyra model](https://dev.writer.com/home/models) or an [external model](https://dev.writer.com/home/external-models) configured for your organization. | To help the model understand when to use the tool, follow these best practices for the `function.description` parameter: - Indicate that the tool is a function that invokes an LLM - Specify the model’s purpose and capabilities - Describe when the tool should be used An example description for a tool using an external model: > “A function that invokes the LLM identified by the given model for detailed analysis. Any user request requiring in-depth analysis should use this tool.” ### Example tool configurations ```json { "tools": [ { "type": "llm", "function": { "description": "A function that will invoke the llm identified by the given model for detailed analysis and reporting. Any user request requiring in-depth analysis should use this tool.", "model": "anthropic.claude-3-sonnet-20240229-v1:0" } } ] } ``` ### Response format When a chat completion uses the LLM tool, the response from the LLM tool is in the `llm_data` object. The `llm_data` object contains the following fields: | Parameter | Type | Description | | --- | --- | --- | | `prompt` | string | The prompt used by the LLM tool. | | `model` | string | The ID of the model used by the LLM tool. | Below is an example of the full response to a chat completion request that uses the LLM tool with an external model. ```json { "id": "1234", "object": "chat.completion", "choices": [ { "index": 0, "finish_reason": "stop", "message": { "content": "The recommended daily intake of calcium for a 30-year-old woman is 1,000 mg per day.", "role": "assistant", "tool_calls": null, "graph_data": { "sources": null, "status": null, "subqueries": null }, "llm_data": { "prompt": "What is the recommended daily intake of calcium for a 30-year-old woman?", "model": "anthropic.claude-3-sonnet-20240229-v1:0" }, "image_data": null, "refusal": null }, "logprobs": null } ], "created": 1741970653, "model": "palmyra-x5", "usage": { "prompt_tokens": 259, "total_tokens": 305, "completion_tokens": 46, "prompt_token_details": null, "completion_tokens_details": null }, "system_fingerprint": "v1", "service_tier": null } ``` ## Usage example Here’s an example of how to use the LLM tool in your application. This example delegates detailed questions to an external model. ### Create a tools array containing an LLM tool To use the LLM tool, create a `tools` array that specifies the model you want to use. ```json { "tools": [ { "type": "llm", "function": { "description": "A function that will invoke the llm identified by the given model for detailed analysis. Any user request requiring in-depth analysis should use this tool.", "model": "anthropic.claude-3-sonnet-20240229-v1:0" } } ] } ``` ### Send the request using chat completions Add the tools array to the chat endpoint call along with your array of messages. Setting `tool_choice` to `auto` allows the model to choose when to use the LLM tool, based on the user’s question and the description of the tool. This example streams the response as the model generates it. ```bash curl --location 'https://api.writer.com/v1/chat' \ --header 'Content-Type: application/json' \ --header "Authorization: Bearer $WRITER_API_KEY" \ --data '{ "model": "palmyra-x5", "temperature": 0.7, "messages": [ { "role": "user", "content": "What is the recommended daily intake of calcium for a 30-year-old woman?" } ], "tool_choice": "auto", "tools": [ { "type": "llm", "function": { "description": "A function that will invoke the llm identified by the given model for detailed analysis. Any user request requiring in-depth analysis should use this tool.", "model": "anthropic.claude-3-sonnet-20240229-v1:0" } } ], "stream": true }' ``` By following this guide, you can delegate specific tasks to another model within your chat applications.