A large language model (LLM) is the central reasoning engine behind Hermes Agent. It processes in­struc­tions, evaluates in­ter­me­di­ate results, decides on the next action and generates struc­tured tool calls. The quality, context length and tool-calling re­li­a­bil­i­ty of the selected model directly affect how reliably and au­tonomous­ly Hermes can complete complex tasks.

Key Takeaways

The key points for choosing an LLM for Hermes Agent:

  • Minimum re­quire­ments: at least 64,000 tokens of context and reliable struc­tured tool calling — Hermes rejects models with smaller context windows at startup.
  • Open­Router: one API key for many models and high flex­i­bil­i­ty, at the cost of an ad­di­tion­al in­ter­me­di­ary layer.
  • Direct provider API (for example Anthropic): a clearer setup with more control over costs and data pro­tec­tion, but less flex­i­bil­i­ty.
  • Running locally: Ollama as the easiest way to start, vLLM for GPU servers with many parallel requests, LM Studio for testing through a graphical interface.
  • Local models: Qwen3-Coder 30B-A3B for code and terminal tasks, Llama 3.3 70B for language-heavy workflows — Llama 3.3 70B needs con­sid­er­ably more hardware.

What re­quire­ments must an LLM for Hermes Agent meet?

Not every chat model is au­to­mat­i­cal­ly suitable for an au­tonomous agent. While a simple chatbot often handles only a single question-and-answer exchange, Hermes Agent uses LLMs to execute multi-step workflows. The model plans an action, calls a tool, receives a result, evaluates it, and then decides on the next step.

For these workflows, two prop­er­ties are par­tic­u­lar­ly important:

  1. a suf­fi­cient­ly large context window,
  2. reliable support for struc­tured tool calling.

Size of the context window

The context window shows how much in­for­ma­tion a model can process at one time. It is measured in tokens. Tokens are small pieces of text. Depending on the language and content, one token can represent part of a word, a short word or a punc­tu­a­tion mark.

The context used by Hermes includes more than just your original task and previous responses. System in­struc­tions, de­scrip­tions of available tools, loaded skills, stored in­for­ma­tion, in­ter­me­di­ate results and tool outputs also take up space in the context window. This means Hermes Agent needs a model with a context length of at least 64,000 tokens. Models with smaller context windows are rejected at startup because they do not provide enough space for multi-step tool workflows.

When choosing a model for Hermes Agent, you should not aim for the minimum limit of 64,000 tokens. A model with some ad­di­tion­al capacity above this level is rec­om­mend­ed. A model with 128,000 tokens or more provides even more room. Hermes can au­to­mat­i­cal­ly compress long con­ver­sa­tions, but this does not replace having a suf­fi­cient­ly large context window from the start. Com­pres­sion helps keep long sessions stable, but it cannot make an un­suit­able 8K or 32K model work for complex workflows.

If the available context window is too small, older in­for­ma­tion may be lost during the process or sum­ma­rized too early. The model may then repeat steps it has already completed, assign a tool result in­cor­rect­ly or lose track of the original goal.

Tool calling

Tool calling allows an LLM to do more than generate a written answer. It can also create a struc­tured function call that tells Hermes which tool to use and what action to perform. For example, the model can decide to read a file, run a terminal command or start a web search.

A sim­pli­fied tool call might look something like this in­ter­nal­ly:

{
  "name": "web_search",
  "arguments": {
    "query": "latest Hermes Agent documentation"
  }
}
json

Hermes rec­og­nizes this struc­tured call, runs the requested tool and sends the result back to the model. The model then evaluates the result and decides whether any further actions are needed.

If a model only describes the required action in plain language instead of gen­er­at­ing a struc­tured tool call, the tool is not executed au­to­mat­i­cal­ly. This means native or properly im­ple­ment­ed tool calling is sig­nif­i­cant­ly more reliable than asking a model to generate JSON‑like text on its own. Claude supports defined tools through struc­tured tool_use blocks, while OpenAI-com­pat­i­ble servers represent function calls using the tools, tool_choice, and tool_calls fields.

A large context window alone is not enough. A model can handle a 128,000-token context window and still not be suitable if it fre­quent­ly generates incorrect function names, forgets required pa­ra­me­ters or mixes tool calls with written answers.

Running Hermes Agent in the cloud: Open­Router or a direct provider API?

For cloud de­ploy­ments, there are two main options. You can connect Hermes Agent to different models through a model ag­gre­ga­tor like Open­Router or connect it directly to a model provider’s API, such as Anthropic’s.

Open­Router: many models through a single API key

Open­Router provides a single API for more than 400 models from different providers. You can filter supported models by context length, price, speed, tool-calling ca­pa­bil­i­ty and other criteria.

The biggest advantage of using Open­Router is flex­i­bil­i­ty. You only need one Open­Router API key and can switch between models such as Claude, Qwen, Gemini, Llama, or DeepSeek. This allows you to use a powerful model for complex tasks and a more af­ford­able model for simpler back­ground tasks. Open­Router can also route requests through different providers that offer the same model. You can sort options by price, through­put, or latency, choose preferred providers, exclude specific providers, and limit requests to endpoints with zero data retention.

A simple Hermes Agent con­fig­u­ra­tion might look like this:

model:
  provider: openrouter
  default: anthropic/claude-sonnet-5

provider_routing:
  sort: latency
  require_parameters: true
  data_collection: deny
yaml

require_parameters: true is par­tic­u­lar­ly useful for agents. This setting tells Open­Router to use only providers that support all pa­ra­me­ters required by the request. These can include tool calling and struc­tured outputs, for example.

You do not store the API key directly in this file. Instead, store it in ~/.hermes/.env:

OPENROUTER_API_KEY=your_OpenRouter_API_Key
bash
Note

Hermes separates secret cre­den­tials from normal con­fig­u­ra­tion settings. API keys belong in the .env file, while the model, provider, endpoint, and routing rules are stored in the central config.yaml.

The downside of Open­Router is the ad­di­tion­al in­ter­me­di­ary layer. Exact latency and behaviour can vary depending on the backend you choose. For sensitive data, you should also keep in mind that both Open­Router and the model provider handling the request may be involved in pro­cess­ing the data.

Direct provider APIs: con­nect­ing models without an ag­gre­ga­tor

Instead of con­nect­ing Hermes Agent through a model ag­gre­ga­tor like Open­Router, you can connect it directly to a model provider’s API. Examples of providers include Anthropic, OpenAI and Google. With this type of direct in­te­gra­tion, Hermes sends requests straight to the selected provider. You need your own account with that provider, a suitable API key, and, if required, a billing setup.

You also configure this con­nec­tion in the central config.yaml file. The provider name and model name you need to enter depend on the provider you choose. A sim­pli­fied con­fig­u­ra­tion might look like this:

model:
  provider: anthropic
  default: claude-sonnet-5
yaml

The API key should also not be stored directly in the con­fig­u­ra­tion file.

Note

If you use a provider other than Anthropic, update the provider name, model ID, and en­vi­ron­ment variable name to match that provider.

The main benefit of a direct provider API is a clearer setup. You know exactly which company processes the model request, what pricing applies, and which data pro­tec­tion re­quire­ments and retention policies you need to consider. Because there is no ad­di­tion­al in­ter­me­di­ary or routing layer, a direct con­nec­tion can also provide more con­sis­tent response times. However, it is not au­to­mat­i­cal­ly faster than Open­Router for every request. Actual latency still depends on the model you choose, the context length, the provider’s current load, and the location of its servers.

A downside of direct in­te­gra­tion is less flex­i­bil­i­ty. If you want to switch to a model from another provider, you often need another account, a new API key, and an updated Hermes con­fig­u­ra­tion. You also need to set up automatic switching between providers yourself using fallback rules.

This means direct provider APIs are a good fit for real-world ap­pli­ca­tions where you have already chosen a specific model and want precise control over costs, data pro­tec­tion, support and how the system behaves tech­ni­cal­ly. In contrast, Open­Router is usually more practical if you want to compare models, switch between providers fre­quent­ly or manage multiple providers through a single interface.

Running locally with Ollama, vLLM, and LM Studio

When running Hermes Agent locally with an LLM, the model runs on your own hardware or a server you manage yourself. Your prompts and model responses do not need to be sent to an external LLM provider. This can be par­tic­u­lar­ly useful when working with internal documents, source code, customer data, or ap­pli­ca­tions that need to run without sending data to external services.

Note

Running locally does not au­to­mat­i­cal­ly mean that all data privacy re­quire­ments are met. You still need to consider factors such as the legal basis for pro­cess­ing, access rights, logging, retention periods, security measures, and any ad­di­tion­al cloud tools you use. Ap­pro­pri­ate technical and or­ga­ni­za­tion­al measures are needed through­out the entire data pro­cess­ing process. For example, if Hermes uses an external web search while running with a local LLM, data can still leave the local system.

Ollama: the easiest way into local models

Ollama is a good choice for beginners. The tool downloads models, runs them locally, and provides an OpenAI-com­pat­i­ble endpoint. Hermes can access this endpoint at http://localhost:11434/v1.

A suitable model can be loaded as follows:

ollama pull qwen3-coder:30b
bash

Then, start Ollama with the required context window size:

OLLAMA_CONTEXT_LENGTH=65536 ollama serve
bash

In Hermes, enter the local endpoint:

model:
  provider: custom
  default: qwen3-coder:30b
  base_url: http://localhost:11434/v1
  context_length: 65536
yaml

The value in the Hermes con­fig­u­ra­tion tells Hermes how much context it can expect to use. It does not au­to­mat­i­cal­ly increase the context window provided by the Ollama server. You must also set the context length on the server using OLLAMA_CONTEXT_LENGTH or an Ollama model file. This value cannot be increased later through the OpenAI-com­pat­i­ble chat endpoint. You can check the currently loaded context window with the following command:

ollama ps
bash

Ollama supports tool calling through its OpenAI-com­pat­i­ble API. However, the selected model itself must support a com­pat­i­ble tool calling format. Qwen-3, Llama-3.x, Mistral, and Hermes models are examples of model families that support this.

vLLM: powerful self-hosting for GPU servers

vLLM is aimed at pro­fes­sion­al users and people who are com­fort­able with technical setups. The software also provides an OpenAI-com­pat­i­ble API, but it is designed for high through­put, parallel requests and efficient use of GPU servers.

A Qwen model can be started as follows, for example:

vllm serve Qwen/Qwen3-Coder-Next \
  --port 8000 \
  --tensor-parallel-size 2 \
  --max-model-len 65536 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder
bash

The pa­ra­me­ters serve different purposes:

  • --max-model-len 65536 sets a context window above the minimum required by Hermes Agent.
  • --enable-auto-tool-choice allows the model to decide whether it needs to use a tool.
  • --tool-call-parser qwen3_coder converts the model-specific tool calls into struc­tured API calls.

For Llama 3.3, vLLM uses the llama3_json tool parser. You should also specify a suitable JSON chat template. This template defines how Hermes passes available tools, required pa­ra­me­ters and pre­vi­ous­ly executed tool calls to the model.

Why long contexts require a lot of VRAM

When running a model locally, not only the model weights need to fit into GPU memory. The server also needs memory for the KV cache. It stores in­for­ma­tion from pre­vi­ous­ly processed tokens so the model can reuse them when gen­er­at­ing the next parts of a response. The larger the context window, the more ad­di­tion­al memory is needed.

Quantized models mainly reduce the memory needed for model weights. However, they do not au­to­mat­i­cal­ly solve the memory demands of a large context window. This means on limited hardware it can make more sense to use a smaller model with reliable tool calling and a true 64K context window than a larger model whose context has to be partially offloaded to the CPU.

LM Studio: local models through a graphical interface

LM Studio provides a graphical interface for Windows, macOS and Linux. You can search for, download, configure, and run models through a local API server. The OpenAI-com­pat­i­ble endpoint runs at http://localhost:1234/v1 by default.

The server can be started from the Developer section of the app or from the command line:

lms server start
lms ls
lms load <QWEN3_CODER_MODEL_ID> --context-length 65536
bash

Hermes has its own LM Studio option in the model assistant. Hermes preserves the context length of an already-loaded LM Studio model. For an unloaded model, LM Studio applies its own context setting unless context_length is ex­plic­it­ly con­fig­ured in Hermes. You can also control and adjust the context length in the model settings. LM Studio supports tool and function calling through its OpenAI-com­pat­i­ble endpoints. Models with native tool calling generally provide more reliable results than models that rely on a general workaround format. For initial local tests, LM Studio’s graphical interface makes it es­pe­cial­ly easy to use. For a con­tin­u­ous­ly running headless server with many parallel requests, however, vLLM is usually the better choice.

VPS Hosting (Virtual Private Servers)
More scale, same great pricing — with the new VPS+
  • NEW: VM cloning, load balancing, block storage, and more
  • Ideal for agentic AI, LLMs, and au­toma­tion at scale
  • Free 24/7 expert support and 99.99%+ uptime
  • Unlimited traffic and no setup fees

Which local models work well with Hermes Agent?

Hermes Agent supports different LLMs. The right choice depends on your specific use case.

Qwen3-Coder

Qwen3-Coder is Qwen’s model family spe­cial­ized in pro­gram­ming and agent-based software tasks. It natively supports a context window of 262,144 tokens. On suitable servers with enough memory, Qwen3-Coder can be extended to up to one million tokens using YaRN. For local Hermes Agent in­stal­la­tions, Qwen3-Coder 30B-A3B-Instruct is a strong option. The model has a total of 30.5 billion pa­ra­me­ters, with 3.3 billion activated at a time. This ar­chi­tec­ture combines a rel­a­tive­ly man­age­able model size with strong ca­pa­bil­i­ties for code gen­er­a­tion, tool calling and agent-based workflows.

Llama 3.3 70B

Llama 3.3 70B Instruct is the newer and more powerful al­ter­na­tive to Llama 3.1 70B. The model supports a 128,000-token context window and includes a format for struc­tured tool calls. This means it generally meets Hermes Agent’s re­quire­ments, provided the local model server provides at least 64,000 tokens and uses the ap­pro­pri­ate tool-calling template. Llama 3.3 is par­tic­u­lar­ly suitable for Hermes Agent workflows involving general text pro­cess­ing, mul­ti­lin­gual tasks, research and tasks that require strong language ca­pa­bil­i­ties alongside tool calling. However, with 70 billion pa­ra­me­ters, the model requires sig­nif­i­cant hardware resources.

LLMs for Hermes Agent compared

Provider/model Location Minimum context length Tool-calling stability Cost
Anthropic direct / Claude Sonnet 5 or 4.6 Anthropic Cloud Hermes: 64,000; model: up to 1 million Very high; native struc­tured tools High, usage-based token costs
Open­Router / Claude Sonnet or another tool-enabled model Cloud through ag­gre­ga­tor At least 64,000; depends on model High to very high; also depends on the provider handling the request Variable from low to high
Ollama or LM Studio / Qwen3-Coder 30B-A3B Own computer or server Hermes: at least 64,000; model natively 262,144 High; optimized for tool calling and agent-based code workflows No API costs; medium to high hardware costs
vLLM or LM Studio / Llama 3.3 70B Own high-per­for­mance server Hermes: at least 64,000; model natively 128K High with a suitable chat template and server-side tool-call handling; for vLLM, use llama3_json. High RAM and VRAM re­quire­ments

The tool calling stability rating is not a guarantee. It assumes that the model, context length, chat template, tool parser and server pa­ra­me­ters are con­fig­ured correctly and work together as expected.

IONOS CLOUD AI Model Hub
Your gateway to a secure mul­ti­modal AI platform
  • One platform for the most powerful AI models
  • Fair and trans­par­ent token-based pricing
  • No vendor lock-in with open source

Which setup fits your use case?

  • For absolute beginners: Open­Router is the easiest way to get started with a cloud-based setup. A single API key lets you try different models without separate provider setups.

  • For complex cloud-based web au­toma­tions: A direct Anthropic con­nec­tion with a current Claude Sonnet model is a par­tic­u­lar­ly reliable option. It offers a very large context window and stable tool calling.

  • For cloud systems used for real workloads with high avail­abil­i­ty: Use a direct provider API as the primary provider and configure Open­Router as a fallback.

  • For local cron jobs and recurring scripts: Ollama with a suitable Qwen or Llama model is an easy way to start running models locally. The main ad­van­tages are local data pro­cess­ing and no token costs.

  • For code and terminal tasks: Qwen3-Coder 30B-A3B is a strong choice for local setups. The model offers a native 256K context window, reliable tool calling and a rel­a­tive­ly efficient MoE ar­chi­tec­ture.

  • For a local multi-user server or pro­duc­tion en­vi­ron­ment: vLLM is the most powerful self-hosting option in this com­par­i­son. It offers high through­put and good GPU uti­liza­tion but requires sig­nif­i­cant­ly more technical expertise.

  • For testing local models with a graphical interface: LM Studio is a good choice if you want to download and configure models without relying heavily on the command line.

  • For low-resource hardware: Llama 3.2 with 1B or 3B pa­ra­me­ters can handle simple tasks. For au­tonomous web research, extensive file editing or long-running learning loops, a larger model is rec­om­mend­ed.

Con­clu­sion: which LLM is the best choice for Hermes Agent?

The best setup for Hermes Agent depends on what matters most to you, such as ease of use, data privacy, cost, speed or maximum re­li­a­bil­i­ty. Re­gard­less of the provider, the model should offer at least 64,000 tokens of effective context, struc­tured tool calling and enough reasoning ca­pa­bil­i­ty for multi-step decisions.

For a simple way to get started in the cloud, Open­Router is par­tic­u­lar­ly flexible. For demanding au­toma­tions used in real-world op­er­a­tions, a direct Claude con­nec­tion provides stable tool calling and a very large context window. For local ex­per­i­ments and simple cron jobs, Ollama is the easiest option, while vLLM is best suited for powerful GPU servers and workloads with many parallel requests.

Reviewer

Go to Main Menu