Skip to content

Generative AI engines

A generative AI engine is a configured connection to a large language model service. Every AI task, Auto QA evaluation, and topic extraction runs through one of these engines. Engine management is an operator-only responsibility: customers do not bring their own LLM — the operator provisions the engines, and tenants simply consume them.

Engines are managed on the Engines tab of Administration › Conversation Analytics › AI Tasks.

Engine types

Two vendor types are supported:

Type Default model Notes
OpenAI-compatible API gpt-4o-mini The API Server URL is configurable, so any provider exposing an OpenAI-compatible endpoint can be used — OpenAI itself, a cloud gateway, or a self-hosted model server.
Google Vertex AI gemini-2.5-flash Authenticates with a service-account key file or an access token; project ID and location are configurable.

There is no other vendor integration — anything reachable through an OpenAI-compatible endpoint is covered by the first type.

Common settings per engine: name, status (Enabled/Disabled), model name, sampling temperature, presence and frequency penalties, and vendor-specific options (reasoning effort for OpenAI-compatible engines, thinking budget for Vertex AI). API keys and other secrets are stored encrypted. After saving, use Test a Connection to verify credentials and reachability.

Token cost ratios

Each engine carries an input token cost ratio (default 1.0) and an output token cost ratio (default 4.0). Metered usage is calculated as input tokens × input ratio + output tokens × output ratio, so tenants' token limits and the usage dashboards account for output tokens being more expensive than input. Adjust the ratios to match the pricing of the model behind the engine.

Visibility and curation

The Visibility column shows each engine's scope:

  • Global — the engine is available to all tenants; it appears in every tenant's task configuration wherever an engine can be selected.
  • Local — the engine is bound to one tenant (the Tenant column names it).

Because global engines are visible across the platform, keep the list curated: use clear, meaningful names and disable or delete experiments. Per-tenant engine assignment — creating a Local engine for one tenant — is supported but uncommon; the typical deployment runs all tenants on a small set of global engines.

Selecting which engine runs the work

  • The AI Assistant job carries the default engine (its AI Assist engine setting) used for every task it executes.
  • An individual AI task can pin its own engine, overriding the job default for that task only.

When replacing an engine (for example, moving to a new model), add the new engine, repoint the job (and any task-level pins), verify results in the Playground, and only then disable the old engine — a disabled or deleted engine that a job still references makes the job fail.