Operator tips
For AI Agents: the Markdown version of this page is at https://ankole.agentbull.com/en-US/docs/tips/index.md. The documentation index is at https://ankole.agentbull.com/en-US/llms.txt.
This page collects practical operator habits that come up across everyday Ankole deployments—each short, grounded in how the runtime works, and much faster to read here than to learn during an incident.
Read logs in pretty mode locally
By default, the control plane emits structured JSON logs—ideal for log aggregators, hard to scan in a terminal. During local development, pipe JSON logs through the devkit pretty-printer:
bun run dev # in one terminal
bun run kit logs pretty < /path/to/log-stream # or pipe a log file through it
In production, ship raw JSON logs to your log collector and reserve kit logs pretty for local inspection.
Scope the log level before a bug hunt
ANKOLE_LOG_LEVEL defaults to info. Drop it to debug only while reproducing a specific bug, then restore info as soon as you finish—leaving a production deployment at debug generates heavy I/O and log noise. Valid values are debug | info | warning | error; Ankole rejects invalid values at boot rather than ignoring them silently.
Get the activation code without the terminal
If the first-visit setup page asks for the bootstrap activation code and you cannot see the bun run dev or control-plane container stdout:
bun run kit show bootstrap-activation-code # local
docker compose logs control-plane | grep "SETUP ACTIVATION CODE" # Compose
kubectl -n ankole logs deployment/ankole-control-plane -c control-plane | grep "SETUP ACTIVATION CODE" # Helm
Always read the code from these runtime sources rather than trying to inspect raw database tables.
Treat model-profile slots as dials
Each Agent’s seven built-in profile slots—primary, light, heavy, vision_fallback, web_search, web_fetch, and image_generate—work like independent dials:
- Point
primaryat a faster, cheaper model when an Agent mainly answers quick operational questions, or dial it up to a frontier reasoning model when accuracy outweighs latency and cost. - Bind
lightto a genuinely fast, low-cost model, since Ankole uses it for high-volume background classification and routing work. - Configure
vision_fallbackonly if the Agent receives images and itsprimarymodel lacks native vision; otherwise leave it empty. - Treat
web_searchandweb_fetchindependently—bind them only on Agents that need public web access. - Configure Brain retrieval across AppConfigure and Console → Model Profiles: the site-wide
brain.embedding_modelin AppConfigure powers shared vector search across all Pods, while thererankslot in Model Profiles (Site -> Pod -> ankole_brain) controls cross-encoder reranking. Leaving the embedding model empty disables vector search, and leavingrerankunset keeps reciprocal-rank fusion order; see Brain.
When an Agent feels sluggish in chat, its primary slot is often bound to a heavier model than the work that it actually performs requires.
Rotate a credential in Environment variables
Updates to Worker environment variables apply starting on the next turn rather than mutating processes mid-turn:
- Save the updated secret in Environment variables in the Console.
- Allow any currently running turn or workspace command to finish with its captured environment.
- Start a new turn to verify the rotated credential.
Never judge whether a credential rotation worked from a turn that was already running when you clicked save.
Reveal encrypted values only when necessary
To rotate a secret, simply paste the replacement value and save—you do not need to view the old secret first. Click Reveal only when diagnosing a configuration mismatch requires inspecting the currently stored value.
Recover from a stuck turn
A turn that appears stuck is almost always waiting on an upstream LLM provider or a long-running Worker command rather than deadlocked inside Ankole. Before cancelling anything:
- Inspect
/api/v2/ai-gateway/conversationsfor the turn’s latest model requests—a long pause there points to upstream provider latency. - Check the Agent’s Worker logs for an active shell or browser command, since a slow tool call looks identical to a quiet turn in chat.
- Only if the work is genuinely wedged, steer the Session or let its inactivity timeout fire. To stop a Background Agent Job cleanly, call
POST /api/v2/background-agent-jobs/:id/cancel, which records the stop and requests interruption of the active turn.
Cancelling turns impulsively is the fastest way to leave external side effects half-finished.
Disable a binding quietly
Calling DELETE /api/v2/signal-bindings/:id disables the routing rule instead of erasing it, keeping all credentials and routing settings recoverable. Use it whenever you want to silence an Agent in a channel temporarily—during credential rotation, a holiday freeze, or an incident—and re-enable it later with PATCH.
Back up before you upgrade, every time
Rolling back a Helm release or container tag never reverses Ecto database migrations. Before every upgrade, run pg_dump across the control-plane PostgreSQL database and every configured BASE_DB_URLS shard database, and back up your bootstrap secrets (ANKOLE_SECRET_BASE, RELEASE_COOKIE, and ANKOLE_FUNCTION_RUNTIME_TOKEN). See Backup and restore for the exact commands.
Adjust the correct durable document
Update AGENTS.md in Meta Ankole → Agents when an Agent’s scope, responsibilities, judgment, or tone needs adjustment, and update DESIGN.md when generated web pages, slides, documents, or charts need consistent visual styling. Because the two contracts own separate concerns, keep behavioral rules out of DESIGN.md and visual tokens out of AGENTS.md.
One robot per DingTalk agent
DingTalk enforces a strict one-to-one mapping: at most one enabled binding per Agent, and at most one Agent per clientId. When scaling out multiple Agents on DingTalk, register a separate DingTalk robot for each Agent—if you add a second enabled binding to the same Agent, Ankole rejects it with dingtalk_binding_already_exists, and if you reuse a clientId on another Agent, Ankole rejects it with dingtalk_app_already_bound. See Quickstart for setup steps.
Teams needs a public endpoint, always
Unlike Lark, Slack, and DingTalk—which connect outbound over persistent WebSockets and tolerate brief ingress blips—Microsoft Teams delivers messages exclusively via inbound Bot Framework HTTPS webhooks. Whenever a Teams bot goes silent, check your public HTTPS endpoint first: an expired TLS certificate, a DNS change, or a reverse-proxy outage will all look like a dead bot.
Next steps
- For the Console interface reference, read Console API reference.
- For environment knobs, read Environment variables.
- For the kit commands, read the kit CLI reference.