Upgrade from 1.0 to 2.0
For AI Agents: the Markdown version of this page is at https://ankole.agentbull.com/en-US/docs/upgrade-to-2/index.md. The documentation index is at https://ankole.agentbull.com/en-US/llms.txt.
This guide walks self-hosted deployments through upgrading from Ankole 1.0 (or a later 1.x release) to 2.0. Schedule a maintenance window and stop all 1.x writers before running the 2.0 migrations—reverting container images cannot undo completed database migrations.
Always rehearse the upgrade against a restored backup in an isolated staging environment before touching production, especially if you are upgrading from an early 1.x build, running custom infrastructure, or carrying local schema changes.
What changes
| Old 1.x layout | 2.0 layout | Operator action |
|---|---|---|
Pooled Workers and shared /agents |
One Worker and private /workspace per Agent |
Copy or adopt Agent Homes and assign a WorkerProvider |
| Installed Skills and Worker script automations in Home | Skills and Agent contracts in Meta Ankole (Ankole.AIAgent.Library PostgreSQL tables); files in Unified Drive; automation logic in published Applet Actions |
Import legacy installed-skills and attachments; let schema migrations separate custom Skills; export script Jobs and recreate them as Applet Jobs |
| Transient Codex state in Workers | Threads in /workspace/.codex; rebuildable cache in /var/cache/ankole/codex |
Preserve Home and attach a separate cache disk |
| In-process script sandboxes and 2 images | External stateless ankole-function-runtime (3 co-versioned images: control-plane, worker, function-runtime) |
Set ANKOLE_FUNCTION_RUNTIME_TOKEN and deploy all three matching images |
| Data and configuration without tenant ownership | Original data placed in the default tenant; Pod / Base storage on PostgreSQL 18+ (with optional BASE_DB_URLS shards) |
Upgrade to PostgreSQL 18+ and run schema and encrypted-config migrations with the original key |
LocalPassword sign-in and single admin group |
100% passwordless Authn (SiteCredential email_otp + enterprise SSO) and per-tenant pod_admin alongside siteadmin |
Pass credential preflight, configure SMTP (authn.email.mail or mix ankole.authn.mail.put) or SSO, and verify Pod admins |
Console at /console |
Site Admin Console at /site-admin-console (or ANKOLE_SITE_ADMIN_CONSOLE_PATH) |
Update bookmarks and reverse-proxy ingress rules |
If a late 1.x deployment already runs per-Agent Workers or drives, perform only the remaining steps—never overwrite existing Home or drive volumes with empty storage.
Before migration, export every agent_script Job definition, its source code and configuration, and any Run history you need. The Applet-only migration permanently deletes these Jobs, their Run history, bindings, and receipts; rolling back container images cannot restore them. After upgrading, publish replacement Applet Actions, create their Jobs, and reconnect schedules or webhooks before restoring automation traffic. The old automation-job CLI commands and automation_jobs.max_running_per_agent setting are removed. Recovery of deleted records requires restoring the pre-upgrade database backup.
Prepare the upgrade
- Record your current and target versions, image digests, Docker Compose project or Helm release name, installation ID, volume or PVC names, and Worker provider settings. All three images (
control-plane,worker, andfunction-runtime) must come from the exact same source revision; officialcontrol-planeimages already reference the matchingworkerimage digest. - Preserve your existing
ANKOLE_SECRET_BASE, database credentials, and deployment configuration, and generate independent random hex secrets forRELEASE_COOKIEandANKOLE_FUNCTION_RUNTIME_TOKENif they are not set yet. Never rotateANKOLE_SECRET_BASEduring the upgrade, because the migration needs the original key to decrypt and reseal stored credentials. - Clear all hard-stop preflight checks on the 1.x deployment before shutting it down:
- Reserved group name collisions (
siteadminandpod_admin): Check for any custom group namedsiteadminin the privileged tenant orpod_adminin any tenant (especiallycomputedgroups) and rename it before upgrading. Both names are reserved in 2.0, and a collision aborts the migration. If a tenant ends up with no unambiguous active administrator after migration, promote one withmix ankole.admin.recover HUMAN_UID OPERATOR REASON --identity-verified --tenant SLUG(omit--tenantto recoversiteadminin the privileged tenant). - Passwordless credential preflight (
Ankole.CredentialUpgrade.preflight!/1): Ankole runs this read-only check before converting credentials and aborts if any non-offboarded Human (active or suspended) lacks a login email, if two retained Principals in the same tenant share a normalized login email, or if one Principal owns multiple login emails. Resolve any reported conflicts in 1.x before retrying. During conversion, Ankole revokes browser password proofs (offline OIDC sessions tied to a unique browser proof retain their evidence; others prompt for re-authorization). Deployments that usedLocalPasswordmust configure either SMTP (authn.email.mailin the Console,mix ankole.authn.mail.put FILE.json, orAnkole.Release.put_authn_mail/1) or an allowed enterprise SSO provider so migratedemail_otpusers can sign in. - Console grant CEL
resourcevariable check: Legacy Console grants migrated tosite_console:*ortenant_console:*cannot referenceresourcein their CELcondition. Audit custom Console grants and remove anyresourcereferences beforehand.
- Reserved group name collisions (
- Quiesce new traffic and scheduled jobs: disable Agents, let running work complete, stop all 1.x Workers cleanly (including paused instances), and then stop every 1.x control-plane replica. Never delete Agents as a shortcut for stopping Workers.
- Confirm that PostgreSQL runs PostgreSQL 18+ (
server_version_num >= 180000withpg_searchinshared_preload_libraries, plus thepg_searchandvectorextensions). Take a coordinated backup of PostgreSQL (including anyBASE_DB_URLSshard databases), object/file storage, all Agent Homes, and deployment secrets, and retain that backup until sign-in, tenant membership, and OIDC flows are verified. Rehearse the 2.0 migrations on a restored database copy first to time your maintenance window. For Docker Compose Home migration, verify that the host has free disk space at least equal to the used bytes in the old shared volume. See Backup and restore.
During migration, Ankole also reseals tenant Identity Provider secrets, splits legacy shared Slack and Teams configurations, and resolves upstream application IDs—so ensure the migration container has the original ANKOLE_SECRET_BASE, valid platform credentials, and outbound network access to those chat platforms. If a step fails, repair the configuration named in the log and re-run; never copy raw ciphertext across scopes by hand.
Upgrade Docker Compose
Fetch the target release’s Compose deployment files—including compose.yaml, worker-providers.yaml, and migrate-agent-homes.sh—while keeping your existing .env and Compose project name. Pin ANKOLE_CONTROL_PLANE_IMAGE and ANKOLE_FUNCTION_RUNTIME_IMAGE to matching 2.0 release tags or digests (the control-plane image supplies the matching Worker image), set ANKOLE_DOCKER_GID, and preserve ANKOLE_WORKER_INSTALLATION.
The commands below assume the default fs drive backend; if you use S3, include the override file from Drive storage configuration and pass the exact same -f flags to every docker compose command, including the one-time file import.
Append RELEASE_COOKIE and ANKOLE_FUNCTION_RUNTIME_TOKEN to .env if either variable is missing:
grep -q '^RELEASE_COOKIE=' .env || printf 'RELEASE_COOKIE=%s\n' "$(openssl rand -hex 32)" >> .env
grep -q '^ANKOLE_FUNCTION_RUNTIME_TOKEN=' .env || printf 'ANKOLE_FUNCTION_RUNTIME_TOKEN=%s\n' "$(openssl rand -hex 32)" >> .env
In worker-providers.yaml, ensure the initial provider seed sets legacy_agents: true and uses the same provider_id as the Home migration service’s WORKER_PROVIDER (docker in the official files). Ankole imports this seed YAML only when the provider table is empty; make any subsequent provider changes in the Console.
Because 2.0 requires PostgreSQL 18+ (pg_search 0.26.4 and vector), complete a PostgreSQL major-version upgrade first if your server runs an older major version, and then update pg_search and run the Compose stack:
docker compose pull
docker compose stop
docker compose up -d postgresql
docker compose exec postgresql psql -U ankole -d ankole -v ON_ERROR_STOP=1 \
-c "ALTER EXTENSION pg_search UPDATE TO '0.26.4'"
docker compose up -d --force-recreate
docker compose ps -a migrate migrate-agent-homes
docker compose logs migrate migrate-agent-homes
Here, migrate runs schema and encrypted-configuration migrations, while migrate-agent-homes stops any containers still mounting the old shared volume, copies each existing Agent’s Home into its own private volume, and sets ownership to uid 1000. The control plane waits until both one-shot services exit 0. Completion markers ensure retries never overwrite finished copies, and if no legacy volume exists, migrate-agent-homes exits cleanly without copying.
Leave Agents disabled while you import legacy files into Unified Drive (replace ankole_agents_data below if you use a custom Compose project name):
docker compose run --rm --no-deps --user root --entrypoint chmod \
-v ankole_agents_data:/agents control-plane -R a+rX /agents
docker compose run --rm --no-deps -v ankole_agents_data:/agents:ro control-plane \
/app/bin/ankole eval 'Ankole.Upgrade202610.import_agent_home("/agents")'
Ankole.Upgrade202610.import_agent_home/1 copies legacy installed-skills directories and unsent Outbox/Delivery attachments into each Agent’s private Drive without overwriting existing paths, while the schema migrations separate custom Skills, Skill package files, and Agent contracts into Meta Ankole (Ankole.AIAgent.Library PostgreSQL tables; manage runnable Skill packages in Meta Ankole → Skills and Agent contracts in Meta Ankole → Agents). General user files and Session/Background Job workspaces stay in Home. Save the report, repair skipped files, and rerun the import. Preserve script source separately for conversion to Applet Actions, and retain the old volume until verification and a restore rehearsal succeed.
Upgrade Kubernetes / Helm
- Scale both the 1.x control-plane Deployment and pooled Worker Deployment to
0replicas and wait for every old Pod to terminate cleanly—never force-delete Pods or pass--grace-period=0. - Prepare cluster FUSE prerequisites required by 2.0 Worker
/drivemounts before starting local Kubernetes Workers: install the FUSE device plugin once per cluster (advertisingsmarter-devices/fuseon Linux nodes with/dev/fuse) and label the application namespacepod-security.kubernetes.io/enforce=privilegedso it pairs with the Helm chart’s cluster-scopedValidatingAdmissionPolicyfor the WorkerServiceAccount(Worker Pods run no privileged containers, host namespaces, orhostPathvolumes, and Helm installation requires permission to manage this policy and binding):
kubectl apply -f tools/deploy/kubernetes/fuse-device-plugin.yaml
kubectl -n kube-system rollout status daemonset/ankole-fuse-devices
kubectl label namespace ankole pod-security.kubernetes.io/enforce=privileged --overwrite
- Confirm that your external or bundled database runs PostgreSQL 18+ (
pg_search0.26.4andvector). If you usesecrets.existingSecret(ankole-bootstrap), patch it soRELEASE_COOKIEandANKOLE_FUNCTION_RUNTIME_TOKENexist alongsideANKOLE_SECRET_BASEandDATABASE_URLbefore runninghelm upgrade:
kubectl -n ankole get secret ankole-bootstrap \
-o go-template='{{if not (index .data "RELEASE_COOKIE")}}RELEASE_COOKIE{{"\n"}}{{end}}{{if not (index .data "ANKOLE_FUNCTION_RUNTIME_TOKEN")}}ANKOLE_FUNCTION_RUNTIME_TOKEN{{"\n"}}{{end}}' |
while IFS= read -r secret_name; do
kubectl -n ankole patch secret ankole-bootstrap --type merge \
-p "$(printf '{"stringData":{"%s":"%s"}}' "$secret_name" "$(openssl rand -hex 32)")"
done
(Patch only keys that are not already present in ankole-bootstrap.)
4. Retain your existing shared Agent Home PVC by pointing worker.agents.persistence.existingClaim at it, and keep the original worker.installation and Helm release name. If your bundled PostgreSQL stores data directly at the volume root, keep its existing postgresql.pgdata path (/var/lib/pgsql/18/data) so PostgreSQL does not initialize an empty directory.
5. Normalize ownership and permissions on the Agent Home PVC before booting the 2.0 control plane (requires storage where container root can run chown; replace OLD_AGENTS_PVC with your PVC name):
kubectl -n ankole run ankole-agent-home-prepare --rm -i --restart=Never \
--image=busybox:1.37 --overrides='{"apiVersion":"v1","spec":{
"containers":[{"name":"prepare","image":"busybox:1.37",
"securityContext":{"runAsUser":0},
"command":["sh","-ec","chown -R 1000:1000 /agents; chmod 2777 /agents; chmod -R a+rX /agents"],
"volumeMounts":[{"name":"agents","mountPath":"/agents"}]}],
"volumes":[{"name":"agents","persistentVolumeClaim":{"claimName":"OLD_AGENTS_PVC"}}]}}'
On NFS exports with default root-squash enabled, a Pod cannot run chown; run the ownership change on the NFS server instead, and export the share so Worker root can write and chown to uid/gid 1000 (or map all client access to uid/gid 1000). Volumes mounted via an AWS EFS access point already enforcing uid/gid 1000 need no conversion. See WorkerProviders and cache disks.
Configure matching 2.0 digests for controlPlane.image and functionRuntime.image (plus worker.image only if you override the Worker image explicitly), your drive backend, and the existing Home PVC in values-production.yaml. Keep Agents disabled so no 2.0 Workers launch while legacy bare-UID directories are still being imported. Assuming namespace and release name ankole with fullnameOverride: ankole, run:
helm upgrade ankole ./tools/deploy/helm/ankole-agent \
--namespace ankole --values values-production.yaml \
--set files.importAgentHomeClaim=OLD_AGENTS_PVC --wait --timeout 15m
kubectl -n ankole exec deployment/ankole-control-plane -c control-plane -- \
/app/bin/ankole eval 'Ankole.Upgrade202610.import_agent_home("/agents")'
Because the control-plane Deployment uses the Recreate strategy, the new Pod’s db-migrate init container completes all Ecto migrations before the control plane starts. Keep Agents disabled until import_agent_home/1 finishes while the legacy bare-UID directories are still in place, and verify the printed report.
Workers using a shared Agent Home claim (storage.sharedClaim / worker.agents.persistence.existingClaim) mount tenant-qualified directory names computed by Ankole.AgentWorkers.ProviderKind.storage_name/2 rather than bare <agent_uid> directories. Query each Agent’s destination directory name from the upgraded control plane:
kubectl -n ankole exec deployment/ankole-control-plane -c control-plane -- \
/app/bin/ankole eval 'IO.puts(Ankole.Repo.with_tenant("TENANT_UUID", fn -> Ankole.AgentWorkers.ProviderKind.storage_name("INSTALLATION", "AGENT_UID") end))'
With the control plane scaled to 0 and all Workers stopped, mount OLD_AGENTS_PVC in a preparation Pod and rename each Agent’s legacy <agent_uid> directory to its ProviderKind.storage_name/2 path (reconciling manually rather than overwriting if a destination directory already exists). Then remove files.importAgentHomeClaim and run helm upgrade once more—keeping the shared Home PVC itself, which Worker Pods continue to mount—before re-enabling Agents. Adjust resource names if your release, namespace, or Deployment names differ; see the Helm deployment guide.
Verify and recover
- Inspect migrated Agents, Identity Providers, signal routing rules, and Background Agent Jobs under the
defaulttenant in the Console. Verify that every existing Agent has an assigned WorkerProvider, since the site default provider applies only to newly created Agents. - Enable a single canary Agent first and exercise file tools and a Background Agent Job—checking legacy Home files, Codex thread resumption, imported Skills and published Applet Jobs, drive uploads/downloads, and chat attachment delivery—before re-enabling your remaining Agents and schedules.
- If a migration step fails, leave 1.x replicas stopped, inspect the Compose
migrateor Kubernetesdb-migratelogs, fix the reported secret, credential, group-name collision, or storage permission issue, and re-run the 2.0 migration. import_agent_home/1logs leftover Home directories belonging to previously deleted Agents without recreating those Agents; delete legacy directories or volumes only after confirming none of their files are needed.- If you must roll back to 1.x, stop all 2.0 control planes,
ankole-function-runtimeinstances, and Workers, restore your coordinated pre-upgrade backup (PostgreSQL, anyBASE_DB_URLSshard databases, object/file storage, Agent Homes, configuration, and originalANKOLE_SECRET_BASE), and start the 1.x images. Runninghelm rollbackor reverting image tags alone cannot undo destructive passwordless or schema migrations, nor preserve post-upgrade writes that were not exported separately.
For day-two operations after upgrading, see Workers, Drive storage configuration, and Disaster recovery.