Skip to main content
Version: 1.3.0

Troubleshooting

Login redirects, signs out, or fails intermittently​

  • Confirm the browser-visible OIDC issuer exactly matches the token issuer.
  • Test OIDC discovery and token endpoints from every application node. One node with blocked egress causes intermittent failures behind a healthy Service.
  • For multiple replicas, configure one shared session-encryption key. Sticky ingress sessions are not the primary fix.
  • In standalone, open http://host.docker.internal:8090, not localhost.

Vault login returns 403​

Decode the pod JWT and confirm its sub matches a ServiceAccount bound to the Vault role. Then inspect vault read auth/<mount>/config:

  • token_reviewer_jwt_set=true: verify the reviewer ServiceAccount and TokenReview permission.
  • token_reviewer_jwt_set=false with disable_local_ca_jwt=true: enable vaultKubernetesAuth.appAuthDelegator; the application ServiceAccount needs TokenReview.
  • Confirm projected-versus-legacy token expectations before enabling legacyToken.

Secret changed but the app still uses old credentials​

Check whether the Secret or ExternalSecret actually synchronized. One missing remote property can leave an ExternalSecret at its last good value. Kubernetes secretKeyRef environment variables are loaded at process start, so roll the Deployment after successful synchronization.

Pod restarts repeatedly (OOMKilled, exit code 137)​

kubectl describe pod shows Last State: Terminated, Reason: OOMKilled and a climbing restart count. The container exceeded its memory limit and the kernel killed it. This is total process memory — heap plus non-heap (metaspace, JIT code cache, thread stacks, Netty/Vert.x direct buffers, the database pool) — not a Java OutOfMemoryError, so the application log usually ends mid-run with no error: the JVM is killed before it can report anything.

The JVM (fast-jar) image launches with -XX:MaxRAMPercentage=80, so at a tight limit the heap alone crowds out everything non-heap. Do not raise -Xmx in response — that makes it worse. Instead:

  • Cap the heap below the limit with the chart's jvm.maxRamPercentage (default 50 → -XX:MaxRAMPercentage=50), leaving room for non-heap.
  • Give the container headroom: the chart default limit is 768Mi (requests 512Mi). 384Mi is too small for this feature set.
  • Confirm it is not a leak: kubectl logs <pod> --previous shows the effective JVM flags on the launch line, and a clean, steady run ending abruptly with no OutOfMemoryError points to an under-provisioned limit, not a leak. Watch kubectl top pod settle after the change.

A native image runs in a far smaller, flatter footprint and avoids this JVM heap-versus-limit tuning entirely.

Registration or cluster probe fails​

Verify routing from the DonkeyFleet runtime, the Vault path and username/password keys, TLS trust, and least-privilege ONTAP access. Workstation connectivity alone is not sufficient.

Run says skipped_observe_incomplete​

An ONTAP read failed, pagination did not finish, or record counts disagreed. Do not bypass this guard. Fix reachability, credentials, rate limits, timeouts, or the endpoint response.

Approved work does not execute​

Check effective dry-run, minimum volume age, current profile membership, max_actions_per_run, baseline concurrency, bandwidth, and unresolved ONTAP jobs. Approval is authorization, not a guardrail bypass.

Apply is Ready or In flight for too long​

Inspect the checkpoint, stored job UUID, and last error. Ready work waits for admission or a baseline slot. In-flight work must poll the existing ONTAP job. Do not manually repeat create or initialize while job outcome is unresolved.

The next active-provisioning follow-up normally runs after 15 seconds; long baseline monitoring returns to the normal five-minute cadence.

Baseline reports unhealthy​

Initialization can remain uninitialized and unhealthy for hours. Check transfer and job state; do not suspend it solely for not being complete. Overrun alerts use the larger of the physical-size estimate and a ten-minute floor, multiplied by the configured factor.

Managed relationship becomes suspended after a completed baseline​

Upgrade to a build containing Flyway V16 and restart against the preserved PostgreSQL database. That migration repairs relationships affected by the earlier completed-baseline lifecycle defect.

Run says skipped_peer_down or skipped_orphan_delta_guardrail​

Repair SVM peer health outside DonkeyFleet. For the orphan guardrail, investigate source visibility and UUID identity before changing either the absolute or percentage limit.