Troubleshooting

Start with the stage that failed. The portal's import preview, replay details, job status, and the Nginx reason header each narrow the problem.

Import reports row errors

Use the template generated for the saved pipeline version. For request-event JSONL, check endpoint_id, method, path, timestamp order, selected fields, and actual body_size. Missing selected fields or non-scalar values are errors. For CSV, check exact feature header names and order, finite numeric cells, and the matching manifest hash. Fix all reported rows; imports with any error are rejected.

If fewer than 20 complete vectors remain, collect more representative data. Request-event rows during the longest-window warm-up do not count as complete vectors.

Training job failed

Read the job's error and audit information in the portal. The worker retries transient failures up to three attempts, and an administrator can retry a failed job explicitly. If an imported dataset's artifact file is missing or corrupt, restore the exact matching artifact with its recorded SHA-256 or reimport and retrain. The database and artifact volume must be retained together.

check has no score

This means the request was not fully scored. Inspect X-PragmaChange-Reason or request replay. Common causes include partition warm-up, missing configured fields, malformed or oversized JSON, disk-buffered request bodies, truncated history, and state capacity. Do not treat the empty score header as 0.

Too many warm-ups

Check whether client identifiers change on every request, the partition secret was rotated, a new pipeline version was deployed, or memory/cardinality pressure is evicting histories. Each proxy host also has separate state, so requests spread across several hosts warm up independently.

Nginx rejects the module or release

Match the module build to the installed Nginx binary's version and configure flags. Validate the archive before installing. Check the local secret file permissions, pragmachange_release path, and nginx -t output. The deployment script restores the previous symlink after an immediate install failure.

A normal request is blocked

Replay representative requests and inspect their feature vectors and scores. Check whether the training dataset reflects ordinary variation for this endpoint, whether feature order changed, and whether thresholds are too aggressive. Keep application rules and authentication in place; anomaly scores alone are not a substitute.

Docker cannot start or the portal is unavailable

Check docker compose ps and docker compose logs --tail 100 api trainer portal. A Docker socket permission error is a host access issue; the application cannot fix it. Keep port 3000 free and use the exact PUBLIC_ORIGIN. Existing administrators retain their stored password even if environment settings change.

An offline recording is rejected

Close/dispose the writer and upload the matching records.jsonl and manifest.json. Unfinished, empty, oversized, or edited files are rejected. Preserve original byte counts and SHA-256; do not edit hashes to disguise a mismatch. Request recordings need the actual saved pipeline/schema/endpoint; behavior recordings need the selected application. See recording limits.

Behavior training has insufficient data

A row count alone is not enough: whole chronological workflows must supply at least five workflows, 100 usable training events, and 20 events in each calibration/test split. Missing boundaries, per-actor caps, or split purging can reduce usable data. Collect representative workflows and inspect job diagnostics. New applications stay in shadow mode until explicitly configured otherwise.

A remote deployment is unknown or its preview expired

Restore connectivity and reconcile the deployment journal. Inspect the host again before preparing another change. Do not treat a connection interruption as success or replay an uncertain activation. Configuration drift invalidates prior previews. Loaded-release verification is timestamped, not continuous monitoring.

Credentials or Assisted setup are unavailable

Check that the API can read the configured vault key and that Docker mounts it at the container path. Keep the original key with database backups. Configure the provider explicitly and inspect job errors; failed provider calls do not trigger automatic paid retries.