Skip to content
Eigenius

13. Troubleshooting and FAQ

Common issues organised by symptom. For each, the diagnosis and the fix.

13.1. Build failures

error: failed to run custom build command for prost-build

Cause: protoc not installed.

Fix: Install protobuf-compiler (Ubuntu/WSL: apt install protobuf-compiler; macOS: brew install protobuf).

error: linker 'cc' not found

Cause: No C/C++ toolchain (Ubuntu/WSL).

Fix: sudo apt-get install -y build-essential.

error: failed to run custom build command for librocksdb-sys

Cause: libclang-dev missing — bindgen can’t read RocksDB’s headers.

Fix: sudo apt-get install -y libclang-dev.

error[E0658]: ... referencing a Rust feature

Cause: rustc older than 1.97, the version deploy/Dockerfile.kernel builds with. Nothing enforces it locally — there is no rust-version key in Cargo.toml and no rust-toolchain file — so an older toolchain shows up as a dependency compile error rather than an MSRV message.

Fix: rustup update to get the latest stable.

cannot locate Lean's include directory from eigenius-lean-worker’s build script

Cause: No Lean toolchain on the host. crates/eigenius-lean-worker/build.rs compiles a C bridge against Lean’s lean.h and panics when it cannot find it. This fails the whole cargo build --workspace, not just the Lean crate.

Fix: Install elan and the pinned toolchain (see chapter 2 §2.1), or point EIGENIUS_LEAN_INCLUDE_DIR at a directory containing lean/lean.h.

Deno Specifier "..." was not found

Cause: Deno cache stale or partial.

Fix: cd orchestration && deno cache --reload src/main.ts.

13.2. Server startup

Error: Address already in use (os error 98) on port 50051

Cause: Another eigenius serve process is already running on the port.

Fix: Find it with lsof -i :50051 (or ss -tlnp | grep 50051) and kill it, or start with --port <other>.

Error: IO error: While lock file: ... LOCK: Resource temporarily unavailable

Cause: RocksDB lock file held by another process. Common causes:

  • eigenius serve --db <path> already running.
  • Previous serve crashed and left the lock (rare; RocksDB usually cleans up).
  • eigenius db stats/compact/export running concurrently.

Fix: Stop the holding process. If you’re sure nothing’s holding it, delete the LOCK file in the database directory and retry.

Drift refusal: embedded ontology hash mismatch

Error: embedded ontology 'urn:eigenius:core' hash differs from persisted manifest
  expected: ...
  found:    ...

Cause: You upgraded eigenius to a version with a different embedded ontology, and tried to start it against an existing database.

Fix: Either roll back to the prior version, or migrate. The migration path: export with the prior version (eigenius db export), delete the database, restart with the new version (re-seeds the manifest), re-load the export.

Kernel hangs on startup with Connection refused (orchestrator)

Cause: --orchestrator <url> points at an orchestrator that isn’t running, or that’s reachable but on a different port.

Fix: Start the orchestrator first; verify with curl http://localhost:8080/health. The kernel won’t fail outright on a missing orchestrator (some operations work without it), but it will retry the connection — which can look like a hang.

13.3. Orchestrator

Module not found on orchestrator start

Cause: Deno hasn’t cached the dependency tree.

Fix: cd orchestration && deno cache src/main.ts.

Orchestrator port already in use (8080)

Cause: Another service on 8080 (common — many development servers default to it).

Fix: Run with a different port: EIGENIUS_ORCHESTRATOR_PORT=8081 deno run ... and tell the kernel to use that endpoint: eigenius serve --orchestrator http://localhost:8081.

13.4. CLI ↔ kernel connection

Error: connection error when running CLI commands

Cause: Kernel server not running or --endpoint URL wrong.

Fix: Verify the kernel is up: eigenius --endpoint http://localhost:50051 inspect "urn:eigenius:core:Class". If that succeeds, your CLI command’s URL is wrong; if it fails, the kernel isn’t running.

Error: gRPC status: ... INTERNAL ...

Cause: Kernel-side error during the operation. The status message usually contains the underlying issue.

Fix: Check the kernel’s stdout for an ERROR log line that fired at the same time. Common underlying causes: validation failure, type-check failure, missing layer, unregistered capability.

CLI command says “load successful” but query returns nothing

Cause: In-process mode (no --endpoint) — the load happened against an ephemeral in-memory chain that’s discarded when the CLI exits.

Fix: Either use --endpoint <url> with a running kernel (load and query against the same chain), or use the --file option of query to load and query in one invocation.

13.5. Capabilities

capability install / --capability / --kind are not recognised

Cause: They were removed with the WASM extensibility path (2026-07-08). CapabilityCommands has three variants — list, inspect, test — and no install; there is no --capability flag and no --kind flag anywhere in the CLI. wasmtime is not a workspace dependency, so there is no fuel budget, no linear-memory limit and no WIT-world check to hit either.

Fix: Components and institutions are declared as ontology resources and committed with eigenius load; execution goes through the runtime substrate (chapter 11). See chapter 4 §4.10.

capability list doesn’t show a component you committed

Cause: The declaration did not commit, or it committed to a branch the kernel is not serving from, or the component IRI is unregistered — an unregistered component IRI is not an error at dispatch, it returns its input unchanged.

Fix: eigenius --endpoint <url> inspect <iri> to confirm the resource is in the chain the kernel resolves against, and check the branch you loaded onto.

The orchestrator cannot reach the kernel

Fix: From the orchestrator container, check the kernel’s port with a gRPC call, not an HTTP one. The kernel serves no HTTP endpoint — curl http://kernel:50051/health cannot work; the port speaks gRPC and gRPC-Web only. The working probe is the one the compose file’s healthcheck uses:

Terminal window
eigenius --endpoint http://kernel:50051 inspect "urn:eigenius:core:Class"

/health exists on the orchestrator (port 8080), not the kernel.

13.6. Layer / data issues

Validation failed: required property 'X' missing on resource 'Y'

Cause: A loaded resource doesn’t carry every property its class declares as requires.

Fix: Add the missing property, or change the class’s requires to recommends if it should be optional.

Error: class 'urn:...' not found in layer chain

Cause: A resource references a class IRI that isn’t loaded into the layer chain.

Fix: Load the class’s defining file before the resource that uses it. Order matters at load time, even though resolution at query time walks the full chain.

Error: subclass cycle detected

Cause: A class declares a subclass_of that transitively reaches back to itself.

Fix: Find the cycle in your ontology and break it. Subclass chains must be acyclic.

13.7. Performance

Queries slow down over time

Cause: RocksDB needs compaction (with persistent mode), or the layer chain has grown deep.

Fix: Run eigenius db compact <path> on your database, with the kernel stopped. If query time is dominated by deep layer-chain walks, consolidate the chain: eigenius --endpoint <url> db consolidate <from-hex>..<to-hex> collapses an inclusive layer range into one resolve-equivalent layer. It ships — re-loading into a fresh database is no longer the workaround. Run it with --dry-run first to see the cost and the predicted layer id. See chapter 4 §4.5 and D25.

eigenius run is slow on programs with many IO calls

Cause: Each component dispatch is a kernel→orchestrator round trip, and a RunRuntimeScript dispatch also spawns a worker container unless one is cached for its image.

Fix: For repeated runs of the same program over the same input, the kernel’s trace store memoises component dispatches — the second run is much faster than the first.

13.8. Frequently-asked questions

Can I run the kernel without the orchestrator?

Yes, for read-only operations (queries, inspection, type-check). For programs that dispatch IO components (RunRuntimeScript, CallRuntimeMethod), the orchestrator is required.

How do I version my ontologies?

Conventionally, version through the URI: urn:my-org:ontology:v1, urn:my-org:ontology:v2. Layers are immutable, so loading the v2 ontology adds new resources without disturbing v1 — both versions are queryable in parallel against the chain.

How do I delete a resource?

Layers are immutable, so nothing is deleted in place. Two shapes exist. Load a new layer that supersedes the resource — later layers shadow earlier ones on resolve. Or tombstone it: eigenius load <file> --explicit-tombstone <iri> commits a tombstone alongside the new layer, and --commit-policy cascade additionally tombstones lower-layer resources that the new layer’s class redefinitions retroactively invalidate (D41 §3.3, §10.1).

Can I run multiple kernels against the same database?

No. RocksDB takes an exclusive directory lock; only one process at a time can open --db <path>. This is also why db stats, db compact and db export need the kernel stopped. For horizontal scaling, run multiple kernel instances each with its own database.

Does the kernel auto-restart on crash?

The kernel itself doesn’t supervise itself. Wrap it in a process supervisor:

  • systemd with Restart=always for bare-metal hosts.
  • Docker Compose with restart: unless-stopped for containerised setups.
  • ContainerApps auto-restarts crashed containers by default.

How do I rotate the ANTHROPIC_API_KEY?

In container environments, update the env var (or the Key Vault secret) and restart the kernel. The orchestrator doesn’t see the key — only the kernel does, for D71 formalization runs.

Where do reasoning traces live?

When --db is set, in the default column family under the key prefix trace:<key_hex>. There is no traces column family: the only column families the store opens are cf_text, cf_vec and cf_embed_cache. The trace shape is specified in D6b — Reasoning trace schema. Without --db, traces are in-memory and discarded on kernel exit.

How do I clean out old traces?

There is no trace eviction policy and no CLI surface for garbage collection. For now: eigenius db export to capture what you want, drop the database, restart fresh.

Why does my --endpoint URL say localhost work locally but not from a container?

Containers don’t share the host’s localhost namespace. From inside a container, localhost refers to the container’s network. To reach a service on the host: use the host’s IP, or (in Docker Compose) use the service name (http://kernel:50051).

Next: 14. Notebook →