Skip to content
Eigenius

11. Runtime substrate

Chapter 10 covered the WASM extension surface. This chapter covers the other extension surface: the runtime substrate — the orchestrator-spawned, container-hosted runtime that lets institutions live in their natural language ecosystem (Julia, eventually Python and others) instead of inside a WASM sandbox.

The two surfaces are peers, not alternatives. WASM institutions are right when sandboxing matters, when the institution is hot-installable, or when the dependency tree is small. Substrate-hosted institutions are right when the institution wants to use a heavy native library (a SAT solver, an ODE integrator, a quantum-chemistry engine) that doesn’t compile to WASM cleanly. Both speak the same chain protocol; the same Institution, QueryClass, ExportFormat, ImportFormat, and Comorphism resources work for both.

The substrate is specified by D26 — Runtime Substrate, D29 — Mirror Generator, and D31 — Institution Lifecycle; the implementation lives at crates/runtime-substrate/.

Cross-link: this chapter is the implementer view from the substrate side. The user view (calling institutions from queries and programs) is shared with WASM institutions and lives in ESL §9 and EigenQL §8. The Julia v1 instantiation has dedicated tutorials under julia-institutions/ — start with the intervals tutorial for the full slow-walk.

11.1. The substrate concept

The runtime substrate is a small Rust crate (crates/runtime-substrate/) loaded into the orchestrator process. Its job is to host external runtimes — workers that aren’t WASM-sandboxed and aren’t in-process Rust, but instead run as sibling containers of the orchestrator (Docker-out-of-Docker via a bind-mounted Docker socket, not nested) and communicate over a Unix domain socket using Eigon-CBOR.

┌──────────────┐        ┌──────────────────┐          ┌──────────────────┐
│   Kernel     │ ──RPC─▶│  Orchestrator    │  spawns  │  Worker          │
│   (Rust)     │  call  │  (Deno + napi-rs │ ────────▶│  (Julia / etc)   │
│              │   ◀────┤  substrate addon)│          │  Eigon-CBOR / UDS│
└──────────────┘ result └──────────────────┘          └──────────────────┘
                            depot bind-mount  ──────  /var/lib/eigenius/
                            (host ⇆ orch ⇆ worker)    substrate-depot/

Three pieces worth naming up front:

  • LanguageRuntime trait (language_runtime.rs) — the trait every substrate-hosted runtime implements. The Julia v1 implementation lives in crates/eigenius-julia/; Python and others are tracked in issue #41. One trait, one method per chain operation (build_environment_image, call_method, etc.).

  • Spawner — the per-runtime container lifecycle. The Julia spawner uses docker create + docker start against the env image’s content-addressed digest. The first call against a new env-image digest spawns a worker; subsequent calls against the same digest reuse the cached ServiceHandle. Workers stay alive between calls (Julia’s startup cost makes per-call containers prohibitively expensive); the substrate keeps a per-image pool in memory.

  • The depot bind-mount — the directory /var/lib/eigenius/substrate-depot/ is bind-mounted into the orchestrator container at the same path it appears on the host. Workers write their UDS sockets into this directory, and the orchestrator reads them at the same absolute path. This is the one piece of host configuration the substrate depends on; without it, the orchestrator can’t reach worker sockets.

The kernel itself doesn’t know any of this. It dispatches via the existing DispatchExternal gRPC, the orchestrator routes the dispatch to the substrate, and the substrate marshals through the env image and worker container. The kernel sees a ResourceCBOR → ResourceCBOR operation; the substrate handles the runtime-specific plumbing.

11.2. The four chain resources

Substrate institutions are declared on the chain through four typed resources. Each is committed via the same eigenius load / eigenius institution install flow that any other ontology declaration uses; nothing about the substrate is operationally privileged.

ResourcePurpose
RuntimePackageMirrorAuto-generated, language-specific source code that mirrors a slice of the chain (selected Class and InductiveType resources) into typed structs the worker can decode and dispatch on. Content-addressed by source hash.
RuntimeEnvironmentPinned identity of a worker image: image_digest (sha256), runtime_version (e.g. 1.12.6), lockfile, lifecycle: Service. Refers to the mirror it bakes in.
RuntimeMethodSignatureTyped input/output contract for a single handler method — input_types: [Class₁, …], output_type: Class. Used by the kernel to marshal arguments and validate returns; used by the mirror generator’s closure walker to discover cross-institution classes.
Institution { runtime: external, requires_environment: <env-iri>, … }The chain-side identity of the institution; refers to the env it dispatches against.

The first three (mirror, env, signatures) are lifecycle resources; the fourth (Institution) is the same resource type WASM and in-process institutions also use, distinguished by its runtime property.

A walk through the contract:

  1. The mirror takes a snapshot of the chain shapes the institution needs to encode/decode. For Julia, this becomes EigeniusMirror.jl with struct BoundedBy ... end + decode_BoundedBy(::Dict)::BoundedBy + encode_BoundedBy(::BoundedBy)::Dict per class. The chain commit is content-addressed — re-running the generator over the same seed produces the same IRI.
  2. The env image bakes the mirror plus the handler package plus the language runtime plus the worker library. eigenius env build runs the buildah-driven build on the host and prints the resulting digest; you then commit the RuntimeEnvironment resource pinning that digest plus the runtime version.
  3. The institution declaration ties everything together — names the environment it requires, declares its RuntimeMethodSignatures, and declares its QueryClasses / ExportFormats / ImportFormats.
  4. At dispatch time, the kernel resolves the QC’s query_handler procedure to a method name on a signature; the substrate looks up the env’s image digest, ensures a worker is running (cold-spawn or warm-pool), marshals the input as Eigon-CBOR, sends a DispatchMethod over UDS, and returns the worker’s typed response.

11.3. Install flow end-to-end

The CLI surface lives in chapter 4 §4.7–§4.9. At a high level, installing a substrate-hosted institution is four steps:

Terminal window
# 1. Make sure the chain has the institution's ontology loaded.
eigenius load <institution-ontology.eigon.json>


# 2. Generate the mirror against the head layer; commit a
#    `RuntimePackageMirror` resource and write the source files locally.
eigenius mirror create \
    --layer "$(eigenius branch show main | awk '{print "urn:eigenius:layer:"$2}')" \
    --filter '<EigenQL query selecting seed classes>' \
    --institution-file <institution-declaration.eigon.json> \
    --language julia \
    --output /tmp/my-institution-mirror \
    --json


# 3. Build the env image with the handler package + mirror baked in.
#    Prints the image digest + runtime version.
eigenius env build \
    --language julia \
    --package-path <handler-package-dir> \
    --mirror <mirror-iri-from-step-2> \
    --base-image docker.io/library/julia:1.12-bookworm \
    --json


# 4. Commit the `RuntimeEnvironment` resource pinning the digest.
eigenius env create \
    --language julia \
    --handler-package <handler-package-dir> \
    --mirror <mirror-iri-from-step-2> \
    --as-iri urn:my-org:my-institution:env:v1 \
    --image-digest <digest-from-step-3> \
    --runtime-version <version-from-step-3>


# 5. Install the institution declaration (commits Institution +
#    RuntimeMethodSignature x N + QueryClass x N + ExportFormat / ImportFormat).
eigenius institution install \
    --definition <institution-declaration.eigon.json>

The slow-walk version of this exact flow — for the IntervalArithmetic.jl institution — lives at julia-institutions/intervals-institution-tutorial.md. That’s the recommended first reading.

The --institution-file closure walker

Step 2’s --institution-file flag is worth flagging. The mirror generator walks the closure of every reachable class from the seed query — properties’ class_types, requires, etc. — but cross-institution return classes aren’t reachable that way. If a Symbolics handler returns an OptimisationProblem (declared by JuMP, not Symbolics), the closure walker can’t find it from a Symbolics-rooted seed.

--institution-file <path> augments the seed by parsing the institution declaration directly and pulling every class IRI mentioned in RuntimeMethodSignature.input_types / output_type. The flag reads the file rather than querying the chain because, in the install order above, the institution declaration only commits in step 5 — after the mirror generator runs. The file is the source of truth at this stage of the pipeline.

11.4. Worked example — see the intervals tutorial

The smallest viable substrate institution is IntervalArithmetic.jl (rigorous numerical bounds). It’s the canonical worked example for the substrate plumbing:

  • One typed shape on the chain (BoundedBy { value, lower, upper }).
  • One handler method (validate_bounded_by) returning a Verdict.
  • One QueryClass with dispatch_role: AutoOnLoad.
  • The full install flow exercised step-by-step against a live kernel.

The slow-walk lives at julia-institutions/intervals-institution-tutorial.md. Reading it once is the fastest way to internalise the substrate’s surface — every concept in this chapter is grounded against a live install in that walkthrough.

For richer surfaces, the same subdirectory has tutorials for the four other institutions in the kinase notebook (Symbolics, Catalyst, DiffEq, JuMP-HiGHS) — each goes one level deeper:

  • Symbolics tutorial — three dispatch roles in one institution; the FormulaTerm-as-EigenTT-fragment story end-to-end.
  • Catalyst tutorial
    • DiffEq tutorial — the comorphism that compiles a chemical reaction network into an ODE.
  • JuMP-HiGHS tutorial — the smart-pow walker rule that keeps quadratic objectives in QuadExpr rather than NonlinearExpr territory.

11.5. Substrate vs. WASM trade-off

AspectWASM institution (chapter 10)Substrate institution (this chapter)
SandboxYes (Wasmtime)No (sibling container)
Hot-installableYes (auto-registration on commit)Requires env build step on the host
Dependency treeRestricted (anything that compiles to WASM)Full language ecosystem (Julia, Python, …)
Native libsNone (or compiled to WASM)Full access (BLAS, MPI, HiGHS, etc.)
Cold-startSub-millisecondSeconds (container spawn + language warm-up)
Steady-state cost per callWasmtime overhead per callUDS round-trip per call
Trust modelUntrusted authors OK (sandboxed)First-party / vetted authors
Best forCap-as-predicate validation; small reasoners; 3rd-party extensionsNumerical / scientific libraries; large precompilation tails; first-party reasoning systems

The kernel dispatches identically against both — same Institution::query trait surface, same Verdict shape, same FIBER param coercion semantics for comorphisms. The difference is operational, not protocol.

11.6. Where the implementation lives

FilePurpose
crates/runtime-substrate/src/language_runtime.rsLanguageRuntime trait
crates/runtime-substrate/src/spawner/Container-spawning logic
crates/runtime-substrate/src/image_build/buildah-driven env image construction
crates/runtime-substrate/src/rpc/Eigon-CBOR-over-UDS protocol
crates/runtime-substrate/src/mirror_generator.rsClosure walker for chain shapes
crates/runtime-substrate/src/boundary.rsTyped boundary marshalling
crates/runtime-substrate/src/chain.rsChain accessor abstraction
crates/runtime-substrate/src/cross_check.rsType validation across the boundary
crates/runtime-substrate/src/facade.rsSubstrate facade exposed to the orchestrator
crates/eigenius-julia/Julia LanguageRuntime instantiation
julia/runtime-worker/The Julia worker source loaded into the env image
julia/common/EigeniusJuliaCommon/Substrate-side Julia utilities
julia/institutions/The five v1 Julia institutions
julia/comorphisms/Cross-institution comorphism declarations

11.7. Cross-references

Next: 12. Deployment →