Crate cognoxium_core 

Cognoxium’s native data and tokenization core.

The crate deliberately contains no network client, model invocation, API key handling, or telemetry. Python bindings translate structured errors at the language boundary.

Structs

ContextItem

Canonical in-memory context record shared by native extensions.

Diagnostic

Locale-independent diagnostic returned across language boundaries.

SourceRef

Stable provenance reference attached to a context item.

Enums

CoreError

Payload

A context payload. Binary storage is supported even when a renderer cannot yet send that payload to a model provider.

Retention

Role

Provider-neutral conversational role.

Sensitivity

Trust

Constants

SCHEMA_VERSION

Stable schema version written into manifests and serialized records.

Traits

Policy

Returns a stable reason code when an item violates a boundary policy.

Ranker

Scores context items in one batch without imposing an output order.

Renderer

Converts selected context into an explicit provider representation.

Source

Loads context without coupling the core to a storage or agent framework.

TokenProfile

Combines deterministic tokenization with provider envelope accounting.

Functions

context_schema

Return the Arrow schema used at the Rust/Python interoperability boundary.

count_huggingface_batch

Count values with a Hugging Face tokenizer JSON definition.

count_tiktoken_batch

Count an OpenAI-compatible encoding in parallel without entering Python.

session_context

Create a DataFusion session for lazy, in-process execution.

sha256_hex

Compute a stable lowercase SHA-256 digest.