Status and non-binding roadmap

This page separates behavior shipped in the 0.1 series from possible future work. Future items are non-binding directions, not release promises or compatibility guarantees.

Implemented in the 0.1 series

  • Typed Text, JSON, Binary, and Reference payload storage

  • JSONL and optional Arrow IPC / Parquet round trips

  • Immutable Python frame plans for filtering, projection, joins, sorting, limits, and AI-specific operations

  • Exact canonical-hash deduplication and conservative lineage merging

  • Lexical and plugin ranking

  • Required / preferred / optional budget selection

  • Structured overflow and policy diagnostics in English and Japanese

  • External-boundary checks for expiry, quarantine, missing provenance, restricted data, and a small set of secret-like patterns

  • Rust-native SHA-256, OpenAI-compatible token counting for o200k_base and cl100k_base, and Hugging Face tokenizer-JSON support in binary wheels

  • OpenAI-shaped, Anthropic-shaped, and plain-text rendering for Text, JSON, and Reference payloads

Current limitations

  • The Python planner executes frame operations in Python. The Rust crate exposes the Arrow schema and a DataFusion session, but DataFusion does not yet execute the Python transformation plan.

  • profiles.openai() uses exact content tokenization when the native wheel is present, but provider envelope accounting is conservative; its manifest remains estimated=True.

  • profiles.approximate() is a UTF-8 byte estimate, not a provider limit.

  • Binary payloads are storage and round-trip only. Optional Binary items are excluded from packing; required Binary items raise UnsupportedContent.

  • The built-in ranker is lexical. There is no semantic ranker or semantic deduplication.

  • Secret detection is pattern based and can produce false negatives and false positives.

  • to_openai() and to_anthropic() use simple role mappings; they do not preserve provider-specific tool-call identifiers.

  • Omitting id or created_at creates a UUID or current timestamp. Supply both for reproducible manifests.

  • There is no direct MCP, A2A, agent-framework, vector-database, or model-provider network integration.

Non-binding directions

Possible directions include moving compatible frame operations behind the existing DataFusion interface, certifying additional strict token-envelope profiles, adding renderer and token-profile plugins for multimodal content, improving schema migration tooling, and publishing more reproducible performance evidence.

The payload schema was designed so that image and audio renderers can be added without replacing the top-level payload structure. That extensibility is not a claim that those renderers exist today.

Roadmap changes must preserve the security contracts documented in Security model and limitations: required content is never silently dropped, unsupported content is never silently stringified, and structured diagnostics remain the programmatic interface.